I've also tested uploading 1mil of simple pages and 8GB of RAM was not enough for this. Since probably generating such amount of pages is not main application domain of XWiki I'm not sure if xwiki devs would consider this to be a bug or unoptimized case. I'm asking since I find kind of bad to generate a *big* document describing xwiki changes in the past hour even there is no one interested in this document and this way consume all the RAM and get XWiki to OOM error. Question: shall I report it as a bug or feature request asking for kind of lazy (when it is actually needed) changelog generation pointing to this thread? Thanks! Karel On 05/30/14 08:31 PM, Thomas Mortagne wrote:
The references to the watclist in your log make me think about something: go to you XWiki user profile, disable "Automatic document watching" in Watchlist section and try again.
By default evey time a user create a page it's added to his watchlist and I'm wondering if your OOM here is watchlist trying to generate a mail about what happen in the past hour which would certainly generate a huge mail for 400k documents.
On Fri, May 30, 2014 at 8:12 PM, Karel Gardas<[email protected]> wrote:
Paul,
On 05/28/14 08:11 PM, Paul Libbrecht wrote:
Of course I know the way how to increase Java's memory space/heap space. The problem is that this will not help here. Simply if I do so and then create 100 millions of pages on one run I will still get to the same issue just it'll take a lot longer.
You won't reach 100 time the memory taken by downloading a 100 million pages after you download a million pages. The caches have limits and you should measure how much memory in your setting is needed to reach that limit. Tests such as yours, after they reach the max, will get slower because of the DB access following the cache eviction; they are not very realistic under that point of view.
yes, I'm sure the test is not that realistic so far since majority of xwiki usecases is probably download/rendering of page than upload of page which I'm testing now. The reason of my test is simple, my task is to attempt importing whole Wikipedia content w/o history into the XWiki. For this I need to have it stable for import which is something I do have problem now.
The best way to follow this is to use JMX and get a graph of the cache values, you should see maximum be reached after a while. At curriki, we have adjusted the caches to be a bit bigger and we do reach the maximum (100'000 if I remember well, for the document cache) after about an hour after the restart. Then things get pretty stable with our max 8G of memory.
I'm glad that your setup is stable. I do have following cache setup here:
$ grep cache `find . -name 'xwiki.cfg'` #-# Put a cache in front of the document store. This greatly improves performance at the cost of memory consumption. xwiki.store.cache=1 #-# Maximum number of documents to keep in the cache. xwiki.store.cache.capacity=100 #-# Maximum number of documents to keep in the rendered cache xwiki.render.cache.capacity=100 # xwiki.authentication.ldap.groupcache_expiration=21600 xwiki.plugin.image.cache.capacity=30
this is XWiki 6.0.1 on top of Tomcat 7.0.53 using PostgreSQL 9.3/64bit. The JVM is 1.7.0_07 everything running on Solaris 11.1. When I give 2GB RAM to catalina by:
$ cat bin/setenv.sh CATALINA_OPTS="-server -Xms2048m -Xmx2048m -XX:MaxPermSize=196m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100" export CATALINA_OPTS
and then run my benchmark which is just using REST to import following simple page:
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>" ++ "<page xmlns=\"http://www.xwiki.org\">" ++ "<title>Semantic XWiki Benchmark page " ++ title ++ "</title>" ++ "<syntax>xwiki/2.0</syntax>" ++ "<content>This is a benchark page." ++ "The list of properties defined for this page is:" ++ "</content>" ++ "</page>"
where title is BenchPage_<number>
Now, when I pushed that, then after around 400k pages imported I hit OOM error. JConsole is thrown out by closed connection and even benchmark tool complains about few 500 HTTP error codes received. The exception output on Tomcat's console looks:
Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space Exception in thread "RMI TCP Connection(idle)" Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread "TxCleanupService,platform.security.authorization.cache,local" java.lang.OutOfMemoryError: Java heap space Exception in thread "TxCleanupService,localization.bundle.document,local" Exception in thread "TxCleanupService,wiki.descriptor.cache.wikiId,local" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread "TxCleanupService,xwiki.store.pageexistcache,local" java.lang.OutOfMemoryError: Java heap space Exception in thread "TxCleanupService,xwiki.store.pagecache,local" java.lang.OutOfMemoryError: Java heap space May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle WARNING: Exception or error caught in status service java.lang.OutOfMemoryError: Java heap space
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle WARNING: Exception or error caught in status service java.lang.OutOfMemoryError: Java heap space at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2442) at java.lang.Class.getMethod0(Class.java:2685) at java.lang.Class.getMethod(Class.java:1620) at org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.getNotificationManager(LegacyNotificationDispatcher.java:94) at org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.onEvent(LegacyNotificationDispatcher.java:109) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:299) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:264) at com.xpn.xwiki.XWiki.saveDocument(XWiki.java:1323) at com.xpn.xwiki.api.Document.saveDocument(Document.java:2299) at com.xpn.xwiki.api.Document.save(Document.java:2202) at com.xpn.xwiki.api.Document.save(Document.java:2196) at org.xwiki.rest.internal.resources.pages.ModifiablePageResource.putPage(ModifiablePageResource.java:67) at org.xwiki.rest.internal.resources.pages.PageResourceImpl.putPage(PageResourceImpl.java:62) at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.restlet.ext.jaxrs.internal.wrappers.AbstractMethodWrapper.internalInvoke(AbstractMethodWrapper.java:171) at org.restlet.ext.jaxrs.internal.wrappers.ResourceMethod.invoke(ResourceMethod.java:291) at org.restlet.ext.jaxrs.JaxRsRestlet.invokeMethod(JaxRsRestlet.java:1043) at org.restlet.ext.jaxrs.JaxRsRestlet.handle(JaxRsRestlet.java:792) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Router.doHandle(Router.java:500) at org.restlet.routing.Router.handle(Router.java:740) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159)
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle WARNING: Exception or error caught in status service java.lang.OutOfMemoryError: Java heap space at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2442) at java.lang.Class.getMethod0(Class.java:2685) at java.lang.Class.getMethod(Class.java:1620) at org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.getNotificationManager(LegacyNotificationDispatcher.java:94) at org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.onEvent(LegacyNotificationDispatcher.java:109) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:299) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:264) at com.xpn.xwiki.XWiki.saveDocument(XWiki.java:1323) at com.xpn.xwiki.api.Document.saveDocument(Document.java:2299) at com.xpn.xwiki.api.Document.save(Document.java:2202) at com.xpn.xwiki.api.Document.save(Document.java:2196) at org.xwiki.rest.internal.resources.pages.ModifiablePageResource.putPage(ModifiablePageResource.java:67) at org.xwiki.rest.internal.resources.pages.PageResourceImpl.putPage(PageResourceImpl.java:62) at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.restlet.ext.jaxrs.internal.wrappers.AbstractMethodWrapper.internalInvoke(AbstractMethodWrapper.java:171) at org.restlet.ext.jaxrs.internal.wrappers.ResourceMethod.invoke(ResourceMethod.java:291) at org.restlet.ext.jaxrs.JaxRsRestlet.invokeMethod(JaxRsRestlet.java:1043) at org.restlet.ext.jaxrs.JaxRsRestlet.handle(JaxRsRestlet.java:792) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Router.doHandle(Router.java:500) at org.restlet.routing.Router.handle(Router.java:740) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159) at org.restlet.routing.Filter.handle(Filter.java:206) at org.restlet.routing.Filter.doHandle(Filter.java:159)
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle WARNING: Exception or error caught in status service java.lang.OutOfMemoryError: Java heap space
2014-05-30 16:12:18,851 [DefaultQuartzScheduler_Worker-7] ERROR o.q.c.JobRunShell - Job DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an unhandled Exception: java.lang.OutOfMemoryError: Java heap space 2014-05-30 16:12:18,856 [XWiki Solr index thread] WARN o.h.u.JDBCExceptionReporter - SQL Error: 0, SQLState: 08001 2014-05-30 16:12:18,856 [XWiki Solr index thread] ERROR o.h.u.JDBCExceptionReporter - The connection attempt failed. 2014-05-30 16:12:18,858 [DefaultQuartzScheduler_Worker-7] ERROR o.q.c.ErrorLogger - Job (DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an exception. org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-1.6.5.jar:1.6.5] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) [quartz-1.6.5.jar:1.6.5] Caused by: java.lang.OutOfMemoryError: Java heap space 2014-05-30 16:12:18,859 [DefaultQuartzScheduler_Worker-7] ERROR c.x.x.p.s.StatusListener - Job (DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an exception. org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-1.6.5.jar:1.6.5] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) [quartz-1.6.5.jar:1.6.5] Caused by: java.lang.OutOfMemoryError: Java heap space
Funny thing is that before the error JConsole does not show any signs that the server was allocating a lot of memory. Probably this should be quick spike in mem consumption or otherwise I don't understand at all why this happen.
Now, the question is: I do have or I think I do have very limited cache setup here. I've increase RAM side to 2GB which is over the recommended size by xwiki.org itself -- they warn about size bigger than 1GB due to slowness of GC then. Anyway, 2GB, limited size of cache and yet I hit OOM. Do you think I shall increase RAM limit even further?
Thanks! Karel
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs