30 May
2014
30 May
'14
10:02 p.m.
Thomas,
very good idea, but unfortunately my user is Admin and fortunately it
does have automatic document watching (found in Watchlist) disabled
since otherwise it would probably attempt to sent email to
webmaster(a)xwiki.org...
Anyway, is there any other user in the Xwiki created by default with
automatic document watching enabled and email set? At least I don't see
any in http://localhost:8080/xwiki/bin/view/Main/UserDirectory (except
Admin)
Thanks!
Karel
On 05/30/14 08:31 PM, Thomas Mortagne wrote:
The references to the watclist in your log make me
think about
something: go to you XWiki user profile, disable "Automatic document
watching" in Watchlist section and try again.
By default evey time a user create a page it's added to his watchlist
and I'm wondering if your OOM here is watchlist trying to generate a
mail about what happen in the past hour which would certainly generate
a huge mail for 400k documents.
On Fri, May 30, 2014 at 8:12 PM, Karel Gardas<karel.gardas(a)centrum.cz> wrote:
Paul,
On 05/28/14 08:11 PM, Paul Libbrecht wrote:
yes, I'm sure the test is not that realistic so far since majority of xwiki
usecases is probably download/rendering of page than upload of page which
I'm testing now. The reason of my test is simple, my task is to attempt
importing whole Wikipedia content w/o history into the XWiki. For this I
need to have it stable for import which is something I do have problem now.
>
> Of course I know the way how to increase Java's memory space/heap
> space. The problem is that this will not help here. Simply if I
> do so and then create 100 millions of pages on one run I will
> still get to the same issue just it'll take a lot longer.
You won't reach 100 time the memory taken by downloading a 100
million pages after you download a million pages. The caches have
limits and you should measure how much memory in your setting is
needed to reach that limit. Tests such as yours, after they reach the
max, will get slower because of the DB access following the cache
eviction; they are not very realistic under that point of view. The best way to follow this is to use JMX and get
a graph of the
cache values, you should see maximum be reached after a while. At
curriki, we have adjusted the caches to be a bit bigger and we do
reach the maximum (100'000 if I remember well, for the document
cache) after about an hour after the restart. Then things get pretty
stable with our max 8G of memory.
I'm glad that your setup is stable. I do have following cache setup here:
$ grep cache `find . -name 'xwiki.cfg'`
#-# Put a cache in front of the document store. This greatly improves
performance at the cost of memory consumption.
xwiki.store.cache=1
#-# Maximum number of documents to keep in the cache.
xwiki.store.cache.capacity=100
#-# Maximum number of documents to keep in the rendered cache
xwiki.render.cache.capacity=100
# xwiki.authentication.ldap.groupcache_expiration=21600
xwiki.plugin.image.cache.capacity=30
this is XWiki 6.0.1 on top of Tomcat 7.0.53 using PostgreSQL 9.3/64bit. The
JVM is 1.7.0_07 everything running on Solaris 11.1. When I give 2GB RAM to
catalina by:
$ cat bin/setenv.sh
CATALINA_OPTS="-server -Xms2048m -Xmx2048m -XX:MaxPermSize=196m
-Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC
-XX:MaxGCPauseMillis=100"
export CATALINA_OPTS
and then run my benchmark which is just using REST to import following
simple page:
"<?xml version=\"1.0\" encoding=\"UTF-8\"
standalone=\"yes\"?>"
++ "<page xmlns=\"http://www.xwiki.org\">"
++ "<title>Semantic XWiki Benchmark page " ++ title ++
"</title>"
++ "<syntax>xwiki/2.0</syntax>"
++ "<content>This is a benchark page."
++ "The list of properties defined for this page is:"
++ "</content>"
++ "</page>"
where title is BenchPage_<number>
Now, when I pushed that, then after around 400k pages imported I hit OOM
error. JConsole is thrown out by closed connection and even benchmark tool
complains about few 500 HTTP error codes received. The exception output on
Tomcat's console looks:
Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" Exception in thread "RMI
TCP
Connection(idle)" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread
"TxCleanupService,platform.security.authorization.cache,local"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "TxCleanupService,localization.bundle.document,local"
Exception in thread "TxCleanupService,wiki.descriptor.cache.wikiId,local"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "TxCleanupService,xwiki.store.pageexistcache,local"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "TxCleanupService,xwiki.store.pagecache,local"
java.lang.OutOfMemoryError: Java heap space
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle
WARNING: Exception or error caught in status service
java.lang.OutOfMemoryError: Java heap space
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle
WARNING: Exception or error caught in status service
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2442)
at java.lang.Class.getMethod0(Class.java:2685)
at java.lang.Class.getMethod(Class.java:1620)
at
org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.getNotificationManager(LegacyNotificationDispatcher.java:94)
at
org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.onEvent(LegacyNotificationDispatcher.java:109)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:299)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:264)
at com.xpn.xwiki.XWiki.saveDocument(XWiki.java:1323)
at com.xpn.xwiki.api.Document.saveDocument(Document.java:2299)
at com.xpn.xwiki.api.Document.save(Document.java:2202)
at com.xpn.xwiki.api.Document.save(Document.java:2196)
at
org.xwiki.rest.internal.resources.pages.ModifiablePageResource.putPage(ModifiablePageResource.java:67)
at
org.xwiki.rest.internal.resources.pages.PageResourceImpl.putPage(PageResourceImpl.java:62)
at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.restlet.ext.jaxrs.internal.wrappers.AbstractMethodWrapper.internalInvoke(AbstractMethodWrapper.java:171)
at
org.restlet.ext.jaxrs.internal.wrappers.ResourceMethod.invoke(ResourceMethod.java:291)
at
org.restlet.ext.jaxrs.JaxRsRestlet.invokeMethod(JaxRsRestlet.java:1043)
at org.restlet.ext.jaxrs.JaxRsRestlet.handle(JaxRsRestlet.java:792)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Router.doHandle(Router.java:500)
at org.restlet.routing.Router.handle(Router.java:740)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle
WARNING: Exception or error caught in status service
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2442)
at java.lang.Class.getMethod0(Class.java:2685)
at java.lang.Class.getMethod(Class.java:1620)
at
org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.getNotificationManager(LegacyNotificationDispatcher.java:94)
at
org.xwiki.legacy.internal.oldcore.notification.LegacyNotificationDispatcher.onEvent(LegacyNotificationDispatcher.java:109)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:299)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:264)
at com.xpn.xwiki.XWiki.saveDocument(XWiki.java:1323)
at com.xpn.xwiki.api.Document.saveDocument(Document.java:2299)
at com.xpn.xwiki.api.Document.save(Document.java:2202)
at com.xpn.xwiki.api.Document.save(Document.java:2196)
at
org.xwiki.rest.internal.resources.pages.ModifiablePageResource.putPage(ModifiablePageResource.java:67)
at
org.xwiki.rest.internal.resources.pages.PageResourceImpl.putPage(PageResourceImpl.java:62)
at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.restlet.ext.jaxrs.internal.wrappers.AbstractMethodWrapper.internalInvoke(AbstractMethodWrapper.java:171)
at
org.restlet.ext.jaxrs.internal.wrappers.ResourceMethod.invoke(ResourceMethod.java:291)
at
org.restlet.ext.jaxrs.JaxRsRestlet.invokeMethod(JaxRsRestlet.java:1043)
at org.restlet.ext.jaxrs.JaxRsRestlet.handle(JaxRsRestlet.java:792)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Router.doHandle(Router.java:500)
at org.restlet.routing.Router.handle(Router.java:740)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
at org.restlet.routing.Filter.handle(Filter.java:206)
at org.restlet.routing.Filter.doHandle(Filter.java:159)
May 30, 2014 4:12:18 PM org.restlet.engine.application.StatusFilter doHandle
WARNING: Exception or error caught in status service
java.lang.OutOfMemoryError: Java heap space
2014-05-30 16:12:18,851 [DefaultQuartzScheduler_Worker-7] ERROR
o.q.c.JobRunShell - Job
DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an unhandled
Exception:
java.lang.OutOfMemoryError: Java heap space
2014-05-30 16:12:18,856 [XWiki Solr index thread] WARN
o.h.u.JDBCExceptionReporter - SQL Error: 0, SQLState: 08001
2014-05-30 16:12:18,856 [XWiki Solr index thread] ERROR
o.h.u.JDBCExceptionReporter - The connection attempt failed.
2014-05-30 16:12:18,858 [DefaultQuartzScheduler_Worker-7] ERROR
o.q.c.ErrorLogger - Job
(DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception.
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
~[quartz-1.6.5.jar:1.6.5]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
[quartz-1.6.5.jar:1.6.5]
Caused by: java.lang.OutOfMemoryError: Java heap space
2014-05-30 16:12:18,859 [DefaultQuartzScheduler_Worker-7] ERROR
c.x.x.p.s.StatusListener - Job
(DEFAULT.xwiki:Scheduler.WatchListHourlyNotifier_0 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception.
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
~[quartz-1.6.5.jar:1.6.5]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
[quartz-1.6.5.jar:1.6.5]
Caused by: java.lang.OutOfMemoryError: Java heap space
Funny thing is that before the error JConsole does not show any signs that
the server was allocating a lot of memory. Probably this should be quick
spike in mem consumption or otherwise I don't understand at all why this
happen.
Now, the question is: I do have or I think I do have very limited cache
setup here. I've increase RAM side to 2GB which is over the recommended size
by xwiki.org itself -- they warn about size bigger than 1GB due to slowness
of GC then. Anyway, 2GB, limited size of cache and yet I hit OOM. Do you
think I shall increase RAM limit even further?
Thanks!
Karel
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs