[xwiki-devs] Preliminary memory tests
Hi, I've played a bit with Yourkit and a default XE to see the memory requirements. Here are some preliminary results (more tests needed): * After home page loaded: 38MB non-heap, 6377 classes * After code macro execution: 38 --> 52MB non heap, 8629 classes <--- a lot * After groovy macro execution: 52 --> 56MB, 9920 classes * After treeview + opened top level nodes: 56MB --> 63MB, 10960 classes <-- a lot * After Import: 63MB --> 65MB, 13036 classes * After Export: no change * After navigating a bit everywhere at random: heap memory needed: 155MB (used: 132MB), non heap: 65MB, classes: 13078 * After adding attachment of 1.1MB: almost no change * After deletion of 1.1MB attachment: used memory from 98MB to 132MB !! But still within the 155MB allocated heap memory Conclusion so far: ============== * We need around < 80MB non heap memory and < 200MB heap memory for the default XE pages and if the user manipulates attachments less than 1MB. * The default permgen size is 64MB so we're just above it (65MB). That's why some users have reported permgen issues I believe. Jython is eating a lot of it (14MB) * After adding a 7.3MB attachment: heap memory max from 155 to 261 (=>110MB!) * After removing it, need 297MB heap (even more) Second conclusion: ============== * Apart from attachments the memory needs seem reasonable IMO (< 255MB). * I'd still need to make sure I test all features of the wiki to be sure What more should I test? What should we do to reduce our permgen needs? Should we reduce it? Thanks -Vincent
On Oct 13, 2009, at 7:03 PM, Vincent Massol wrote:
Hi,
I've played a bit with Yourkit and a default XE to see the memory requirements. Here are some preliminary results (more tests needed):
* After home page loaded: 38MB non-heap, 6377 classes * After code macro execution: 38 --> 52MB non heap, 8629 classes <--- a lot * After groovy macro execution: 52 --> 56MB, 9920 classes * After treeview + opened top level nodes: 56MB --> 63MB, 10960 classes <-- a lot * After Import: 63MB --> 65MB, 13036 classes * After Export: no change * After navigating a bit everywhere at random: heap memory needed: 155MB (used: 132MB), non heap: 65MB, classes: 13078 * After adding attachment of 1.1MB: almost no change * After deletion of 1.1MB attachment: used memory from 98MB to 132MB !! But still within the 155MB allocated heap memory
Conclusion so far: ============== * We need around < 80MB non heap memory and < 200MB heap memory for the default XE pages and if the user manipulates attachments less than 1MB. * The default permgen size is 64MB so we're just above it (65MB). That's why some users have reported permgen issues I believe. Jython is eating a lot of it (14MB)
* After adding a 7.3MB attachment: heap memory max from 155 to 261 (=>110MB!) * After removing it, need 297MB heap (even more)
Second conclusion: ============== * Apart from attachments the memory needs seem reasonable IMO (< 255MB). * I'd still need to make sure I test all features of the wiki to be sure
What more should I test? What should we do to reduce our permgen needs? Should we reduce it?
Thanks -Vincent
BTW a good article on permgen: http://ombhakar.blogspot.com/2007/05/permgen-outofmemory.html Thanks -Vincent
On Oct 13, 2009, at 7:03 PM, Vincent Massol wrote:
Hi,
I've played a bit with Yourkit and a default XE to see the memory requirements. Here are some preliminary results (more tests needed):
* After home page loaded: 38MB non-heap, 6377 classes * After code macro execution: 38 --> 52MB non heap, 8629 classes <--- a lot * After groovy macro execution: 52 --> 56MB, 9920 classes * After treeview + opened top level nodes: 56MB --> 63MB, 10960 classes <-- a lot * After Import: 63MB --> 65MB, 13036 classes * After Export: no change * After navigating a bit everywhere at random: heap memory needed: 155MB (used: 132MB), non heap: 65MB, classes: 13078 * After adding attachment of 1.1MB: almost no change * After deletion of 1.1MB attachment: used memory from 98MB to 132MB !! But still within the 155MB allocated heap memory
Conclusion so far: ============== * We need around < 80MB non heap memory and < 200MB heap memory for the default XE pages and if the user manipulates attachments less than 1MB. * The default permgen size is 64MB so we're just above it (65MB). That's why some users have reported permgen issues I believe. Jython is eating a lot of it (14MB)
* After adding a 7.3MB attachment: heap memory max from 155 to 261 (=>110MB!) * After removing it, need 297MB heap (even more)
Second conclusion: ============== * Apart from attachments the memory needs seem reasonable IMO (< 255MB). * I'd still need to make sure I test all features of the wiki to be sure
* Obviously: We must absolutely fix the attachment memory issue BTW I had a max memory of 300MB and I got a OOM error so it's possible that the 7.3MB attachment actually needed even more than the 297MB used. Thanks -Vincent
What more should I test? What should we do to reduce our permgen needs? Should we reduce it?
Thanks -Vincent
On Tue, Oct 13, 2009 at 10:03 AM, Vincent Massol <[email protected]> wrote:
... * After deletion of 1.1MB attachment: used memory from 98MB to 132MB !! But still within the 155MB allocated heap memory ... * After adding a 7.3MB attachment: heap memory max from 155 to 261 (=>110MB!) * After removing it, need 297MB heap (even more)
I think a good fix for this would be to turn off attachment versioning and recycle-bin by default. In http://lists.xwiki.org/pipermail/users/2009-March/015435.html and http://www.mail-archive.com/[email protected]/msg07136.html I suggest using: #-# Whether the attachment recycle bin feature is activated or not storage.attachment.recyclebin=0 #-# Whether the attachment versioning feature is activated or not xwiki.store.attachment.versioning=0 #-# Whether the attachments should also be rolled back when a document is reverted. xwiki.store.rollbackattachmentwithdocuments=0 Another suggestion:
some kind of heuristic could be employed, as the default, e.g. the recylcle-bin is used to hold deleted attachments, except when the attachments exceed a certain size, say 1Mb. In that case the user is warned of permanent deletion and can "ok" it away.
Finally, http://www.mail-archive.com/[email protected]/msg10311.html has some additional info re jython, classloading, etc: http://blog.headius.com/2009/01/my-favorite-hotspot-jvm-flags.html
-XX:OnOutOfMemoryError="mail -s 'OOM on `hostname` at `date`' [email protected] <<< ''" as a way to send out email when there's an OutOfMemoryError. Poor-man's monitoring!
-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintCompilation prints out the name of each Java method Hotspot
decides to JIT compile. The list will usually show a bunch of core Java class methods initially, and then turn to methods in your application. In JRuby, it eventually starts to show Ruby methods as well.
-XX:+PrintGCDetails includes the data from -verbose:gc but also adds
information about the size of the new generation and more accurate timings.
-XX:+TraceClassLoading and -XX:+TraceClassUnloading print information class
loads and unloads. Useful for investigating if you have a class leak or if old classes (like JITed Ruby methods in JRuby) are getting collected or not.
Trying this out on Xwiki 2.0M2 gives some interesting, voluminous results: http://nielsmayer.com/xwiki-catalina-with-PrintGCDetails-PrintCompilation-Tr... -- Niels http://nielsmayer.com
participants (2)
-
Niels Mayer -
Vincent Massol