[xwiki-devs] Preliminary memory analysis of xwiki.org dump of 2013-12-01
Hi devs, Yesterday xwiki.org crashed and I had configured it to take a heap dump. I’ve done a quick analysis that I’m sharing here (I’ll continue to analyse): Memory retained: 1GB Main contenders: 1) Document cache: 178MB 2) Lucene WeightedSpanTermExtractor: 166MB 3) IRCBot Threads: 165MB 4) Velocity RuntimeInstance: 38MB 5) SOLR LRUCache (Lucene Document): 38MB 6) EM DefaultCoreExtensionRepository: 38MB 7) NamespaceURLClassLoader: 23MB I’ve started analyzing some below. 1 - Document Cache Analysis ======================= * There are 3552 XWikiDocument in memory for 195MB * The document cache size is 2000 on xwiki.org * Large documents (such as Test Reports) take 6MB each (XDOM caching) * So if we had only large documents in the wiki the cache would need 2000*6MB = 12TO * I don’t think this cache is memory aware, meaning it doesn’t free its entries when memory is low * 178MB for 2000 docs means average of 89K per document. Huge variation between docs with big content and docs with no or small content This means that when memory is low on xwiki.org it should be enough to call a few pages with some large content to get an OOM. 4 ideas to explore: Idea 1: Use a cache that evicts entries when some max threshold is reached ** Infinispan doesn’t support this yet: https://issues.jboss.org/browse/ISPN-863 and https://community.jboss.org/thread/165951?start=0&tstart=0 ** Guava seems to support size-based eviction with the ability to pass a weight function: http://code.google.com/p/guava-libraries/wiki/CachesExplained#Size-based_Evi... Idea 2: Another idea is a usage of a distributed cache such as memcached or elasticsearch. I wonder if the overhead of the network communication is too high to make it interesting vs not caching the XDOM and rendering it every time it’s needed. Idea 3: Try to reduce even more how the XDOM is stored in memory Idea 4: Don’t cache the XDOM and render every time and use a title cache for titles. Also do that for getting sections. I think they are the 2 main uses cases for getting the XDOM. As a short term action, I’d recommend to immediately reduce the document cache size from 2000 to 1000 on wiki.org or double the heap memory. 2 - Lucene WeightedSpanTermExtractor Analysis ===================================== I’m not sure what this is about yet but it looks strange. * There is 166MB stored in the Map<String,AtomicReaderContext> of WeightedSpanTermExtractor. * That map contains 192 instances * Example of map items: “doccontent_pt” (2.4MB), “title_ru” (1.8MB), “title_ro” (1.8MB), etc Any idea Marius? 3 - IRCBot Analysis =============== * We use 3 IRCBot threads. They take 55MB each! * The 55MB is taken by the ExecutionContext * More precisely the 55MB are held in 77371 org.apache.velocity.runtime.parser.node.Node[] objects I need to understand more why it’s so large since it doesn’t look normal. I also wonder if it keeps increasing or not. 5 - SOLR LRUCache Analysis ======================= * It’s map of 512 entries (Lucene Document objects). 512 is the cache size. * Entries are instances of DocSlice Looks ok and normal. 6 - EM DefaultCoreExtensionRepository Analysis ====================================== * 38MB in "Map<String, DefaultCoreExtension> extensions” * 33MB in org.codehaus.plexus.util.xml.Xpp3Dom instances (44844 instances) which I guess corresponds to the pom.xml of all our core extensions mostly. Looks normal even though 33MB is quite a lot. 7 - NamespaceURLClassLoader Analysis ================================ * 23MB in org.eclipse.jgit.storage.file.WindowCache * So this seems related to the XWiki Git Module used by the GitHubStats application installed on dev.xwiki.org This looks ok and normal according to http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/... Thanks -Vincent
Hi Le dimanche 1 décembre 2013, [email protected] a écrit :
Hi devs,
Yesterday xwiki.org crashed and I had configured it to take a heap dump. I’ve done a quick analysis that I’m sharing here (I’ll continue to analyse):
Memory retained: 1GB
Did it crash OOM ?
Main contenders:
1) Document cache: 178MB
This does not look that big too me
2) Lucene WeightedSpanTermExtractor: 166MB
This however seems important
3) IRCBot Threads: 165MB
This too
4) Velocity RuntimeInstance: 38MB 5) SOLR LRUCache (Lucene Document): 38MB 6) EM DefaultCoreExtensionRepository: 38MB 7) NamespaceURLClassLoader: 23MB
I’ve started analyzing some below.
1 - Document Cache Analysis =======================
* There are 3552 XWikiDocument in memory for 195MB * The document cache size is 2000 on xwiki.org * Large documents (such as Test Reports) take 6MB each (XDOM caching) * So if we had only large documents in the wiki the cache would need 2000*6MB = 12TO * I don’t think this cache is memory aware, meaning it doesn’t free its entries when memory is low * 178MB for 2000 docs means average of 89K per document. Huge variation between docs with big content and docs with no or small content
This means that when memory is low on xwiki.org it should be enough to call a few pages with some large content to get an OOM.
4 ideas to explore:
Idea 1: Use a cache that evicts entries when some max threshold is reached ** Infinispan doesn’t support this yet: https://issues.jboss.org/browse/ISPN-863 and https://community.jboss.org/thread/165951?start=0&tstart=0 ** Guava seems to support size-based eviction with the ability to pass a weight function: http://code.google.com/p/guava-libraries/wiki/CachesExplained#Size-based_Evi...
Clearly using a cache that is size aware would be much better.
Idea 2: Another idea is a usage of a distributed cache such as memcached or elasticsearch. I wonder if the overhead of the network communication is too high to make it interesting vs not caching the XDOM and rendering it every time it’s needed.
I think this would not per fast enough at least for some docs like the pref doc. I have done this is a google app engine experiment and a local cache was needed in addition to memcache otherwise it would be slow.
Idea 3: Try to reduce even more how the XDOM is stored in memory
Indeed this looks big. Similar to the issue of caching big attachments. Initially I used a soft reference but real life experience showed that the JVM would not drop them and OOM was still possible. This would be a good solution if it works also for XDOM cache
Idea 4: Don’t cache the XDOM and render every time and use a title cache for titles. Also do that for getting sections. I think they are the 2 main uses cases for getting the XDOM.
As a short term action, I’d recommend to immediately reduce the document cache size from 2000 to 1000 on wiki.org or double the heap memory.
Do both :)
2 - Lucene WeightedSpanTermExtractor Analysis =====================================
I’m not sure what this is about yet but it looks strange.
* There is 166MB stored in the Map<String,AtomicReaderContext> of WeightedSpanTermExtractor. * That map contains 192 instances * Example of map items: “doccontent_pt” (2.4MB), “title_ru” (1.8MB), “title_ro” (1.8MB), etc
Any idea Marius?
3 - IRCBot Analysis ===============
* We use 3 IRCBot threads. They take 55MB each! * The 55MB is taken by the ExecutionContext * More precisely the 55MB are held in 77371 org.apache.velocity.runtime.parser.node.Node[] objects
I need to understand more why it’s so large since it doesn’t look normal.
I also wonder if it keeps increasing or not.
You should do a dump after a while clearing the cache before, then do the same a couple of hours later and compare Ludovic
5 - SOLR LRUCache Analysis =======================
* It’s map of 512 entries (Lucene Document objects). 512 is the cache size. * Entries are instances of DocSlice
Looks ok and normal.
6 - EM DefaultCoreExtensionRepository Analysis ======================================
* 38MB in "Map<String, DefaultCoreExtension> extensions” * 33MB in org.codehaus.plexus.util.xml.Xpp3Dom instances (44844 instances) which I guess corresponds to the pom.xml of all our core extensions mostly.
Looks normal even though 33MB is quite a lot.
7 - NamespaceURLClassLoader Analysis ================================
* 23MB in org.eclipse.jgit.storage.file.WindowCache * So this seems related to the XWiki Git Module used by the GitHubStats application installed on dev.xwiki.org
This looks ok and normal according to http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/...
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] <javascript:;> http://lists.xwiki.org/mailman/listinfo/devs
-- Sent from Mobile
On Sun, Dec 1, 2013 at 9:01 PM, Ludovic Dubost <[email protected]> wrote:
Hi
Le dimanche 1 décembre 2013, [email protected] a écrit :
Hi devs,
Yesterday xwiki.org crashed and I had configured it to take a heap dump. I’ve done a quick analysis that I’m sharing here (I’ll continue to analyse):
Memory retained: 1GB
Did it crash OOM ?
Main contenders:
1) Document cache: 178MB
This does not look that big too me
2) Lucene WeightedSpanTermExtractor: 166MB
This however seems important
3) IRCBot Threads: 165MB
This too
4) Velocity RuntimeInstance: 38MB 5) SOLR LRUCache (Lucene Document): 38MB 6) EM DefaultCoreExtensionRepository: 38MB 7) NamespaceURLClassLoader: 23MB
I’ve started analyzing some below.
1 - Document Cache Analysis =======================
* There are 3552 XWikiDocument in memory for 195MB * The document cache size is 2000 on xwiki.org * Large documents (such as Test Reports) take 6MB each (XDOM caching) * So if we had only large documents in the wiki the cache would need 2000*6MB = 12TO * I don’t think this cache is memory aware, meaning it doesn’t free its entries when memory is low * 178MB for 2000 docs means average of 89K per document. Huge variation between docs with big content and docs with no or small content
This means that when memory is low on xwiki.org it should be enough to call a few pages with some large content to get an OOM.
4 ideas to explore:
Idea 1: Use a cache that evicts entries when some max threshold is reached ** Infinispan doesn’t support this yet: https://issues.jboss.org/browse/ISPN-863 and https://community.jboss.org/thread/165951?start=0&tstart=0 ** Guava seems to support size-based eviction with the ability to pass a weight function: http://code.google.com/p/guava-libraries/wiki/CachesExplained#Size-based_Evi...
Clearly using a cache that is size aware would be much better.
Idea 2: Another idea is a usage of a distributed cache such as memcached or elasticsearch. I wonder if the overhead of the network communication is too high to make it interesting vs not caching the XDOM and rendering it every time it’s needed.
I think this would not per fast enough at least for some docs like the pref doc. I have done this is a google app engine experiment and a local cache was needed in addition to memcache otherwise it would be slow.
Idea 3: Try to reduce even more how the XDOM is stored in memory
Indeed this looks big. Similar to the issue of caching big attachments. Initially I used a soft reference but real life experience showed that the JVM would not drop them and OOM was still possible. This would be a good solution if it works also for XDOM cache
Idea 4: Don’t cache the XDOM and render every time and use a title cache for titles. Also do that for getting sections. I think they are the 2 main uses cases for getting the XDOM.
As a short term action, I’d recommend to immediately reduce the document cache size from 2000 to 1000 on wiki.org or double the heap memory.
Do both :)
2 - Lucene WeightedSpanTermExtractor Analysis =====================================
I’m not sure what this is about yet but it looks strange.
* There is 166MB stored in the Map<String,AtomicReaderContext> of WeightedSpanTermExtractor. * That map contains 192 instances * Example of map items: “doccontent_pt” (2.4MB), “title_ru” (1.8MB), “title_ro” (1.8MB), etc
Any idea Marius?
3 - IRCBot Analysis ===============
* We use 3 IRCBot threads. They take 55MB each! * The 55MB is taken by the ExecutionContext * More precisely the 55MB are held in 77371 org.apache.velocity.runtime.parser.node.Node[] objects
I need to understand more why it’s so large since it doesn’t look normal.
I also wonder if it keeps increasing or not.
You should do a dump after a while clearing the cache before, then do the same a couple of hours later and compare
Ludovic
5 - SOLR LRUCache Analysis =======================
* It’s map of 512 entries (Lucene Document objects). 512 is the cache size. * Entries are instances of DocSlice
Looks ok and normal.
6 - EM DefaultCoreExtensionRepository Analysis ======================================
* 38MB in "Map<String, DefaultCoreExtension> extensions” * 33MB in org.codehaus.plexus.util.xml.Xpp3Dom instances (44844 instances) which I guess corresponds to the pom.xml of all our core extensions mostly.
I will look into it, the javadoc of the field that contain this kind of object in the Maven Model is "<b>Deprecated</b>. Now ignored by Maven."...
Looks normal even though 33MB is quite a lot.
7 - NamespaceURLClassLoader Analysis ================================
* 23MB in org.eclipse.jgit.storage.file.WindowCache * So this seems related to the XWiki Git Module used by the GitHubStats application installed on dev.xwiki.org
This looks ok and normal according to http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/...
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] <javascript:;> http://lists.xwiki.org/mailman/listinfo/devs
-- Sent from Mobile _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On 1 Dec 2013 at 21:01:37, Ludovic Dubost ([email protected]) wrote: Hi Le dimanche 1 décembre 2013, [email protected] a écrit :
Hi devs, Yesterday xwiki.org crashed and I had configured it to take a heap dump. I’ve done a quick analysis that I’m sharing here (I’ll continue to analyse): Memory retained: 1GB
Did it crash OOM ? Yes (see https://www.evernote.com/shard/s119/sh/f6b7f17b-cb5e-4dd4-9c41-0ef3e6561c0e/...) but the logs show an issue with the communication with MySQL: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997) I don’t know if this caused the memory to grow for some reason but it was probably the real cause of the crash. That said memory was quite high and when you sum you the computations I did we were close to using the full memory anyway.
Main contenders: 1) Document cache: 178MB
This does not look that big too me
2) Lucene WeightedSpanTermExtractor: 166MB
This however seems important
3) IRCBot Threads: 165MB
This too
4) Velocity RuntimeInstance: 38MB 5) SOLR LRUCache (Lucene Document): 38MB 6) EM DefaultCoreExtensionRepository: 38MB 7) NamespaceURLClassLoader: 23MB I’ve started analyzing some below. 1 - Document Cache Analysis ======================= * There are 3552 XWikiDocument in memory for 195MB * The document cache size is 2000 on xwiki.org * Large documents (such as Test Reports) take 6MB each (XDOM caching) * So if we had only large documents in the wiki the cache would need 2000*6MB = 12TO * I don’t think this cache is memory aware, meaning it doesn’t free its entries when memory is low * 178MB for 2000 docs means average of 89K per document. Huge variation between docs with big content and docs with no or small content This means that when memory is low on xwiki.org it should be enough to call a few pages with some large content to get an OOM. 4 ideas to explore: Idea 1: Use a cache that evicts entries when some max threshold is reached ** Infinispan doesn’t support this yet: https://issues.jboss.org/browse/ISPN-863 and https://community.jboss.org/thread/165951?start=0&tstart=0 ** Guava seems to support size-based eviction with the ability to pass a weight function: http://code.google.com/p/guava-libraries/wiki/CachesExplained#Size-based_Evi...
Clearly using a cache that is size aware would be much better.
Idea 2: Another idea is a usage of a distributed cache such as memcached or elasticsearch. I wonder if the overhead of the network communication is too high to make it interesting vs not caching the XDOM and rendering it every time it’s needed.
I think this would not per fast enough at least for some docs like the pref doc. I have done this is a google app engine experiment and a local cache I didn’t suggest to not cache XWikiDocuments but to not cache the XDOM. Since XWikiPreferences has little or no content, there’s no rendering to do for it in term of XDOM, except when you view the page which doesn’t happen often. Now not caching the XDOM is probably not going to work, I agree, because the cached XDOM is checked at very display of pages. was needed in addition to memcache otherwise it would be slow. And memcache or elastcisearch wouldn’t work either since you’d need to serialize/unserialize Java Objects which would be too slow.
Idea 3: Try to reduce even more how the XDOM is stored in memory
Indeed this looks big. Similar to the issue of caching big attachments. Initially I used a soft reference but real life experience showed that the JVM would not drop them and OOM was still possible. This would be a good solution if it works also for XDOM cache I believe optimizing the XDOM memory footage is the only real solution but we’ve already done a first pass on that and I don’t know much more we can win. We should still have a second look and be creative ;) For example, I’m sure we can find better ways for storing SpaceBlock, SymbolBlock, etc.
Idea 4: Don’t cache the XDOM and render every time and use a title cache for titles. Also do that for getting sections. I think they are the 2 main uses cases for getting the XDOM. As a short term action, I’d recommend to immediately reduce the document cache size from 2000 to 1000 on wiki.org or double the heap memory.
Do both :)
2 - Lucene WeightedSpanTermExtractor Analysis ===================================== I’m not sure what this is about yet but it looks strange. * There is 166MB stored in the Map<String,AtomicReaderContext> of WeightedSpanTermExtractor. * That map contains 192 instances * Example of map items: “doccontent_pt” (2.4MB), “title_ru” (1.8MB), “title_ro” (1.8MB), etc Any idea Marius? 3 - IRCBot Analysis =============== * We use 3 IRCBot threads. They take 55MB each! * The 55MB is taken by the ExecutionContext * More precisely the 55MB are held in 77371 org.apache.velocity.runtime.parser.node.Node[] objects I need to understand more why it’s so large since it doesn’t look normal. I also wonder if it keeps increasing or not.
You should do a dump after a while clearing the cache before, then do the same a couple of hours later and compare Thanks -Vincent Ludovic
5 - SOLR LRUCache Analysis ======================= * It’s map of 512 entries (Lucene Document objects). 512 is the cache size. * Entries are instances of DocSlice Looks ok and normal. 6 - EM DefaultCoreExtensionRepository Analysis ====================================== * 38MB in "Map<String, DefaultCoreExtension> extensions” * 33MB in org.codehaus.plexus.util.xml.Xpp3Dom instances (44844 instances) which I guess corresponds to the pom.xml of all our core extensions mostly. Looks normal even though 33MB is quite a lot. 7 - NamespaceURLClassLoader Analysis ================================ * 23MB in org.eclipse.jgit.storage.file.WindowCache * So this seems related to the XWiki Git Module used by the GitHubStats application installed on dev.xwiki.org This looks ok and normal according to http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/... Thanks -Vincent
On 2 Dec 2013 at 10:26:33, [email protected] ([email protected](mailto://[email protected])) wrote: [snip]
1 - Document Cache Analysis =======================
* There are 3552 XWikiDocument in memory for 195MB * The document cache size is 2000 on xwiki.org * Large documents (such as Test Reports) take 6MB each (XDOM caching) * So if we had only large documents in the wiki the cache would need 2000*6MB = 12TO * I don’t think this cache is memory aware, meaning it doesn’t free its entries when memory is low * 178MB for 2000 docs means average of 89K per document. Huge variation between docs with big content and docs with no or small content
This means that when memory is low on xwiki.org it should be enough to call a few pages with some large content to get an OOM.
4 ideas to explore:
[snip]
I believe optimizing the XDOM memory footage is the only real solution but we’ve already done a first pass on that and I don’t know much more we can win. We should still have a second look and be creative ;)
For example, I’m sure we can find better ways for storing SpaceBlock, SymbolBlock, etc.
FTR: * SpaceBlock (710035 instances): 22MB * SpecialSymbolBlock (371857 instances): 14MB * WordBlock (796535 instances): 79MB * NewLinesBlock (25640): 820K For example for storing just the letter “y” in a WordBlock, it takes 624 bytes vs 56 bytes for the “y” in a String. That’s because for we have parent refs, previous/next sibling refs, children blocks, param blocks to save too. Said differently, our current XDOM structure is not made to be memory efficient. [snip] Thanks -Vincent PS: The reason we use different instances for SpaceBlock and SpecialSymbolBlock ATM: /** * A new line block. Note that we don't make NewLineBlock a singleton since that would cause problems when using * Block APIs to manipulate a tree of blocks (for example to find the position of a new line block in a list using * {@link java.util.List#indexOf(Object)} wich would always return the first new line block). * @since 2.6RC1 */
participants (3)
-
Ludovic Dubost -
Thomas Mortagne -
vincent＠massol.net