[xwiki-devs] Measuring cache memory usage

Hi developers, A while ago I was looking for some ways to track how much memory is used by our internal cache and was not able to find anything. I've tried it again and this time I found the following code: http://www.javamex.com/classmexer/ This requires a simple instrumentation to work, but I was able to get some results out of it to measure the size of our documents in cache. You can see the result on a personal server: Measuring one page: http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage Measuring all pages in cache: http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage?page=all The first results I can see, is that with no surprise the items taking most memory are: - attachment content - attachment archive - archive What I was able to see is that as expected these fields won't consume memory until we are asking for the data. And after a while, the memory is indeed discarded for these fields, so the usage of SoftReferences for them seem to work. Now what I can see is that the attachment archive can be very very very costly in memory. Also it does not seem clear how the memory from these fields is garbage collected (a GC did not recover it). With some experience of massive loading of attachments that lead to OutOfMemory errors in the server, I do suspect that the SoftReferences are not necessarly discarded fast enough to avoid the OutOfMemory. I also believe that a search engine that is walking all our pages including our archive pages can genearate important memory usage that could lead to problems. But this is only an intuition that needs to be proved. I believe we need to run some testing under stress to see if the cache and memory usage do behave properly and if the cache(s) are never able to go over the memory usage. We also should try the classmexer on servers that are heavily used an be able to look at the memory usage and see if we are "controling" it. I'm not 100% sure how intrusive the "instrumentation" module is but I believe it's quite light. We could try it on xwiki.org or on myxwiki.org. WDYT ? Ludovic -- Ludovic Dubost Blog:http://blog.ludovic.org/ XWiki:http://www.xwiki.com Skype: ldubost GTalk: ldubost