Hi all, For a particular project, I have a plugin that reads data from an external database using some JDBC queries. For each row of my JDBC ResultSet, the plugin creates an XWiki document, the associated objects, and fill these objects. All is running fine except that I have around 50.000 objects to import :-D. Then I get a Java Heap Space exception. I re-re-re-re-re-read my source code, I don't keep any reference to the documents created. Every 200 document creation, I do the following calls : context.getWiki().flushCache(); context.getWiki().getStore().cleanUp(context); System.gc(); Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception. Does anybody has an idea for me ? Who thinks that I could be a XWiki memory leak ? Thanks, -- Arnaud Thimel Société Code Lutin - http://www.codelutin.com
Hi Arnaud, On Apr 04, Arnaud Thimel wrote :
All is running fine except that I have around 50.000 objects to import :-D. Then I get a Java Heap Space exception. [...]
Does anybody has an idea for me ? Who thinks that I could be a XWiki memory leak ?
Have you tried to do a heap profiling ? It will help you determining where is your memory going, and might reveal possible memory leaks. Kind regards, Pablo PS: You can have a look at http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
Hi Arnaud, I faced this problem before. The importation of huge data need the configuration of java parameters of the heap. I was not able to import and create 300000 documents - without object - until I changed the following parameters : -Xms512m -Xmx521m (you could add or change them in startup.bat in windows or catalina.sh on linux). These parameters depends on your machine. Then you can increase (-->1024) or decrease them (1024). Hope this will solve your problem. Cheers. ----------- Youcef Bey Quoting Arnaud Thimel <[email protected]>:
Hi all,
For a particular project, I have a plugin that reads data from an external database using some JDBC queries. For each row of my JDBC ResultSet, the plugin creates an XWiki document, the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to import :-D. Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the documents created. Every 200 document creation, I do the following calls :
context.getWiki().flushCache(); context.getWiki().getStore().cleanUp(context); System.gc();
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
Does anybody has an idea for me ? Who thinks that I could be a XWiki memory leak ?
Thanks,
-- Arnaud Thimel Société Code Lutin - http://www.codelutin.com
------------------------------------------------- envoyé via Webmail/IMAG !
On Apr 5, 2007, at 7:50 PM, [email protected] wrote:
Hi Arnaud,
I faced this problem before.
The importation of huge data need the configuration of java parameters of the heap. I was not able to import and create 300000 documents - without object - until I changed the following parameters :
-Xms512m -Xmx521m (you could add or change them in startup.bat in windows or catalina.sh on linux).
These parameters depends on your machine. Then you can increase (--
1024) or decrease them (1024).
Hope this will solve your problem.
if anyone has the time, it would be nice to use a profile to understand why this is required... Thanks -Vincent
Quoting Arnaud Thimel <[email protected]>:
Hi all,
For a particular project, I have a plugin that reads data from an external database using some JDBC queries. For each row of my JDBC ResultSet, the plugin creates an XWiki document, the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to import :-D. Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the documents created. Every 200 document creation, I do the following calls :
context.getWiki().flushCache(); context.getWiki().getStore().cleanUp(context); System.gc();
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
Does anybody has an idea for me ? Who thinks that I could be a XWiki memory leak ?
Thanks,
-- Arnaud Thimel Société Code Lutin - http://www.codelutin.com
------------------------------------------------- envoyé via Webmail/IMAG !
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws
(Sorry for the mistakes in my last email) Just for correction : -Xms512m (-Xmx521m) --> -Xmx512m Possible values : 64 and 1024. Best wishes ------------- Youcef Bey Vincent Massol a écrit :
On Apr 5, 2007, at 7:50 PM, [email protected] wrote:
Hi Arnaud,
I faced this problem before.
The importation of huge data need the configuration of java parameters of the heap. I was not able to import and create 300000 documents - without object - until I changed the following parameters :
-Xms512m -Xmx521m (you could add or change them in startup.bat in windows or catalina.sh on linux).
These parameters depends on your machine. Then you can increase (-->1024) or decrease them (1024).
Hope this will solve your problem.
if anyone has the time, it would be nice to use a profile to understand why this is required...
Thanks -Vincent
Quoting Arnaud Thimel <[email protected]>:
Hi all,
For a particular project, I have a plugin that reads data from an external database using some JDBC queries. For each row of my JDBC ResultSet, the plugin creates an XWiki document, the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to import :-D. Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the documents created. Every 200 document creation, I do the following calls :
context.getWiki().flushCache(); context.getWiki().getStore().cleanUp(context); System.gc();
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
Does anybody has an idea for me ? Who thinks that I could be a XWiki memory leak ?
Thanks,
--Arnaud Thimel Société Code Lutin - http://www.codelutin.com
------------------------------------------------- envoyé via Webmail/IMAG !
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
------------------------------------------------------------------------
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
-- Youcef Bey Graduate School of Education The University of Tokyo, Mail address : [email protected] Home page : http://panflute.p.u-tokyo.ac.jp/~bey/ On study leave GETALP team GETALP, LIG Lab., IMAG-Campus, BP 53 385, rue de la bibliotheque 38041 Grenoble Cedex 9 France
Arnaud Thimel a écrit :
Hi all,
Hi Arnaud,
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
I don't know if it's related to your problem, but I remember that I had to flush some caches in XWikiContext (classCache, archiveCache) when I was using the same XWikiContext to create a heap of pages. As you can see in the code these two caches are always used to add some objects in them, nether to remove them. The two caches are not accessible by default, so you will have to add some code in XWikiContext to flush them. HTH, Seb.
Hi all, I kept searching about my problem. @Pablo I tried to use profiling with Hprof and jmap+jhat. But the information I got, didn't allow me to solve my problem. @Youcef Sorry, I forgot to mention that I was already able to import all my DB, but I need to set the maximum heap size to more than 1.5Go !!! @Sébastien You found the thing I was missing :) When I flush the xwiki caches, I now also flush the "classCache" and "archiveCache" from the XWikiContext. I can now import my thousands of objects using only the default heap size (64Mo). Anyway, I don't know how, but XWiki could be improved by increasing the way we use these caches. Thanks all, and best regards. Arnaud. Sébastien Gaïde wrote:
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
I don't know if it's related to your problem, but I remember that I had to flush some caches in XWikiContext (classCache, archiveCache) when I was using the same XWikiContext to create a heap of pages. As you can see in the code these two caches are always used to add some objects in them, nether to remove them.
The two caches are not accessible by default, so you will have to add some code in XWikiContext to flush them.
HTH,
Seb.
-- Arnaud Thimel Société Code Lutin - http://www.codelutin.com
It might be a misconfiguration somewhere. Somebody should look at the way we use these caches. On 4/10/07, Arnaud Thimel <[email protected]> wrote:
Hi all,
I kept searching about my problem.
@Pablo I tried to use profiling with Hprof and jmap+jhat. But the information I got, didn't allow me to solve my problem.
@Youcef Sorry, I forgot to mention that I was already able to import all my DB, but I need to set the maximum heap size to more than 1.5Go !!!
@Sébastien You found the thing I was missing :) When I flush the xwiki caches, I now also flush the "classCache" and "archiveCache" from the XWikiContext. I can now import my thousands of objects using only the default heap size (64Mo).
Anyway, I don't know how, but XWiki could be improved by increasing the way we use these caches.
Thanks all, and best regards.
Arnaud.
Arnaud Thimel a écrit :
Hi all,
I kept searching about my problem.
@Pablo I tried to use profiling with Hprof and jmap+jhat. But the information I got, didn't allow me to solve my problem.
@Youcef Sorry, I forgot to mention that I was already able to import all my DB, but I need to set the maximum heap size to more than 1.5Go !!!
@Sébastien You found the thing I was missing :) When I flush the xwiki caches, I now also flush the "classCache" and "archiveCache" from the XWikiContext. I can now import my thousands of objects using only the default heap size (64Mo).
Anyway, I don't know how, but XWiki could be improved by increasing the way we use these caches.
Thanks all, and best regards.
Arnaud.
Sébastien Gaïde wrote:
Anyway, the amount of memory used doesn't go down after flushing the caches. And I still get my Heap Space exception.
I don't know if it's related to your problem, but I remember that I had to flush some caches in XWikiContext (classCache, archiveCache) when I was using the same XWikiContext to create a heap of pages. As you can see in the code these two caches are always used to add some objects in them, nether to remove them. Right.. the XWikiContext with the classCache and archiveCache (which are needed) is not made for being use for too long without cleaning up.. Maybe there should be some automated cleaning up when it gets too big..
Ludovic
The two caches are not accessible by default, so you will have to add some code in XWikiContext to flush them.
HTH,
Seb.
------------------------------------------------------------------------
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
-- Ludovic Dubost Blog: http://www.ludovic.org/blog/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost AIM: nvludo Yahoo: ludovic
On Apr 09, Ludovic Dubost wrote :
Right.. the XWikiContext with the classCache and archiveCache (which are needed) is not made for being use for too long without cleaning up.. Maybe there should be some automated cleaning up when it gets too big..
A simple non invasive solution, would be to use a LRUMap instead of a simple HashMap. It can be set to a maximum size and will use a Least Recently Used elimination when full and adding a new object. See patch below. Index: core/src/main/java/com/xpn/xwiki/XWikiContext.java =================================================================== --- core/src/main/java/com/xpn/xwiki/XWikiContext.java (revision 2320) +++ core/src/main/java/com/xpn/xwiki/XWikiContext.java (working copy) @@ -31,6 +31,7 @@ import com.xpn.xwiki.web.*; import com.xpn.xwiki.validation.XWikiValidationStatus; import org.apache.xmlrpc.server.XmlRpcServer; +import org.apache.commons.collections.map.LRUMap; import java.net.URL; import java.util.*; @@ -65,11 +66,13 @@ private String wikiOwner; private XWikiDocument wikiServer; private int cacheDuration = 0; + private int classCacheSize = 10; + private int archiveCacheSize = 10; // Used to avoid recursive loading of documents if there are recursives usage of classes - private Map classCache = new HashMap(); + private Map classCache = new LRUMap(classCacheSize); // Used to avoir reloading archives in the same request - private Map archiveCache = new HashMap(); + private Map archiveCache = new LRUMap(archiveCacheSize); -- Pablo
On Apr 10, 2007, at 10:15 AM, Pablo Oliveira wrote:
On Apr 09, Ludovic Dubost wrote :
Right.. the XWikiContext with the classCache and archiveCache (which are needed) is not made for being use for too long without cleaning up.. Maybe there should be some automated cleaning up when it gets too big..
A simple non invasive solution, would be to use a LRUMap instead of a simple HashMap. It can be set to a maximum size and will use a Least Recently Used elimination when full and adding a new object. See patch below.
Looks very good to me Pablo. Some other things to take into account: 1) I see there's this method in XWikiContext: public BaseClass getBaseClass(String name) { return (BaseClass) classCache.get(name); } This is too dangerous. I've checked and it's only used internally by XWikiContext.getClass(). It's dangerous because if we have a LRU strategy it could return null if some piece of code uses it. I believe this method should be removed altogether and the get from the map should be done directly in the getClass() method. 2) In XWikiHibernateVersioningStore, there's: // This will also make sure that the Archive has a strong reference // and will not be discarded as long as the context exists. context.addDocumentArchive(key, archivedoc); This will no longer be true. I'm not sure who wrote this comment and why it's good that a strong reference is kept during the whole lifecycle of the context... Whoever did this please come forward to explain! :) 3) Arnaud, do you think you could try what you've done with Pablo's patch and tell how it went? We need some verifications that it works fine before applying it. 4) Last, it would be great if someone could create a jira issue and attach pablo's patch Thanks -Vincent
Index: core/src/main/java/com/xpn/xwiki/XWikiContext.java =================================================================== --- core/src/main/java/com/xpn/xwiki/XWikiContext.java (revision 2320) +++ core/src/main/java/com/xpn/xwiki/XWikiContext.java (working copy) @@ -31,6 +31,7 @@ import com.xpn.xwiki.web.*; import com.xpn.xwiki.validation.XWikiValidationStatus; import org.apache.xmlrpc.server.XmlRpcServer; +import org.apache.commons.collections.map.LRUMap;
import java.net.URL; import java.util.*; @@ -65,11 +66,13 @@ private String wikiOwner; private XWikiDocument wikiServer; private int cacheDuration = 0; + private int classCacheSize = 10; + private int archiveCacheSize = 10;
// Used to avoid recursive loading of documents if there are recursives usage of classes - private Map classCache = new HashMap(); + private Map classCache = new LRUMap(classCacheSize); // Used to avoir reloading archives in the same request - private Map archiveCache = new HashMap(); + private Map archiveCache = new LRUMap(archiveCacheSize);
-- Pablo
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws
Vincent Massol a écrit :
3) Arnaud, do you think you could try what you've done with Pablo's patch and tell how it went? We need some verifications that it works fine before applying it.
Hi Vincent, Ok, I will try and tell you as soon as possible. Arnaud.
Vincent Massol a écrit :
3) Arnaud, do you think you could try what you've done with Pablo's patch and tell how it went? We need some verifications that it works fine before applying it.
4) Last, it would be great if someone could create a jira issue and attach pablo's patch
Ok, Pablo's patch works perfectly in my case. I submitted the patch here : http://jira.xwiki.org/jira/browse/XWIKI-1101 Regards, Arnaud.
On Apr 18, Arnaud Thimel wrote :
Vincent Massol a écrit :
3) Arnaud, do you think you could try what you've done with Pablo's patch and tell how it went? We need some verifications that it works fine before applying it.
4) Last, it would be great if someone could create a jira issue and attach pablo's patch
Ok, Pablo's patch works perfectly in my case. I submitted the patch here : http://jira.xwiki.org/jira/browse/XWIKI-1101
Thanks for the testing Arnaud ;) Pablo.
Hi Arnaud,
@Youcef Sorry, I forgot to mention that I was already able to import all my DB, but I need to set the maximum heap size to more than 1.5Go !!!
I didn't think so that I suggested setting the heap size to 1.5 Go :-)! Here are the values that I found on the web: -Xmx64m -Xms64m; or you could change them to 128, 256, 512 or 1024. May be we can extend it to other values but I am not sure!
Anyway, I don't know how, but XWiki could be improved by increasing the way we use these caches.
In fact, the importation of my data was not easy, even changing these values and also creating new indexes. I agree with your remark, the manner with which the "cache" is managed have to be checked again - I think there are somewhere problem with the "garbage collector". Sorry, I would like to participate but I have not enough skills to do that. Best wishes Youcef
On Apr 11, 2007, at 12:57 PM, Youcef BEY wrote:
Hi Arnaud,
@Youcef Sorry, I forgot to mention that I was already able to import all my DB, but I need to set the maximum heap size to more than 1.5Go !!!
I didn't think so that I suggested setting the heap size to 1.5 Go :-)!
Here are the values that I found on the web:
-Xmx64m -Xms64m; or you could change them to 128, 256, 512 or 1024.
May be we can extend it to other values but I am not sure!
Anyway, I don't know how, but XWiki could be improved by increasing the way we use these caches.
In fact, the importation of my data was not easy, even changing these values and also creating new indexes.
I agree with your remark, the manner with which the "cache" is managed have to be checked again - I think there are somewhere problem with the "garbage collector".
Sorry, I would like to participate but I have not enough skills to do that.
If you check this thread you'll see Pablo has proposed a solution. I'm just waiting for someone to try it out and says it works to apply it. Thanks -Vincent
participants (9)
-
Arnaud Thimel -
bey -
Ludovic Dubost -
Pablo Oliveira -
Sergiu Dumitriu -
Sébastien Gaïde -
Vincent Massol -
Youcef BEY
-
Youcef.Bey＠imag.fr