4 Apr
2007
4 Apr
'07
8:24 p.m.
Hi all,
For a particular project, I have a plugin that reads data from an
external database using some JDBC queries.
For each row of my JDBC ResultSet, the plugin creates an XWiki document,
the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to import :-D.
Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the
documents created.
Every 200 document creation, I do the following calls :
context.getWiki().flushCache();
context.getWiki().getStore().cleanUp(context);
System.gc();
Anyway, the amount of memory used doesn't go down after flushing the caches.
And I still get my Heap Space exception.
Does anybody has an idea for me ?
Who thinks that I could be a XWiki memory leak ?
Thanks,
--
Arnaud Thimel
Société Code Lutin - http://www.codelutin.com
4 Apr
4 Apr
8:54 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Hi Arnaud,
On Apr 04, Arnaud Thimel wrote :
All is running fine except that I have around 50.000
objects to import :-D.
Then I get a Java Heap Space exception.
[...]
Does anybody has an idea for me ?
Who thinks that I could be a XWiki memory leak ?
Have you tried to do a heap profiling ? It will help you determining
where is your memory going, and might reveal possible memory leaks.
Kind regards,
Pablo
PS: You can have a look at
http://java.sun.com/developer/technicalArticles/Programming/HPROF.html
5 Apr
5 Apr
7:50 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Hi Arnaud,
I faced this problem before.
The importation of huge data need the configuration of java parameters of the
heap. I was not able to import and create 300000 documents - without object -
until I changed the following parameters :
-Xms512m -Xmx521m (you could add or change them in startup.bat in windows or
catalina.sh on linux).
These parameters depends on your machine. Then you can increase (-->1024) or
decrease them (1024).
Hope this will solve your problem.
Cheers.
-----------
Youcef Bey
Quoting Arnaud Thimel <thimel(a)codelutin.com>om>:
Hi all,
For a particular project, I have a plugin that reads data from an
external database using some JDBC queries.
For each row of my JDBC ResultSet, the plugin creates an XWiki document,
the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to import :-D.
Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the
documents created.
Every 200 document creation, I do the following calls :
context.getWiki().flushCache();
context.getWiki().getStore().cleanUp(context);
System.gc();
Anyway, the amount of memory used doesn't go down after flushing the caches.
And I still get my Heap Space exception.
Does anybody has an idea for me ?
Who thinks that I could be a XWiki memory leak ?
Thanks,
--
Arnaud Thimel
Société Code Lutin - http://www.codelutin.com
-------------------------------------------------
envoyé via Webmail/IMAG !
9:11 p.m.
New subject: [xwiki-dev] Possible memory leak ?
On Apr 5, 2007, at 7:50 PM, Youcef.Bey(a)imag.fr wrote:
Hi Arnaud,
I faced this problem before.
The importation of huge data need the configuration of java
parameters of the
heap. I was not able to import and create 300000 documents -
without object -
until I changed the following parameters :
-Xms512m -Xmx521m (you could add or change them in startup.bat in
windows or
catalina.sh on linux).
These parameters depends on your machine. Then you can increase (--
if anyone has the time, it would be nice to use a profile to
understand why this is required...
Thanks
-Vincent
1024) or
decrease them (1024).
Hope this will solve your problem. Quoting Arnaud Thimel <thimel(a)codelutin.com>om>:
Hi all,
For a particular project, I have a plugin that reads data from an
external database using some JDBC queries.
For each row of my JDBC ResultSet, the plugin creates an XWiki
document,
the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to
import :-D.
Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference
to the
documents created.
Every 200 document creation, I do the following calls :
context.getWiki().flushCache();
context.getWiki().getStore().cleanUp(context);
System.gc();
Anyway, the amount of memory used doesn't go down after flushing
the caches.
And I still get my Heap Space exception.
Does anybody has an idea for me ?
Who thinks that I could be a XWiki memory leak ?
Thanks,
--
Arnaud Thimel
Société Code Lutin - http://www.codelutin.com
-------------------------------------------------
envoyé via Webmail/IMAG !
--
You receive this message as a subscriber of the xwiki-
dev(a)objectweb.org mailing list.
To unsubscribe: mailto:xwiki-dev-unsubscribe@objectweb.org
For general help: mailto:sympa@objectweb.org?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/
wws
6 Apr
6 Apr
3:37 a.m.
New subject: [xwiki-dev] Possible memory leak ?
(Sorry for the mistakes in my last email)
Just for correction : -Xms512m (-Xmx521m) --> -Xmx512m
Possible values : 64 and 1024.
Best wishes
-------------
Youcef Bey
Vincent Massol a écrit :
On Apr 5, 2007, at 7:50 PM, Youcef.Bey(a)imag.fr wrote:
--
Youcef Bey
Graduate School of Education
The University of Tokyo,
Mail address : bey(a)p.u-tokyo.ac.jp
Home page : http://panflute.p.u-tokyo.ac.jp/~bey/
On study leave GETALP team
GETALP, LIG Lab., IMAG-Campus, BP 53
385, rue de la bibliotheque
38041 Grenoble Cedex 9
France
Hi Arnaud,
I faced this problem before.
The importation of huge data need the configuration of java
parameters of the
heap. I was not able to import and create 300000 documents - without
object -
until I changed the following parameters :
-Xms512m -Xmx521m (you could add or change them in startup.bat in
windows or
catalina.sh on linux).
These parameters depends on your machine. Then you can increase
(-->1024) or
decrease them (1024).
Hope this will solve your problem.
if anyone has the time, it would be nice to use a profile to
understand why this is required...
Thanks
-Vincent
Quoting Arnaud Thimel
<thimel(a)codelutin.com>om>:
------------------------------------------------------------------------
--
You receive this message as a subscriber of the xwiki-dev(a)objectweb.org mailing list.
To unsubscribe: mailto:xwiki-dev-unsubscribe@objectweb.org
For general help: mailto:sympa@objectweb.org?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
Hi all,
For a particular project, I have a plugin that reads data from an
external database using some JDBC queries.
For each row of my JDBC ResultSet, the plugin creates an XWiki
document,
the associated objects, and fill these objects.
All is running fine except that I have around 50.000 objects to
import :-D.
Then I get a Java Heap Space exception.
I re-re-re-re-re-read my source code, I don't keep any reference to the
documents created.
Every 200 document creation, I do the following calls :
context.getWiki().flushCache();
context.getWiki().getStore().cleanUp(context);
System.gc();
Anyway, the amount of memory used doesn't go down after flushing the
caches.
And I still get my Heap Space exception.
Does anybody has an idea for me ?
Who thinks that I could be a XWiki memory leak ?
Thanks,
--Arnaud Thimel
Société Code Lutin - http://www.codelutin.com
-------------------------------------------------
envoyé via Webmail/IMAG !
--
You receive this message as a subscriber of the
xwiki-dev(a)objectweb.org mailing list.
To unsubscribe: mailto:xwiki-dev-unsubscribe@objectweb.org
For general help: mailto:sympa@objectweb.org?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws 
7 Apr
7 Apr
8:46 a.m.
New subject: [xwiki-dev] Possible memory leak ?
Arnaud Thimel a écrit :
Hi all,
Hi Arnaud,
Anyway, the amount of memory used doesn't go down
after flushing the
caches.
And I still get my Heap Space exception.
I don't know if it's related to your problem, but I remember that I had
to flush some caches in XWikiContext (classCache, archiveCache) when I
was using the same XWikiContext to create a heap of pages. As you can
see in the code these two caches are always used to add some objects in
them, nether to remove them.
The two caches are not accessible by default, so you will have to add
some code in XWikiContext to flush them.
HTH,
Seb.
9 Apr
9 Apr
11:01 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Hi all,
I kept searching about my problem.
@Pablo
I tried to use profiling with Hprof and jmap+jhat. But the information I
got, didn't allow me to solve my problem.
@Youcef
Sorry, I forgot to mention that I was already able to import all my DB,
but I need to set the maximum heap size to more than 1.5Go !!!
@Sébastien
You found the thing I was missing :) When I flush the xwiki caches, I
now also flush the "classCache" and "archiveCache" from the
XWikiContext.
I can now import my thousands of objects using only the default heap
size (64Mo).
Anyway, I don't know how, but XWiki could be improved by increasing the
way we use these caches.
Thanks all, and best regards.
Arnaud.
Sébastien Gaïde wrote:
--
Arnaud Thimel
Société Code Lutin - http://www.codelutin.com
Anyway, the
amount of memory used doesn't go down after flushing the
caches.
And I still get my Heap Space exception.
I don't know if it's related to your problem, but I remember that I
had to flush some caches in XWikiContext (classCache, archiveCache)
when I was using the same XWikiContext to create a heap of pages. As
you can see in the code these two caches are always used to add some
objects in them, nether to remove them.
The two caches are not accessible by default, so you will have to add
some code in XWikiContext to flush them.
HTH,
Seb.
11:20 p.m.
New subject: [xwiki-dev] Possible memory leak ?
It might be a misconfiguration somewhere. Somebody should look at the way we
use these caches.
On 4/10/07, Arnaud Thimel <thimel(a)codelutin.com> wrote:
Hi all,
I kept searching about my problem.
@Pablo
I tried to use profiling with Hprof and jmap+jhat. But the information I
got, didn't allow me to solve my problem.
@Youcef
Sorry, I forgot to mention that I was already able to import all my DB,
but I need to set the maximum heap size to more than 1.5Go !!!
@Sébastien
You found the thing I was missing :) When I flush the xwiki caches, I
now also flush the "classCache" and "archiveCache" from the
XWikiContext.
I can now import my thousands of objects using only the default heap
size (64Mo).
Anyway, I don't know how, but XWiki could be improved by increasing the
way we use these caches.
Thanks all, and best regards.
Arnaud.
--
http://purl.org/net/sergiu
11:38 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Arnaud Thimel a écrit :
--
Ludovic Dubost
Blog: http://www.ludovic.org/blog/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
AIM: nvludo Yahoo: ludovic
Hi all,
I kept searching about my problem.
@Pablo
I tried to use profiling with Hprof and jmap+jhat. But the information
I got, didn't allow me to solve my problem.
@Youcef
Sorry, I forgot to mention that I was already able to import all my
DB, but I need to set the maximum heap size to more than 1.5Go !!!
@Sébastien
You found the thing I was missing :) When I flush the xwiki caches, I
now also flush the "classCache" and "archiveCache" from the
XWikiContext.
I can now import my thousands of objects using only the default heap
size (64Mo).
Anyway, I don't know how, but XWiki could be improved by increasing
the way we use these caches.
Thanks all, and best regards.
Arnaud.
Sébastien Gaïde wrote:
>> Anyway, the amount of memory used doesn't go down after flushing the
>> caches.
>> And I still get my Heap Space exception.
>>
>
> I don't know if it's related to your problem, but I remember that I
> had to flush some caches in XWikiContext (classCache, archiveCache)
> when I was using the same XWikiContext to create a heap of pages. As
> you can see in the code these two caches are always used to add some
> objects in them, nether to remove them.
Right.. the XWikiContext with the
classCache and archiveCache (which are
needed) is not made for being use for too long without cleaning up..
Maybe there should be some automated cleaning up when it gets too big..
Ludovic
The two caches are not accessible by default, so you will have to add
some code in XWikiContext to flush them.
HTH,
Seb.
------------------------------------------------------------------------
--
You receive this message as a subscriber of the xwiki-dev(a)objectweb.org mailing list.
To unsubscribe: mailto:xwiki-dev-unsubscribe@objectweb.org
For general help: mailto:sympa@objectweb.org?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
10 Apr
10 Apr
10:15 a.m.
New subject: [xwiki-dev] Possible memory leak ?
On Apr 09, Ludovic Dubost wrote :
Right.. the XWikiContext with the classCache and
archiveCache (which are
needed) is not made for being use for too long without cleaning up..
Maybe there should be some automated cleaning up when it gets too big..
A simple non invasive solution, would be to use a LRUMap instead
of a simple HashMap.
It can be set to a maximum size and will use a Least Recently Used elimination
when full and adding a new object. See patch below.
Index: core/src/main/java/com/xpn/xwiki/XWikiContext.java
===================================================================
--- core/src/main/java/com/xpn/xwiki/XWikiContext.java (revision 2320)
+++ core/src/main/java/com/xpn/xwiki/XWikiContext.java (working copy)
@@ -31,6 +31,7 @@
import com.xpn.xwiki.web.*;
import com.xpn.xwiki.validation.XWikiValidationStatus;
import org.apache.xmlrpc.server.XmlRpcServer;
+import org.apache.commons.collections.map.LRUMap;
import java.net.URL;
import java.util.*;
@@ -65,11 +66,13 @@
private String wikiOwner;
private XWikiDocument wikiServer;
private int cacheDuration = 0;
+ private int classCacheSize = 10;
+ private int archiveCacheSize = 10;
// Used to avoid recursive loading of documents if there are recursives usage of
classes
- private Map classCache = new HashMap();
+ private Map classCache = new LRUMap(classCacheSize);
// Used to avoir reloading archives in the same request
- private Map archiveCache = new HashMap();
+ private Map archiveCache = new LRUMap(archiveCacheSize);
--
Pablo
10:54 a.m.
New subject: [xwiki-dev] Possible memory leak ?
On Apr 10, 2007, at 10:15 AM, Pablo Oliveira wrote:
On Apr 09, Ludovic Dubost wrote :
Looks very good to me Pablo.
Some other things to take into account:
1) I see there's this method in XWikiContext:
public BaseClass getBaseClass(String name) {
return (BaseClass) classCache.get(name);
}
This is too dangerous. I've checked and it's only used internally by
XWikiContext.getClass(). It's dangerous because if we have a LRU
strategy it could return null if some piece of code uses it. I
believe this method should be removed altogether and the get from the
map should be done directly in the getClass() method.
2) In XWikiHibernateVersioningStore, there's:
// This will also make sure that the Archive has a
strong reference
// and will not be discarded as long as the context
exists.
context.addDocumentArchive(key, archivedoc);
This will no longer be true. I'm not sure who wrote this comment and
why it's good that a strong reference is kept during the whole
lifecycle of the context... Whoever did this please come forward to
explain! :)
3) Arnaud, do you think you could try what you've done with Pablo's
patch and tell how it went? We need some verifications that it works
fine before applying it.
4) Last, it would be great if someone could create a jira issue and
attach pablo's patch
Thanks
-Vincent
Right.. the XWikiContext with the classCache and
archiveCache
(which are
needed) is not made for being use for too long without cleaning up..
Maybe there should be some automated cleaning up when it gets too
big..
A simple non invasive solution, would be to use a LRUMap instead
of a simple HashMap.
It can be set to a maximum size and will use a Least Recently Used
elimination
when full and adding a new object. See patch below. Index:
core/src/main/java/com/xpn/xwiki/XWikiContext.java
===================================================================
--- core/src/main/java/com/xpn/xwiki/XWikiContext.java (revision
2320)
+++ core/src/main/java/com/xpn/xwiki/XWikiContext.java (working copy)
@@ -31,6 +31,7 @@
import com.xpn.xwiki.web.*;
import com.xpn.xwiki.validation.XWikiValidationStatus;
import org.apache.xmlrpc.server.XmlRpcServer;
+import org.apache.commons.collections.map.LRUMap;
import java.net.URL;
import java.util.*;
@@ -65,11 +66,13 @@
private String wikiOwner;
private XWikiDocument wikiServer;
private int cacheDuration = 0;
+ private int classCacheSize = 10;
+ private int archiveCacheSize = 10;
// Used to avoid recursive loading of documents if there are
recursives usage of classes
- private Map classCache = new HashMap();
+ private Map classCache = new LRUMap(classCacheSize);
// Used to avoir reloading archives in the same request
- private Map archiveCache = new HashMap();
+ private Map archiveCache = new LRUMap(archiveCacheSize);
--
Pablo
--
You receive this message as a subscriber of the xwiki-
dev(a)objectweb.org mailing list.
To unsubscribe: mailto:xwiki-dev-unsubscribe@objectweb.org
For general help: mailto:sympa@objectweb.org?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/
wws
17 Apr
17 Apr
5:39 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Vincent Massol a écrit :
3) Arnaud, do you think you could try what you've
done with Pablo's
patch and tell how it went? We need some verifications that it works
fine before applying it.
Hi Vincent,
Ok, I will try and tell you as soon as possible.
Arnaud.
18 Apr
18 Apr
1:49 a.m.
New subject: [xwiki-dev] Possible memory leak ?
Vincent Massol a écrit :
3) Arnaud, do you think you could try what you've
done with Pablo's
patch and tell how it went? We need some verifications that it works
fine before applying it.
4) Last, it would be great if someone could create a jira issue and
attach pablo's patch
Ok, Pablo's patch works perfectly in my case.
I submitted the patch here : http://jira.xwiki.org/jira/browse/XWIKI-1101
Regards,
Arnaud.
10:58 a.m.
New subject: [xwiki-dev] Possible memory leak ?
On Apr 18, Arnaud Thimel wrote :
Vincent Massol a écrit :
Thanks for the testing Arnaud ;)
Pablo.
3) Arnaud, do you think you could try what
you've done with Pablo's
patch and tell how it went? We need some verifications that it works
fine before applying it.
4) Last, it would be great if someone could create a jira issue and
attach pablo's patch
Ok, Pablo's patch works perfectly in my case.
I submitted the patch here : http://jira.xwiki.org/jira/browse/XWIKI-1101
11 Apr
11 Apr
12:57 p.m.
New subject: [xwiki-dev] Possible memory leak ?
Hi Arnaud,
@Youcef
Sorry, I forgot to mention that I was already able to import all my DB,
but I need to set the maximum heap size to more than 1.5Go !!!
I didn't think so that I suggested setting the heap size to 1.5 Go :-)!
Here are the values that I found on the web:
-Xmx64m -Xms64m; or you could change them to 128, 256, 512 or 1024.
May be we can extend it to other values but I am not sure!
Anyway, I don't know how, but XWiki could be
improved by increasing the
way we use these caches.
In fact, the importation of my data was not easy, even changing these values
and also creating new indexes.
I agree with your remark, the manner with which the "cache" is managed have
to be checked again - I think there are somewhere problem with the "garbage
collector".
Sorry, I would like to participate but I have not enough skills to do that.
Best wishes
Youcef
10 Apr
10 Apr
9:08 p.m.
New subject: [xwiki-dev] Possible memory leak ?
On Apr 11, 2007, at 12:57 PM, Youcef BEY wrote:
Hi Arnaud,
If you check this thread you'll see Pablo has proposed a solution.
I'm just waiting for someone to try it out and says it works to apply
it.
Thanks
-Vincent
@Youcef
Sorry, I forgot to mention that I was already able to import all
my DB,
but I need to set the maximum heap size to more than 1.5Go !!!
I didn't think so that I suggested setting the heap size to 1.5
Go :-)!
Here are the values that I found on the web:
-Xmx64m -Xms64m; or you could change them to 128, 256, 512 or 1024.
May be we can extend it to other values but I am not sure!
Anyway, I don't know how, but XWiki could be
improved by
increasing the
way we use these caches.
In fact, the importation of my data was not easy, even changing
these values and also creating new indexes.
I agree with your remark, the manner with which the "cache" is
managed have to be checked again - I think there are somewhere
problem with the "garbage collector".
Sorry, I would like to participate but I have not enough skills to
do that.
6361
days inactive
6375
days old
15 comments
9 participants
participants (9)
-
Arnaud Thimel -
bey -
Ludovic Dubost -
Pablo Oliveira -
Sergiu Dumitriu -
Sébastien Gaïde -
Vincent Massol -
Youcef BEY
-
Youcef.Bey＠imag.fr