[xwiki-dev] Number of document decrease XWiki performance
bey
youcef.bey at imag.fr
Fri Feb 9 13:10:55 CET 2007
Hi Luduvic,
Thank you very much for your email.
Yes, I checked these options.
- I set the temporary java memory to 1 Go and increase the number of
files to 1000000 in xml.cfg. Unfortunately, this configuration do not
solve the problem.
- With the MySql administrator : a simple request for displaying around
10000 entries is done in few seconds and when deleting them it takes
5.0459 sec.
- Deleting one document in XWiki containing 10000 documents is done in
10 minutes, Strange...!
- The importation of a XWiki database with 13369 entries takes a few
seconds from my second XWiki using an external database administrator.
- I created the indexes that Vincent suggested me to do. I remarked an
increasing of speed but not really significant.
ps. I deal with data using UTF-8 encoding : Jetty CharSet = UTF-8, SQL
DB with collation "utf8_general_ci", XWiki CharSet = UTF-8, etc.
I run such kind of tasks on WinXP (1 Ghz, 500 Mo RAM, 20 Go HD) and
also with last version of Debian (2,5 Ghz, 2 Go RAM and 100 Go HD).
This is what I got in the Shell during the work of XWiki:
-----------------------------------------------------------
WARN P1-16 http://localhost:8080/xwiki/bin/delete/A.H/WebHome?confirm=1&
RegexTokenFilter:filter:99 - <span class="error">Error</span>:
com.xpn.xwiki.render.filter.XWikiHeadingFilter at 3bc63:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
18:15:26.764 INFO [P1-14]
org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134)
>55> Velocity [error] Left side ($!hidecolumns) of '==' operation has
null value. If a reference, it may not be in the context. Operation not
possible. [line 9, column 21]
18:15:26.766 INFO [P1-14]
org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134)
>39> Velocity [error] RHS of #set statement is null. Context will not
be modified. [line 11, column 1]
-----------------------------------------------------------
Any suggestion will be deeply appreciated?
Thanks in advances.
Ludovic Dubost a écrit :
>
> Hi Youcef,
>
> Have you modified the configuration for memory in Java from the
> default of 64M. You might want to setup at least 256M.
> If you have a lot of documents you might want to increase the number
> of cached docs in xwiki.cfg also
>
> Ludovic
>
> Youcef BEY a écrit :
>> Hi Vincent, Jeremi and Sebastien
>>
>> Thank you very much for your reply and I apologize for late reply.
>>
>> Your solution is not really enough for sloving the problem. Sorry for
>> that.
>>
>> In fact, I created all the index so it seems that xwiki increase a
>> little the speed but not really enough.
>>
>> I have compilted for testing it 10000 but xwiki become unusable!
>>
>> I did not use class in my document. The format of document is textual
>> and small size (<2 Ko)
>>
>> Sebastien, I will send you by tomorrow a file containing 350000
>> entries and all programs. As now I am transfering all my data and
>> XWiki to Vine Linux (Debian) and also decided to leave Win XP for
>> ever, I really enjoyed Linux :-)
>>
>> When I installed XWiki on my Linux machine (Vine Linux) I have got an
>> error. Even I have an experience with XWiki but this error can't find
>> any issues for solving it. I prefer to send it on another email
>>
>> Thank again.
>>
>> Youcef
>>
>> From: "Vincent Massol" <vincent at massol.net>
>> To: <xwiki-dev at objectweb.org>
>> Sent: Tuesday, February 06, 2007 1:22 AM
>> Subject: Re: [xwiki-dev] Number of document decrease XWiki performance
>>
>>
>> Hi Youcef,
>>
>> (Please do not hijack existing thread, this is bad practice... :-)).
>>
>> Regarding your issue you have to set up indexes if you're working
>> with big tables (these are not set automatically as of now).
>>
>> I'm not sure exactly what indexes need to be set but here's some idea
>> of what should be done (people in the know please confirm):
>>
>> create index xwd_name on xwikidoc (xwd_name);
>> create index xwd_fullname on xwikidoc (xwd_fullname);
>> create index xwd_web on xwikidoc (xwd_web);
>> create index xwd_name on xwikiobjects (xwo_name);
>> create index xwl_value on xwikilongs (xwl_value);
>> create index xwi_value on xwikiintegers (xwi_value);
>> create index xws_value on xwikistrings (xws_value);
>> create index xwl_value on xwikilargestrings (xwl_value(50));
>> create index xwo_classname on xwikiobjects (xwo_classname);
>> create index xwd_creation_date on xwikidoc (xwd_creation_date);
>> create index xwd_date on xwikidoc (xwd_date);
>> create index xwd_content_update_date on xwikidoc
>> (xwd_content_update_date);
>> create index xwd_content_author on xwikidoc (xwd_content_author);
>> create index xwd_author on xwikidoc (xwd_author);
>> create index xwd_creator on xwikidoc (xwd_creator);
>> create index xwd_language on xwikidoc (xwd_language);
>> create index xwd_default_language on xwikidoc (xwd_default_language);
>> create index xwd_title on xwikidoc (xwd_title);
>> create index xwd_parent on xwikidoc (xwd_parent(50));
>>
>> Note that we aready have a jira issue for this (http://jira.xwiki.org/
>> jira/browse/XWIKI-605). We just don't know yet if we can have a
>> generic script that'll work for all databases.
>>
>> There might be other issues but I'll let others with experience with
>> big XWiki instances answer...
>>
>> Thanks
>> -Vincent
>>
>> On Feb 7, 2007, at 3:00 AM, Youcef BEY wrote:
>>
>>> Hi all,
>>>
>>> I have until now not solved this problem (since more than 10 month):
>>>
>>> - using this program to create automatically a document work well
>>> for few documents:
>>>
>>> XW = xwiki.XWiki;
>>> XWContext = context.context;
>>> XWikiDocument NewEntry = new XWikiDocument(request.getParameter
>>> ("web"), SourceValue);
>>> NewEntry.setContent(EntryContent);
>>> XW.saveDocument(NewEntry, XWContext);
>>>
>>>
>>> When putting these instructions in a LOOP for 10000 entries to
>>> transfer from another TABLE in the same XWiki database cause a
>>> problem. The probleme is not an error but the XWiki crash! and
>>> without finishing the LOOP, the xwiki display error "out of memory!"
>>>
>>> But now, taking the same program and run it in the shell (javac,
>>> java) to transfert these 10000 entries to TABLE in the XWiki
>>> database, it's done in less 1 minute?
>>>
>>> Strange for me ...
>>>
>>> - More ...and this is the more importante thing :
>>>
>>> One time I success to transfer 25 000 documents in the XWiki
>>> database (using the same script above) the XWiki become very heavy
>>> and cannot really work with it at all.
>>>
>>> MySql tables are supposed to contain millions of records, and when
>>> using the database adminstrator and check with SQL statment, I get
>>> resultat in few seconds, that means that the problem is in XWiki.
>>>
>>> My Question : Why with 25 000 documents XWiki don't work as usually
>>>
>>> Anyone faced this problem before? Do you have made testing of XWiki
>>> for 1 millions documents for exemple?
>>>
>>> I have XP, 1.2 MHz and 500 Mo but the following experience on MySQL
>>> gives more arguments that not MySQL is causing this problem :
>>>
>>> (Web link : http://www.guilde.asso.fr/lurker/message/
>>> 20050817.183426.a1b8e267.en.html)
>>>
>>> -----------------------------------------------------------------
>>> Il y a quelques années, j'ai utilisé MySQL pour gérér 1.7 millions
>>> d'enregistrements dans une table. La table faisait environ 500Mo, plus
>>> 400Mo d'indexes pour la recherche de textes dans un des champs.
>>>
>>> Coté perfs, sur un ordinateur portable à 500Mhz et 128Mo de ram qui
>>> faisait tourner un Windows NT4 bourré d'applis en mémoire, il fallait
>>> environ 5s à 10s pour recherche les tuples contenant certains mots-
>>> clefs
>>> stokés dans le fameux champs de texte.
>>>
>>> Bref, des performances tout à fait acceptable, sur une machine qui
>>> n'était pourtant pas du tout fait pour faire serveur de base de
>>> données
>>> (et je ne parle pas que de l'OS...).
>>> --------------------------------------------------------------
>>>
>>> Many thanks for any suggestion.
>>>
>>> Best wishes
>>> ----------------------------------
>>> Youcef Bey
>>>
>>>
>>> --
>>> You receive this message as a subscriber of the xwiki-
>>> dev at objectweb.org mailing list.
>>> To unsubscribe: mailto:xwiki-dev-unsubscribe at objectweb.org
>>> For general help: mailto:sympa at objectweb.org?subject=help
>>> ObjectWeb mailing lists service home page: http://www.objectweb.org/
>>> wws
>>
>>
>>
>>
>>
>>
>> ___________________________________________________________________________
>>
>> Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et
>> son interface révolutionnaire.
>> http://fr.mail.yahoo.com
>>
>>
>>
>>
>> --------------------------------------------------------------------------------
>>
>>
>>
>>>
>>> --
>>> You receive this message as a subscriber of the
>>> xwiki-dev at objectweb.org mailing list.
>>> To unsubscribe: mailto:xwiki-dev-unsubscribe at objectweb.org
>>> For general help: mailto:sympa at objectweb.org?subject=help
>>> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> --
>> You receive this message as a subscriber of the
>> xwiki-dev at objectweb.org mailing list.
>> To unsubscribe: mailto:xwiki-dev-unsubscribe at objectweb.org
>> For general help: mailto:sympa at objectweb.org?subject=help
>> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>>
>
>
> ------------------------------------------------------------------------
>
>
> --
> You receive this message as a subscriber of the xwiki-dev at objectweb.org mailing list.
> To unsubscribe: mailto:xwiki-dev-unsubscribe at objectweb.org
> For general help: mailto:sympa at objectweb.org?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xwiki.org/pipermail/devs/attachments/20070209/fa4d1dc9/attachment.htm
More information about the devs
mailing list