Hello Vincent, To add to the alternatives you presented, you could even use proper xar (http://code.google.com/p/xar/). It is actually a quite nice archive format, and it does have support for metadata, checksums, table of contents etc. On the other hand, it is not an established archive format, and I don't think there is a Java implementation for it, so the overhead of implementing this would be a lot more substantial than using zip or jar. Catalin On 2/3/07, Vincent Massol <[email protected]> wrote:

Hi Jeremi,

I'm +1 on my side but as you voted -1 we'll need to convince you to change your vote ;-) Let me give it a try below...

On Feb 3, 2007, at 9:41 AM, jeremi joslin wrote:

...

Hi,

-1

I'm not for changing to jar. it's not a Java archive, it's an XWiki Archive. I don't see the point to change. People will think it's javacode.

* JAR is an archive format. It can be used for lots of different things. There are even software that use JARs for their installation. * I really believe that in the future our applications will not be made only of XML files but will also contain other libraries (jar files) and java classes. For example imagine the IRC bot application. There are some xwiki pages but we also need the picoirc jar. We'll need to bundle all that together. * XWiki is a Java application so all users are familiar with the JAR format.

Of course we can use the XAR format for that. XAR looks nice because there's XWiki inside but there's a cost of inventing a new archive format: * It needs to be described * There needs to be tools for it (or people need to rename it to zip...). BTW if it's a zip we might as well name it zip... The problem with Zip is that it does not support meta data information such as the jar format does. And we need these... And there are APIs to get these metadata so we don't have to reimplement all the code (as we've done for the package.xml file). * It makes it difficult for the build (this is the same argument as above with tools) as all the existing plugins do not understand the XAR format so we have to reinvent all the tools... This is why I had to implement the xar handler and the xar plugin in our repository. It would make it much easier if it were an existing format such as JAR.

I think it would be much better to use an existing format (which we do except we're using Zip instead of JAR), already documented rather than invent a new one. There's no point in us hiding the underlying format we use by changing the extension. We might as well use that format's extension...

...

if you look at this page : http://filext.com/detaillist.php?extdetail=XAR there is some other software who are using this extension. So why XWiki can't use it?

Some people have thrown nuclear bombs in the past. Is that a good reason for others to do it? Probably not.

It's not because people are doing bad things that we have to follow them ;-)

Thanks -Vincent

...

On 2/3/07, Catalin Hritcu <[email protected]> wrote:

...

Hello all,

What would you think of using the JAR extension instead of (mis-) using XAR for XWiki applications, and using the JAR manifest for storing the information that is now stored in package.xml ? Any other way this can be fixed ?

Already created a JIRA task about this

http://jira.xwiki.org/jira/browse/XWIKI-802

but I would like to hear your opinions before spending the time to implement a change you might actually not want :)

Catalin

---------- Forwarded message ---------- From: Vincent Massol <[email protected]> Date: Feb 2, 2007 7:56 PM Subject: Re: [xwiki-dev] .xar Extension NOT Used Properly To: Catalin Hritcu <[email protected]>

Hi Catalin,

I think it's best to discuss all this on the list.

Also, I've provided some answer in JIRA.

Thanks -Vincent

On Feb 1, 2007, at 11:22 PM, Catalin Hritcu wrote:

...

Hi Vincent,

I created a JIRA issue for this and assigned it to myself: http://jira.xwiki.org/jira/browse/XWIKI-802

At the time I don't really know much about what needs to be changed to make it happen, but I'll look into it. Suggestions welcome :)

...

I guess we could use a JAR format and use information in the manifest to describe the content of an archive

Can you also be more precise about the information you want stored in the manifest? http://java.sun.com/j2se/1.5.0/docs/guide/jar/jar.html#JAR% 20Manifest According to this it seems that a manifest can store many things.

Regards, Catalin

On 1/31/07, Vincent Massol <[email protected]> wrote:

...

Hi Catalin,

See my other email on this topic. Basically I think I agree with you but we're currently focusing on releasing XWiki 1.0 and fixing bugs. this is an improvement request that will have to wait for quite long unless someone contributes a patch for it (including patches for the documentation ;-)).

Thanks -Vincent

On Jan 8, 2007, at 11:32 AM, Catalin Hritcu wrote:

The new way to import/export wiki pages is very useful. However, I think that using the .xar extension for the zip(!) archive is highly confusing. This because there is already a XAR(eXtensible ARchiver) archiving utility which uses the .xar extension for quite a while. And while now XAR is standalone, it will be quite probable used by the MacOSX package system in the future. Then download your archive will yield an error like this (safari automatically expands archives):

"Error opening xar archive: xwiki-1.0-beta-1.xar"

Links: http://www.opendarwin.org/projects/xar/ http://www.nabble.com/xar-in-Mac-OS-X-t2081148.html

<messagefooter.txt>

_____________________________________________________________________ ______ Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire. http://fr.mail.yahoo.com

-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http:// www.objectweb.org/wws

-- jeremi

-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws

___________________________________________________________________________ Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire. http://fr.mail.yahoo.com

-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Jeremi, On Feb 4, 2007, at 4:44 PM, jeremi joslin wrote:

On 2/3/07, Vincent Massol <[email protected]> wrote:

...

Hi Jeremi,

I'm +1 on my side but as you voted -1 we'll need to convince you to change your vote ;-) Let me give it a try below...

On Feb 3, 2007, at 9:41 AM, jeremi joslin wrote:

...

Hi,

-1

I'm not for changing to jar. it's not a Java archive, it's an XWiki Archive. I don't see the point to change. People will think it's javacode.

* JAR is an archive format. It can be used for lots of different things. There are even software that use JARs for their installation. yes, but a JAR in java term is a zip. There is also JAR compression method. so if you go this way, it's the same problem.

A JAR isn't a Zip in Java term. It's using the same compression algorithm. It's very different in term of metadata. I won't go in more details but you can check the JAR specification to know more about this.

...

* I really believe that in the future our applications will not be made only of XML files but will also contain other libraries (jar files) and java classes. For example imagine the IRC bot application. There are some xwiki pages but we also need the picoirc jar. We'll need to bundle all that together. And what's wrong with putting a jar inside a xar? even if we use a jar, it will only be readable by XWiki.

It's possible but harder. For example to read a JAR inside a XAR you'll probably need to write and register special handlers against various classes (such as URL factories, etc). No, xwiki isn't the only one. All tools will be able to read a JAR or a ZIP (be them IDE, build tools, desktop tools, etc).

...

* XWiki is a Java application so all users are familiar with the JAR format. But it's not a normal jar. That will lead to misunderstanding.

I'm not sure what you call a normal jar :-) You probably means classes. However a JAR is not meant to contain only classes. It's a generic format to contain any other kind of resources. The nice thing it has though is support for dependencies and class-paths.

...

Of course we can use the XAR format for that. XAR looks nice because there's XWiki inside but there's a cost of inventing a new archive format: * It needs to be described * There needs to be tools for it (or people need to rename it to zip...). BTW if it's a zip we might as well name it zip... The problem with Zip is that it does not support meta data information such as the jar format does. And we need these... And there are APIs to get these metadata so we don't have to reimplement all the code (as we've done for the package.xml file). * It makes it difficult for the build (this is the same argument as above with tools) as all the existing plugins do not understand the XAR format so we have to reinvent all the tools... This is why I had to implement the xar handler and the xar plugin in our repository. It would make it much easier if it were an existing format such as JAR.

Why not renaming the xar to the zip. it's just that people don't need to know what is inside, but if they want they can.

As I said renaming to zip is a solution. Probably a better one than the current XAR. However the pity is that a zip doesn't support metadata by default. You need to define them, document them, teach them, write parsers for them, etc. JARs have this already (this is called the Manifest).

I dont see the point, we are using an API for reading and writing the metadata: XML. We will also need to reimplement the writing of metadata in the manifest file.

No. The JDK provides an API for reading information from the manifest.

So it's already done isn't it?

And it's possibly buggy :-) And you're the only one who knows about it. And it isn't documented. And you haven't implemented it for Maven2 (it's not easy and has some issues) nor any of the tool that will crop up. The question is not that it's done (it's not finished BTW - No doc, no tests). It's about removing code and using standards (to reduce cost even further). Any line of code that is removed saves a lot (fixing bug, documentation, test, teaching, etc).

...

I think it would be much better to use an existing format (which we do except we're using Zip instead of JAR), already documented rather than invent a new one. There's no point in us hiding the underlying format we use by changing the extension. We might as well use that format's extension...

Ok, rename it to zip if you prefer, that's only a problem the extension to be xar for openning it under windows

No that's not true. It's a problem for a lot of tools (Maven2 included). BTW I'm on Mac OS X (which isn't windows) and I can't open a XAR either. I have to rename it to ZIP before... Unfortunately there are very few tools that look inside a file to know what format it is. Most will simply look at the file extension.

. But i prefer people to think it's a file where they have all there pages, and don't need to read what is inside. But if they want, they can.

Sure that's ideal. Problem is that you always have to do this... For example there's no selective export so you have to expand it as a zip, remove files, edit the package.xml file and repackage.

...

...

if you look at this page : http://filext.com/detaillist.php?extdetail=XAR there is some other software who are using this extension. So why XWiki can't use it?

Some people have thrown nuclear bombs in the past. Is that a good reason for others to do it? Probably not.

It's not because people are doing bad things that we have to follow them ;-)

So why changing to the jar format? ;-)

Oh, because I think it's an improvement (it certainly is at least for issues I have with Maven2 and it'll win us hundreds of lines of code). See below for explanations on lightweight container support.

And even the xar format is not so used. I ask to my friends, no one know it. Did you know it before? I thing changing to the jar format will bring more misunderstanding.

I didn't know about the other XAR format and it wouldn't be a good choice (not known enough and thus not enough tool support). I'm not going to push for the format change because we have a lot of work to be done right now and I'm not proposing to do it myself either. What I was just saying here is that I agree that inventing a format is probably not the best solution and reusing an existing well- known format with metadata support is probably a better choice. JAR is such a choice. It looks to me like a good choice, especially considering that we have a need to put java classes in applications. I see you're against it. That's fine. It means it's not a clear cut decision and we should think more about it. In any case as I said above we have more urgent problems. We'll need to revisit this when we move to full fledge application support. These will contain: - pages - java classes (plugins, macros, etc) - external libs - resources (HTML, js files, images, etc). For example a GWT XWiki application will contain java and resources + the necessary pages and possibly external libs (we'll probably bundle them though). The IRC Bot will require an external lib for sure. We need to revisit this in view with the new architecture (component- based architecture based on a microkernel, aka lightweight container). You'll see that most of these containers use the JAR format (OSGi, Plexus, etc) so we'll probably have to use the JAR format anyway... :-) Thanks -Vincent ___________________________________________________________________________ Yahoo! Mail r�invente le mail ! D�couvrez le nouveau Yahoo! Mail et son interface r�volutionnaire. http://fr.mail.yahoo.com

ok 0, but just not to block the decision. do as you want. but I don't see the difference between writing in an XML file or a manifest. You will both have to document it. I don't like manifest because AFAIK it's a plain text file and not an xml. You will have other problems I think.

A JAR isn't a Zip in Java term. It's using the same compression algorithm. It's very different in term of metadata. I won't go in more details but you can check the JAR specification to know more about this. A xar is not a zip in XWiki term :-)

It's possible but harder. For example to read a JAR inside a XAR you'll probably need to write and register special handlers against various classes (such as URL factories, etc). XAR is just an IMPORT/EXPORT format so you will not need this. you just need to extract the jar from it!

And it's possibly buggy :-) And you're the only one who knows about it. And it isn't documented. And you haven't implemented it for Maven2 (it's not easy and has some issues) nor any of the tool that will crop up. No, xwiki isn't the only one. All tools will be able to read a JAR or a ZIP (be them IDE, build tools, desktop tools, etc). When I've done we were not working with maven, but we were with ant. and it's implemented in ant (by ludovic).

I'm not he only one who know how the xar is working.

For example there's no selective export so you have to expand it as a zip, remove files, edit the package.xml file and repackage. No, I've done one, and Vincent you know it.

... that can be a long discussion. Do as you want, i will not argue anymore on this. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: [email protected] http://www.xwiki.com - http://www.pengyou-project.info

On 2/4/07, Vincent Massol <[email protected]> wrote:

On Feb 4, 2007, at 7:36 PM, jeremi joslin wrote:

[snip]

...

...

For example there's no selective export so you have to expand it as a zip, remove files, edit the package.xml file and repackage. No, I've done one, and Vincent you know it.

Oh yes I know but Ludovic has removed it when we moved to using templates so it needs to be done again. Hopefully it can be done quickly but we still need to re-implement it.

And the process I've described above is exactly what I do at each release since Beta 1.

...

... that can be a long discussion. Do as you want, i will not argue anymore on this.

Arguing is good and healthy. As I said in my previous email we've reached a point where we *all* need to step back and think more about this. IMO seen the work we have to do for XWiki 1.0 it's better to postpone this post 1.0 release and think about it in the context of the new architecture.

Yes, but sometime it's lose of time. We have more important things for now to work on. I agree we will have to step back as you say after 1.0. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: [email protected] http://www.xwiki.com - http://www.pengyou-project.info

Hi Youcef, (Please do not hijack existing thread, this is bad practice... :-)). Regarding your issue you have to set up indexes if you're working with big tables (these are not set automatically as of now). I'm not sure exactly what indexes need to be set but here's some idea of what should be done (people in the know please confirm): create index xwd_name on xwikidoc (xwd_name); create index xwd_fullname on xwikidoc (xwd_fullname); create index xwd_web on xwikidoc (xwd_web); create index xwd_name on xwikiobjects (xwo_name); create index xwl_value on xwikilongs (xwl_value); create index xwi_value on xwikiintegers (xwi_value); create index xws_value on xwikistrings (xws_value); create index xwl_value on xwikilargestrings (xwl_value(50)); create index xwo_classname on xwikiobjects (xwo_classname); create index xwd_creation_date on xwikidoc (xwd_creation_date); create index xwd_date on xwikidoc (xwd_date); create index xwd_content_update_date on xwikidoc (xwd_content_update_date); create index xwd_content_author on xwikidoc (xwd_content_author); create index xwd_author on xwikidoc (xwd_author); create index xwd_creator on xwikidoc (xwd_creator); create index xwd_language on xwikidoc (xwd_language); create index xwd_default_language on xwikidoc (xwd_default_language); create index xwd_title on xwikidoc (xwd_title); create index xwd_parent on xwikidoc (xwd_parent(50)); Note that we aready have a jira issue for this (http://jira.xwiki.org/ jira/browse/XWIKI-605). We just don't know yet if we can have a generic script that'll work for all databases. There might be other issues but I'll let others with experience with big XWiki instances answer... Thanks -Vincent On Feb 7, 2007, at 3:00 AM, Youcef BEY wrote:

Hi all,

I have until now not solved this problem (since more than 10 month):

- using this program to create automatically a document work well for few documents:

XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter ("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext);

When putting these instructions in a LOOP for 10000 entries to transfer from another TABLE in the same XWiki database cause a problem. The probleme is not an error but the XWiki crash! and without finishing the LOOP, the xwiki display error "out of memory!"

But now, taking the same program and run it in the shell (javac, java) to transfert these 10000 entries to TABLE in the XWiki database, it's done in less 1 minute?

Strange for me ...

- More ...and this is the more importante thing :

One time I success to transfer 25 000 documents in the XWiki database (using the same script above) the XWiki become very heavy and cannot really work with it at all.

MySql tables are supposed to contain millions of records, and when using the database adminstrator and check with SQL statment, I get resultat in few seconds, that means that the problem is in XWiki.

My Question : Why with 25 000 documents XWiki don't work as usually

Anyone faced this problem before? Do you have made testing of XWiki for 1 millions documents for exemple?

I have XP, 1.2 MHz and 500 Mo but the following experience on MySQL gives more arguments that not MySQL is causing this problem :

(Web link : http://www.guilde.asso.fr/lurker/message/ 20050817.183426.a1b8e267.en.html)

----------------------------------------------------------------- Il y a quelques années, j'ai utilisé MySQL pour gérér 1.7 millions d'enregistrements dans une table. La table faisait environ 500Mo, plus 400Mo d'indexes pour la recherche de textes dans un des champs.

Coté perfs, sur un ordinateur portable à 500Mhz et 128Mo de ram qui faisait tourner un Windows NT4 bourré d'applis en mémoire, il fallait environ 5s à 10s pour recherche les tuples contenant certains mots- clefs stokés dans le fameux champs de texte.

Bref, des performances tout à fait acceptable, sur une machine qui n'était pourtant pas du tout fait pour faire serveur de base de données (et je ne parle pas que de l'OS...). --------------------------------------------------------------

Many thanks for any suggestion.

Best wishes ---------------------------------- Youcef Bey

-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws

___________________________________________________________________________ Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire. http://fr.mail.yahoo.com

Hi Youcef, Have you modified the configuration for memory in Java from the default of 64M. You might want to setup at least 256M. If you have a lot of documents you might want to increase the number of cached docs in xwiki.cfg also Ludovic Youcef BEY a écrit :

Hi Vincent, Jeremi and Sebastien

Thank you very much for your reply and I apologize for late reply.

Your solution is not really enough for sloving the problem. Sorry for that.

In fact, I created all the index so it seems that xwiki increase a little the speed but not really enough.

I have compilted for testing it 10000 but xwiki become unusable!

I did not use class in my document. The format of document is textual and small size (<2 Ko)

Sebastien, I will send you by tomorrow a file containing 350000 entries and all programs. As now I am transfering all my data and XWiki to Vine Linux (Debian) and also decided to leave Win XP for ever, I really enjoyed Linux :-)

When I installed XWiki on my Linux machine (Vine Linux) I have got an error. Even I have an experience with XWiki but this error can't find any issues for solving it. I prefer to send it on another email

Thank again.

Youcef

From: "Vincent Massol" <[email protected]> To: <[email protected]> Sent: Tuesday, February 06, 2007 1:22 AM Subject: Re: [xwiki-dev] Number of document decrease XWiki performance

Hi Youcef,

(Please do not hijack existing thread, this is bad practice... :-)).

Regarding your issue you have to set up indexes if you're working with big tables (these are not set automatically as of now).

I'm not sure exactly what indexes need to be set but here's some idea of what should be done (people in the know please confirm):

create index xwd_name on xwikidoc (xwd_name); create index xwd_fullname on xwikidoc (xwd_fullname); create index xwd_web on xwikidoc (xwd_web); create index xwd_name on xwikiobjects (xwo_name); create index xwl_value on xwikilongs (xwl_value); create index xwi_value on xwikiintegers (xwi_value); create index xws_value on xwikistrings (xws_value); create index xwl_value on xwikilargestrings (xwl_value(50)); create index xwo_classname on xwikiobjects (xwo_classname); create index xwd_creation_date on xwikidoc (xwd_creation_date); create index xwd_date on xwikidoc (xwd_date); create index xwd_content_update_date on xwikidoc (xwd_content_update_date); create index xwd_content_author on xwikidoc (xwd_content_author); create index xwd_author on xwikidoc (xwd_author); create index xwd_creator on xwikidoc (xwd_creator); create index xwd_language on xwikidoc (xwd_language); create index xwd_default_language on xwikidoc (xwd_default_language); create index xwd_title on xwikidoc (xwd_title); create index xwd_parent on xwikidoc (xwd_parent(50));

Note that we aready have a jira issue for this (http://jira.xwiki.org/ jira/browse/XWIKI-605). We just don't know yet if we can have a generic script that'll work for all databases.

There might be other issues but I'll let others with experience with big XWiki instances answer...

Thanks -Vincent

On Feb 7, 2007, at 3:00 AM, Youcef BEY wrote:

...

Hi all,

I have until now not solved this problem (since more than 10 month):

- using this program to create automatically a document work well for few documents:

XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter ("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext);

When putting these instructions in a LOOP for 10000 entries to transfer from another TABLE in the same XWiki database cause a problem. The probleme is not an error but the XWiki crash! and without finishing the LOOP, the xwiki display error "out of memory!"

But now, taking the same program and run it in the shell (javac, java) to transfert these 10000 entries to TABLE in the XWiki database, it's done in less 1 minute?

Strange for me ...

- More ...and this is the more importante thing :

One time I success to transfer 25 000 documents in the XWiki database (using the same script above) the XWiki become very heavy and cannot really work with it at all.

MySql tables are supposed to contain millions of records, and when using the database adminstrator and check with SQL statment, I get resultat in few seconds, that means that the problem is in XWiki.

My Question : Why with 25 000 documents XWiki don't work as usually

Anyone faced this problem before? Do you have made testing of XWiki for 1 millions documents for exemple?

I have XP, 1.2 MHz and 500 Mo but the following experience on MySQL gives more arguments that not MySQL is causing this problem :

(Web link : http://www.guilde.asso.fr/lurker/message/ 20050817.183426.a1b8e267.en.html)

----------------------------------------------------------------- Il y a quelques années, j'ai utilisé MySQL pour gérér 1.7 millions d'enregistrements dans une table. La table faisait environ 500Mo, plus 400Mo d'indexes pour la recherche de textes dans un des champs.

Coté perfs, sur un ordinateur portable à 500Mhz et 128Mo de ram qui faisait tourner un Windows NT4 bourré d'applis en mémoire, il fallait environ 5s à 10s pour recherche les tuples contenant certains mots- clefs stokés dans le fameux champs de texte.

Bref, des performances tout à fait acceptable, sur une machine qui n'était pourtant pas du tout fait pour faire serveur de base de données (et je ne parle pas que de l'OS...). --------------------------------------------------------------

Many thanks for any suggestion.

Best wishes ---------------------------------- Youcef Bey

-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws

___________________________________________________________________________

Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire. http://fr.mail.yahoo.com

--------------------------------------------------------------------------------

...

-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

------------------------------------------------------------------------

-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

-- Ludovic Dubost Blog: http://www.ludovic.org/blog/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost AIM: nvludo Yahoo: ludovic

On 2/9/07, bey <[email protected]> wrote:

Hi Luduvic,

Thank you very much for your email.

Yes, I checked these options.

- I set the temporary java memory to 1 Go and increase the number of files to 1000000 in xml.cfg. Unfortunately, this configuration do not solve the problem.

- With the MySql administrator : a simple request for displaying around 10000 entries is done in few seconds and when deleting them it takes 5.0459 sec.

- Deleting one document in XWiki containing 10000 documents is done in 10 minutes, Strange...!

- The importation of a XWiki database with 13369 entries takes a few seconds from my second XWiki using an external database administrator.

- I created the indexes that Vincent suggested me to do. I remarked an increasing of speed but not really significant.

ps. I deal with data using UTF-8 encoding : Jetty CharSet = UTF-8, SQL DB with collation "utf8_general_ci", XWiki CharSet = UTF-8, etc.

I run such kind of tasks on WinXP (1 Ghz, 500 Mo RAM, 20 Go HD) and also with last version of Debian (2,5 Ghz, 2 Go RAM and 100 Go HD).

This is what I got in the Shell during the work of XWiki:

-----------------------------------------------------------

WARN P1-16 http://localhost:8080/xwiki/bin/delete/A.H/WebHome?confirm=1& RegexTokenFilter:filter:99 - <span class="error">Error</span>: com.xpn.xwiki.render.filter.XWikiHeadingFilter@3bc63: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 18:15:26.764 INFO [P1-14] org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134)

...

55> Velocity [error] Left side ($!hidecolumns) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. [line 9, column 21] 18:15:26.766 INFO [P1-14] org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134) 39> Velocity [error] RHS of #set statement is null. Context will not be modified. [line 11, column 1]

-----------------------------------------------------------

Any suggestion will be deeply appreciated?

ok, so the problem is what ludovic says. increase the memory of your java.Look inside the catalina.sh to change the memory size, and increase it to at least 256M. it's a parameter (looks like this i think : "-X64m") to java. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: [email protected] http://www.xwiki.com - http://www.pengyou-project.info

Hi All, Sorry for the late reply and thank you very much. Yes,I checked these options and it solved the problem of speed and the problem of java heap management as well. The exact parameters are : -Xms512m -Xmx521m (tuned for my machine) Thanks again. Cheers ------------- Youcef Bey jeremi joslin a écrit :

On 2/9/07, bey <[email protected]> wrote:

...

Hi Luduvic,

Thank you very much for your email.

Yes, I checked these options.

- I set the temporary java memory to 1 Go and increase the number of files to 1000000 in xml.cfg. Unfortunately, this configuration do not solve the problem.

- With the MySql administrator : a simple request for displaying around 10000 entries is done in few seconds and when deleting them it takes 5.0459 sec.

- Deleting one document in XWiki containing 10000 documents is done in 10 minutes, Strange...!

- The importation of a XWiki database with 13369 entries takes a few seconds from my second XWiki using an external database administrator.

- I created the indexes that Vincent suggested me to do. I remarked an increasing of speed but not really significant.

ps. I deal with data using UTF-8 encoding : Jetty CharSet = UTF-8, SQL DB with collation "utf8_general_ci", XWiki CharSet = UTF-8, etc.

I run such kind of tasks on WinXP (1 Ghz, 500 Mo RAM, 20 Go HD) and also with last version of Debian (2,5 Ghz, 2 Go RAM and 100 Go HD).

This is what I got in the Shell during the work of XWiki:

-----------------------------------------------------------

WARN P1-16 http://localhost:8080/xwiki/bin/delete/A.H/WebHome?confirm=1& RegexTokenFilter:filter:99 - <span class="error">Error</span>: com.xpn.xwiki.render.filter.XWikiHeadingFilter@3bc63: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 18:15:26.764 INFO [P1-14] org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134)

...

55> Velocity [error] Left side ($!hidecolumns) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. [line 9, column 21] 18:15:26.766 INFO [P1-14] org.mortbay.jetty.servlet.ServletHandler$Context.log(ServletHandler.java:1134)

...

39> Velocity [error] RHS of #set statement is null. Context will not be modified. [line 11, column 1]

-----------------------------------------------------------

Any suggestion will be deeply appreciated?

ok, so the problem is what ludovic says. increase the memory of your java.Look inside the catalina.sh to change the memory size, and increase it to at least 256M. it's a parameter (looks like this i think : "-X64m") to java.

jeremi

------------------------------------------------------------------------

-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Youcef BEY a écrit :

Hi all,

Hi Youcef,

- using this program to create automatically a document work well for few documents:

XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext);

could you please send us the real script you are using ? (I mean as much 'real' as possible).

LOOP, the xwiki display error "out of memory!"

have you got a stack trace ?

But now, taking the same program and run it in the shell (javac, java) to transfert these 10000 entries to TABLE in the XWiki database, it's done in less 1 minute?

could you please send us the source of this ? I'm generating wikis automatically using XWiki apis, the biggest wiki created hold 165 000 pages (each page containing between one to 5 objects). the only problem I got was about the archive handling that I had to disable (which is not a problem since the pages are generated and not edited), this was with an old XWiki version (svn 1226), since then the archive system relies on another framework, so this may be not an issue anymore (we are still using the same XWiki version). I'm using only 4 indexes (I will post a reply tomorrow with the indexed columns, I don't remember exactly and I can't access this information right now). These indexes have greatly improved response time (at first the biggest page (2,5 MB!) took 25 minutes to render ;-) after indexes creation, and some other tweaking, it took only a few seconds). indexes will improve response time, but it will not protect you from an out of memory exception. if you can send the scripts you are using to the list, I will be able to run them on our wikis to see the result ... Seb.

Hi Sebastien, I am really sorry for the late reply. Now it remain this problem : it remain that the script used to transfer data from an internal table (I created it) to the XWiki not continu to its end. I have about 200000 entries but the program transfers only 52000 entries without any error message (program is finished but data are not all transfered in the XWiki database). Sébastien Gaïde a écrit :

...

XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext);

- using this program to create automatically a document work well for few documents:

could you please send us the real script you are using ? (I mean as much 'real' as possible).

Here it is : <form action="$doc.name" enctype="multipart/form-data" method="get"> Space <input type="text" name="web" value="" size="20" /> <input type="submit" value="Transfert" onclick="form.datafile.value = form.filepath.value"/> </form> <% import com.xpn.xwiki.doc.XWikiDocument import java.io.* import java.sql.* import java.lang.* if (request.getParameter("web") != null)){ // information access String user = "xwiki"; String password = "xwiki"; // connection : the table name is eijiro : eword (text type) and jword (text type) (encoding UTF-8) String url = "jdbc:mysql://localhost/xwiki" + "?user=" + user + "&password=" + password + "&autoReconnect=true" + "&characterEncoding=utf8"; conn = DriverManager.getConnection(url, user, password) Statement stmt = conn.createStatement() // get records SQLstat = "select * from eijiro"; ResultSet st = stmt.executeQuery(SQLstat); recordnbre = 0; while (st.next()){ SourceLangue = "en" SourceValue = st.getString('eword') ExpressionKataPronunciation = "?" TargetLangue = "jp" ExpressionValue = st.getString('jword'); // several translation are possible! EntryContent = "* *Headword* ("+SourceLangue+") \n{code} "+ SourceValue + "{code}\n * *Translation* ("+TargetLangue+") \n{code} " // document creation XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext); recordnbre++ } } %> ===============================END

...

LOOP, the xwiki display error "out of memory!"

have you got a stack trace ? No, sorry for that.

...

But now, taking the same program and run it in the shell (javac, java) to transfert these 10000 entries to TABLE in the XWiki database, it's done in less 1 minute?

could you please send us the source of this ?

Here it is : import java.io.*; import java.sql.*; import java.lang.*; /** the table name is eijiro : eword (text type) and jword (text type) */ public class toDBs{ public static void main(String[] args){ // dictionary data text file [eword,jwords[jw1<\t>jw2..]<\n>] String file = "./utf8-gdbm.txt"; File LD_File; FileInputStream Stream_LD=null; BufferedReader Data_LD = null; Reader IR_LD = null; String Line_LD = ""; Connection conn = null; //System.out.println("before"); // database connect data String user = "xwiki"; String password = "xwiki"; String url = "jdbc:mysql://localhost/xwiki" + "?user=" + user + "&password=" + password + "&autoReconnect=true" + "&characterEncoding=utf8"; try{ LD_File = new File(file); Stream_LD = new FileInputStream(LD_File); IR_LD = new InputStreamReader(Stream_LD,"UTF-8"); Data_LD = new BufferedReader(IR_LD); Class.forName( "com.mysql.jdbc.Driver" ).newInstance(); conn = DriverManager.getConnection(url, user, password); Statement stmt = conn.createStatement(); String line; String eword; String jword; int cnt = 0; // read line System.out.println("before"); float compter = 0; while( ( line = Data_LD.readLine() ) != null) { // br.readLine() // split string is "," compter++; String[] split = line.split( "," ); // "'" convert to "\'" eword = split[0].replaceAll( "'", "\\\\'" ); jword = split[1].replaceAll( "'", "\\\\'" ); // insert to database String MySQLStat = "insert into eijiro values ( '"+eword+"', '"+jword+"','')"; stmt.executeUpdate( MySQLStat); //stmt.executeUpdate( "insert into TriaxSEijiro values ( '" + compter + "', '" +eword+ "', '" + "en" + "','"+ "neant" + "')" ); //stmt.executeUpdate( "insert into TriaxTEijiro values ( '" + compter + "', '" +compter+ "', '" +"1"+ "', '" +jword+ "','"+"jp"+ "' )" ); } // close Data_LD.close(); stmt.close(); conn.close(); } catch (Exception e){ System.out.println(e); if (e instanceof SQLException){ System.out.println("SQLException: " + e.getMessage()); System.out.println("SQLState: " + ((SQLException)e).getSQLState()); System.out.println("Code: " + ((SQLException)e).getErrorCode()); } } finally{ if (conn != null){ try{ conn.close(); }catch (Exception e){ System.out.println("SQLException: " + e.getMessage()); System.out.println("SQLState: " + ((SQLException)e).getSQLState()); System.out.println("Code: " + ((SQLException)e).getErrorCode()); } } } } } ==============================================END

I'm generating wikis automatically using XWiki apis, the biggest wiki created hold 165 000 pages (each page containing between one to 5 objects). the only problem I got was about the archive handling that I had to disable (which is not a problem since the pages are generated and not edited), this was with an old XWiki version (svn 1226), since then the archive system relies on another framework, so this may be not an issue anymore (we are still using the same XWiki version).

This kind of functionalities interest me, I would like that you lead me how to do that? Is that transfer done by an external program or an integrated script in the core of XWiki? In fact, I tried to transfer 300000 entries in the doc table of XWiki database (I fill all data and also created and unique ID of each document). The only problem was with the magic ID of document that XWiki API create:-)! When I take randomly any document that I have put in the doc table and changing its ID by an existing ID (of another document) that have been generated automaticaly by XWiki (e.g created in the environment), so the considered document can be detected and managed by XWiki, else no document of 300000 that exists in the doc table have been detected by XWiki. I read the code source but I did not understand the method that have been followed for the generation of the ID?

I'm using only 4 indexes (I will post a reply tomorrow with the indexed columns, I don't remember exactly and I can't access this information right now). These indexes have greatly improved response time (at first the biggest page (2,5 MB!) took 25 minutes to render ;-) after indexes creation, and some other tweaking, it took only a few seconds).

indexes will improve response time, but it will not protect you from an out of memory exception.

Please let me know these indexes... Many thanks in advance. Cheers -------------------------- Youcef Bey

------------------------------------------------------------------------

-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws