Hello Vincent, To add to the alternatives you presented, you could even use proper xar (http://code.google.com/p/xar/). It is actually a quite nice archive format, and it does have support for metadata, checksums, table of contents etc. On the other hand, it is not an established archive format, and I don't think there is a Java implementation for it, so the overhead of implementing this would be a lot more substantial than using zip or jar. Catalin On 2/3/07, Vincent Massol <vincent(a)massol.net> wrote:

Hi Jeremi, On Feb 4, 2007, at 4:44 PM, jeremi joslin wrote: A JAR isn't a Zip in Java term. It's using the same compression algorithm. It's very different in term of metadata. I won't go in more details but you can check the JAR specification to know more about this. It's possible but harder. For example to read a JAR inside a XAR you'll probably need to write and register special handlers against various classes (such as URL factories, etc). No, xwiki isn't the only one. All tools will be able to read a JAR or a ZIP (be them IDE, build tools, desktop tools, etc). I'm not sure what you call a normal jar :-) You probably means classes. However a JAR is not meant to contain only classes. It's a generic format to contain any other kind of resources. The nice thing it has though is support for dependencies and class-paths. As I said renaming to zip is a solution. Probably a better one than the current XAR. However the pity is that a zip doesn't support metadata by default. You need to define them, document them, teach them, write parsers for them, etc. JARs have this already (this is called the Manifest). No. The JDK provides an API for reading information from the manifest. And it's possibly buggy :-) And you're the only one who knows about it. And it isn't documented. And you haven't implemented it for Maven2 (it's not easy and has some issues) nor any of the tool that will crop up. The question is not that it's done (it's not finished BTW - No doc, no tests). It's about removing code and using standards (to reduce cost even further). Any line of code that is removed saves a lot (fixing bug, documentation, test, teaching, etc). No that's not true. It's a problem for a lot of tools (Maven2 included). BTW I'm on Mac OS X (which isn't windows) and I can't open a XAR either. I have to rename it to ZIP before... Unfortunately there are very few tools that look inside a file to know what format it is. Most will simply look at the file extension. Sure that's ideal. Problem is that you always have to do this... For example there's no selective export so you have to expand it as a zip, remove files, edit the package.xml file and repackage. Oh, because I think it's an improvement (it certainly is at least for issues I have with Maven2 and it'll win us hundreds of lines of code). See below for explanations on lightweight container support. I didn't know about the other XAR format and it wouldn't be a good choice (not known enough and thus not enough tool support). I'm not going to push for the format change because we have a lot of work to be done right now and I'm not proposing to do it myself either. What I was just saying here is that I agree that inventing a format is probably not the best solution and reusing an existing well- known format with metadata support is probably a better choice. JAR is such a choice. It looks to me like a good choice, especially considering that we have a need to put java classes in applications. I see you're against it. That's fine. It means it's not a clear cut decision and we should think more about it. In any case as I said above we have more urgent problems. We'll need to revisit this when we move to full fledge application support. These will contain: - pages - java classes (plugins, macros, etc) - external libs - resources (HTML, js files, images, etc). For example a GWT XWiki application will contain java and resources + the necessary pages and possibly external libs (we'll probably bundle them though). The IRC Bot will require an external lib for sure. We need to revisit this in view with the new architecture (component- based architecture based on a microkernel, aka lightweight container). You'll see that most of these containers use the JAR format (OSGi, Plexus, etc) so we'll probably have to use the JAR format anyway... :-) Thanks -Vincent ___________________________________________________________________________ Yahoo! Mail r�invente le mail ! D�couvrez le nouveau Yahoo! Mail et son interface r�volutionnaire. http://fr.mail.yahoo.com

ok 0, but just not to block the decision. do as you want. but I don't see the difference between writing in an XML file or a manifest. You will both have to document it. I don't like manifest because AFAIK it's a plain text file and not an xml. You will have other problems I think. A xar is not a zip in XWiki term :-) XAR is just an IMPORT/EXPORT format so you will not need this. you just need to extract the jar from it! No, xwiki isn't the only one. All tools will be able to read a JAR or a ZIP (be them IDE, build tools, desktop tools, etc). When I've done we were not working with maven, but we were with ant. and it's implemented in ant (by ludovic). I'm not he only one who know how the xar is working. No, I've done one, and Vincent you know it. ... that can be a long discussion. Do as you want, i will not argue anymore on this. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: jeremi23(a)gmail.com http://www.xwiki.com - http://www.pengyou-project.info

On 2/4/07, Vincent Massol <vincent(a)massol.net> wrote: Yes, but sometime it's lose of time. We have more important things for now to work on. I agree we will have to step back as you say after 1.0. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: jeremi23(a)gmail.com http://www.xwiki.com - http://www.pengyou-project.info

Hi Youcef, (Please do not hijack existing thread, this is bad practice... :-)). Regarding your issue you have to set up indexes if you're working with big tables (these are not set automatically as of now). I'm not sure exactly what indexes need to be set but here's some idea of what should be done (people in the know please confirm): create index xwd_name on xwikidoc (xwd_name); create index xwd_fullname on xwikidoc (xwd_fullname); create index xwd_web on xwikidoc (xwd_web); create index xwd_name on xwikiobjects (xwo_name); create index xwl_value on xwikilongs (xwl_value); create index xwi_value on xwikiintegers (xwi_value); create index xws_value on xwikistrings (xws_value); create index xwl_value on xwikilargestrings (xwl_value(50)); create index xwo_classname on xwikiobjects (xwo_classname); create index xwd_creation_date on xwikidoc (xwd_creation_date); create index xwd_date on xwikidoc (xwd_date); create index xwd_content_update_date on xwikidoc (xwd_content_update_date); create index xwd_content_author on xwikidoc (xwd_content_author); create index xwd_author on xwikidoc (xwd_author); create index xwd_creator on xwikidoc (xwd_creator); create index xwd_language on xwikidoc (xwd_language); create index xwd_default_language on xwikidoc (xwd_default_language); create index xwd_title on xwikidoc (xwd_title); create index xwd_parent on xwikidoc (xwd_parent(50)); Note that we aready have a jira issue for this (http://jira.xwiki.org/ jira/browse/XWIKI-605). We just don't know yet if we can have a generic script that'll work for all databases. There might be other issues but I'll let others with experience with big XWiki instances answer... Thanks -Vincent On Feb 7, 2007, at 3:00 AM, Youcef BEY wrote: ___________________________________________________________________________ Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire. http://fr.mail.yahoo.com

Hi Youcef, Have you modified the configuration for memory in Java from the default of 64M. You might want to setup at least 256M. If you have a lot of documents you might want to increase the number of cached docs in xwiki.cfg also Ludovic Youcef BEY a écrit : -- Ludovic Dubost Blog: http://www.ludovic.org/blog/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost AIM: nvludo Yahoo: ludovic

On 2/9/07, bey <youcef.bey(a)imag.fr> wrote: ok, so the problem is what ludovic says. increase the memory of your java.Look inside the catalina.sh to change the memory size, and increase it to at least 256M. it's a parameter (looks like this i think : "-X64m") to java. jeremi -- Jeremi Joslin (http://www.jeremi.info) skype: jeremi23 - jabber: jeremi23(a)gmail.com http://www.xwiki.com - http://www.pengyou-project.info

Hi All, Sorry for the late reply and thank you very much. Yes,I checked these options and it solved the problem of speed and the problem of java heap management as well. The exact parameters are : -Xms512m -Xmx521m (tuned for my machine) Thanks again. Cheers ------------- Youcef Bey jeremi joslin a écrit :

Youcef BEY a écrit : Hi Youcef, could you please send us the real script you are using ? (I mean as much 'real' as possible). have you got a stack trace ? could you please send us the source of this ? I'm generating wikis automatically using XWiki apis, the biggest wiki created hold 165 000 pages (each page containing between one to 5 objects). the only problem I got was about the archive handling that I had to disable (which is not a problem since the pages are generated and not edited), this was with an old XWiki version (svn 1226), since then the archive system relies on another framework, so this may be not an issue anymore (we are still using the same XWiki version). I'm using only 4 indexes (I will post a reply tomorrow with the indexed columns, I don't remember exactly and I can't access this information right now). These indexes have greatly improved response time (at first the biggest page (2,5 MB!) took 25 minutes to render ;-) after indexes creation, and some other tweaking, it took only a few seconds). indexes will improve response time, but it will not protect you from an out of memory exception. if you can send the scripts you are using to the list, I will be able to run them on our wikis to see the result ... Seb.

Hi Sebastien, I am really sorry for the late reply. Now it remain this problem : it remain that the script used to transfer data from an internal table (I created it) to the XWiki not continu to its end. I have about 200000 entries but the program transfers only 52000 entries without any error message (program is finished but data are not all transfered in the XWiki database). Sébastien Gaïde a écrit : Here it is : <form action="$doc.name" enctype="multipart/form-data" method="get"> Space <input type="text" name="web" value="" size="20" /> <input type="submit" value="Transfert" onclick="form.datafile.value = form.filepath.value"/> </form> <% import com.xpn.xwiki.doc.XWikiDocument import java.io.* import java.sql.* import java.lang.* if (request.getParameter("web") != null)){ // information access String user = "xwiki"; String password = "xwiki"; // connection : the table name is eijiro : eword (text type) and jword (text type) (encoding UTF-8) String url = "jdbc:mysql://localhost/xwiki" + "?user=" + user + "&password=" + password + "&autoReconnect=true" + "&characterEncoding=utf8"; conn = DriverManager.getConnection(url, user, password) Statement stmt = conn.createStatement() // get records SQLstat = "select * from eijiro"; ResultSet st = stmt.executeQuery(SQLstat); recordnbre = 0; while (st.next()){ SourceLangue = "en" SourceValue = st.getString('eword') ExpressionKataPronunciation = "?" TargetLangue = "jp" ExpressionValue = st.getString('jword'); // several translation are possible! EntryContent = "* *Headword* ("+SourceLangue+") \n{code} "+ SourceValue + "{code}\n * *Translation* ("+TargetLangue+") \n{code} " // document creation XW = xwiki.XWiki; XWContext = context.context; XWikiDocument NewEntry = new XWikiDocument(request.getParameter("web"), SourceValue); NewEntry.setContent(EntryContent); XW.saveDocument(NewEntry, XWContext); recordnbre++ } } %> ===============================END No, sorry for that. Here it is : import java.io.*; import java.sql.*; import java.lang.*; /** the table name is eijiro : eword (text type) and jword (text type) */ public class toDBs{ public static void main(String[] args){ // dictionary data text file [eword,jwords[jw1<\t>jw2..]<\n>] String file = "./utf8-gdbm.txt"; File LD_File; FileInputStream Stream_LD=null; BufferedReader Data_LD = null; Reader IR_LD = null; String Line_LD = ""; Connection conn = null; //System.out.println("before"); // database connect data String user = "xwiki"; String password = "xwiki"; String url = "jdbc:mysql://localhost/xwiki" + "?user=" + user + "&password=" + password + "&autoReconnect=true" + "&characterEncoding=utf8"; try{ LD_File = new File(file); Stream_LD = new FileInputStream(LD_File); IR_LD = new InputStreamReader(Stream_LD,"UTF-8"); Data_LD = new BufferedReader(IR_LD); Class.forName( "com.mysql.jdbc.Driver" ).newInstance(); conn = DriverManager.getConnection(url, user, password); Statement stmt = conn.createStatement(); String line; String eword; String jword; int cnt = 0; // read line System.out.println("before"); float compter = 0; while( ( line = Data_LD.readLine() ) != null) { // br.readLine() // split string is "," compter++; String[] split = line.split( "," ); // "'" convert to "\'" eword = split[0].replaceAll( "'", "\\\\'" ); jword = split[1].replaceAll( "'", "\\\\'" ); // insert to database String MySQLStat = "insert into eijiro values ( '"+eword+"', '"+jword+"','')"; stmt.executeUpdate( MySQLStat); //stmt.executeUpdate( "insert into TriaxSEijiro values ( '" + compter + "', '" +eword+ "', '" + "en" + "','"+ "neant" + "')" ); //stmt.executeUpdate( "insert into TriaxTEijiro values ( '" + compter + "', '" +compter+ "', '" +"1"+ "', '" +jword+ "','"+"jp"+ "' )" ); } // close Data_LD.close(); stmt.close(); conn.close(); } catch (Exception e){ System.out.println(e); if (e instanceof SQLException){ System.out.println("SQLException: " + e.getMessage()); System.out.println("SQLState: " + ((SQLException)e).getSQLState()); System.out.println("Code: " + ((SQLException)e).getErrorCode()); } } finally{ if (conn != null){ try{ conn.close(); }catch (Exception e){ System.out.println("SQLException: " + e.getMessage()); System.out.println("SQLState: " + ((SQLException)e).getSQLState()); System.out.println("Code: " + ((SQLException)e).getErrorCode()); } } } } } ==============================================END This kind of functionalities interest me, I would like that you lead me how to do that? Is that transfer done by an external program or an integrated script in the core of XWiki? In fact, I tried to transfer 300000 entries in the doc table of XWiki database (I fill all data and also created and unique ID of each document). The only problem was with the magic ID of document that XWiki API create:-)! When I take randomly any document that I have put in the doc table and changing its ID by an existing ID (of another document) that have been generated automaticaly by XWiki (e.g created in the environment), so the considered document can be detected and managed by XWiki, else no document of 300000 that exists in the doc table have been detected by XWiki. I read the code source but I did not understand the method that have been followed for the generation of the ID? Please let me know these indexes... Many thanks in advance. Cheers -------------------------- Youcef Bey