UTF-8 characters (again)
Hi. Is anybody able to write/read any non-latin1 documents in xwiki? I'm trying with XWiki Version 0.9.793 and a new default database. I'v changed default values to UTF-8 in: WEB-INF/xwiki.cfg: xwiki.encoding WEB-INF/web.xml: init param 'encoding' in 'Set Character Encoding' filter XWiki.Preferences -> Presentation: http-equiv meta header Tomcat 4.1.30 is using JDK 1.4.2, running in locale en_US. The behaviour is as follows: 1. I'm creating a new document, entering some non-latin1 chars. 2. I can preview, save and edit this document - all works fine 3. I'm restarting tomcat serving xwiki 4. I cannot display the created document - an exception occurs: $msg.get("problemoccured") $msg.get("detailedinformation"): Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:638) at com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:123) [...] Wrapped Exception: org.hibernate.PropertyAccessException: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:203) at org.hibernate.tuple.PojoTuplizer.setPropertyValues(PojoTuplizer.java:173) at org.hibernate.persister.entity.BasicEntityPersister.setPropertyValues(BasicEntityPersister.java:2900) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:510) [...] Caused by: net.sf.cglib.beans.BulkBeanException: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562792127\n1120562808542\n1.2\n" at com.xpn.xwiki.doc.XWikiDocument$$BulkBeanByCGLIB$$9e5f0943.setPropertyValues() at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:200) ... 63 more Caused by: org.apache.commons.jrcs.rcs.TokenMgrError: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562792127\n1120562808542\n1.2\n" at org.apache.commons.jrcs.rcs.ArchiveParserTokenManager.getNextToken(ArchiveParserTokenManager.java:800) at org.apache.commons.jrcs.rcs.ArchiveParser.jj_consume_token(ArchiveParser.java:558) at org.apache.commons.jrcs.rcs.ArchiveParser.text(ArchiveParser.java:429) at org.apache.commons.jrcs.rcs.ArchiveParser.archive(ArchiveParser.java:102) at org.apache.commons.jrcs.rcs.ArchiveParser.load(ArchiveParser.java:55) at org.apache.commons.jrcs.rcs.Archive.(Archive.java:230) at com.xpn.xwiki.doc.XWikiDocument.setArchive(XWikiDocument.java:475) ... 65 more When I was using previous version of xwiki (probably with different settings) all non-latin1 characters were converted into html-entities; If I copy-paste such a document into this new installation - it looks like it should look, but newly entered characters appear in database as proper utf-8 characters (which in my opinion is fine - database is set to utf-8), and probably cannot be properly read by rendering module. Does anybody have an idea if I can reconfigure something to make it work? TIA, Richard. -- "First they ignore you. Then they laugh at you. Then they fight you. Then you win." - Mohandas Gandhi.
xWiki works with UTF-8 Characters after some modifications, live example is on http://zh.objectweb.org There are some modifications to do in the code of xWiki, you need to configure xWiki to manage UTF-8 and to use a database without non UTF-8 code. If you want I can send you my xWiki package. Regards -- Xavier MOGHRABI - Consortium ObjectWeb Email: xavier.moghrabi at objectweb.org Phone: +33 4 76 61 52 35 - Skype ID: xavier.moghrabi.bureau Le Mardi 5 Juillet 2005 13:42, Ryszard Lach a écrit :
Hi.
Is anybody able to write/read any non-latin1 documents in xwiki?
I'm trying with XWiki Version 0.9.793 and a new default database. I'v changed default values to UTF-8 in:
WEB-INF/xwiki.cfg: xwiki.encoding WEB-INF/web.xml: init param 'encoding' in 'Set Character Encoding' filter XWiki.Preferences -> Presentation: http-equiv meta header
Tomcat 4.1.30 is using JDK 1.4.2, running in locale en_US.
The behaviour is as follows:
1. I'm creating a new document, entering some non-latin1 chars. 2. I can preview, save and edit this document - all works fine 3. I'm restarting tomcat serving xwiki 4. I cannot display the created document - an exception occurs:
$msg.get("problemoccured")
$msg.get("detailedinformation"):
Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.ja va:638) at com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:123) [...] Wrapped Exception: org.hibernate.PropertyAccessException: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplize r.java:203) at org.hibernate.tuple.PojoTuplizer.setPropertyValues(PojoTuplizer.java:173) at org.hibernate.persister.entity.BasicEntityPersister.setPropertyValues(Basic EntityPersister.java:2900) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:51 0) [...] Caused by: net.sf.cglib.beans.BulkBeanException: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562 792127\n1120562808542\n1.2\n" at com.xpn.xwiki.doc.XWikiDocument$$BulkBeanByCGLIB$$9e5f0943.setPropertyValue s() at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplize r.java:200) ... 63 more Caused by: org.apache.commons.jrcs.rcs.TokenMgrError: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562 792127\n1120562808542\n1.2\n" at org.apache.commons.jrcs.rcs.ArchiveParserTokenManager.getNextToken(ArchiveP arserTokenManager.java:800) at org.apache.commons.jrcs.rcs.ArchiveParser.jj_consume_token(ArchiveParser.ja va:558) at org.apache.commons.jrcs.rcs.ArchiveParser.text(ArchiveParser.java:429) at org.apache.commons.jrcs.rcs.ArchiveParser.archive(ArchiveParser.java:102) at org.apache.commons.jrcs.rcs.ArchiveParser.load(ArchiveParser.java:55) at org.apache.commons.jrcs.rcs.Archive.(Archive.java:230) at com.xpn.xwiki.doc.XWikiDocument.setArchive(XWikiDocument.java:475) ... 65 more
When I was using previous version of xwiki (probably with different settings) all non-latin1 characters were converted into html-entities; If I copy-paste such a document into this new installation - it looks like it should look, but newly entered characters appear in database as proper utf-8 characters (which in my opinion is fine - database is set to utf-8), and probably cannot be properly read by rendering module.
Does anybody have an idea if I can reconfigure something to make it work?
TIA,
Richard.
On Wed, Jul 06, 2005 at 10:16:53AM +0200, Xavier MOGHRABI wrote:
xWiki works with UTF-8 Characters after some modifications, live example is on http://zh.objectweb.org
There are some modifications to do in the code of xWiki, you need to configure xWiki to manage UTF-8 and to use a database without non UTF-8 code.
I've just noticed, that I forgot about the JRCS stuff (as described on http://www.xwiki.org/xwiki/bin/view/Dev/CharactersSets). I probably don't understand it, what should I do with attached files? Saving both jars to WEB-INF/lib causes an exception: root cause java.lang.UnsupportedClassVersionError: org/apache/commons/jrcs/rcs/Archive (Unsupported major.minor version 49.0) What am I doing wrong? -- "First they ignore you. Then they laugh at you. Then they fight you. Then you win." - Mohandas Gandhi.
Hi Xavier On 7/6/05, Xavier MOGHRABI <[email protected]> wrote:
xWiki works with UTF-8 Characters after some modifications, live example is on http://zh.objectweb.org
There are some modifications to do in the code of xWiki, you need to configure xWiki to manage UTF-8 and to use a database without non UTF-8 code.
I am using the xwiki-db-0.9.1.sql download from xwiki.org. I suppose that there is some non-UTF-8 code in it and something failed during the import into the MySql. I recall there is a way to clear the xwiki database and start from an empty wiki. But I can't found the link anymore. Do you know where that document is? Or do you have a database with UTF-8 code only. Thanks Patrick Lee
Hi I joined in attachment a database dump which should only have UTF-8 encoded characters. Regards -- Xavier MOGHRABI - Consortium ObjectWeb Email: xavier.moghrabi at objectweb.org Phone: +33 4 76 61 52 35 - Skype ID: xavier.moghrabi.bureau Le Mercredi 6 Juillet 2005 10:40, Patrick Lee a écrit :
Hi Xavier
On 7/6/05, Xavier MOGHRABI <[email protected]> wrote:
xWiki works with UTF-8 Characters after some modifications, live example is on http://zh.objectweb.org
There are some modifications to do in the code of xWiki, you need to configure xWiki to manage UTF-8 and to use a database without non UTF-8 code.
I am using the xwiki-db-0.9.1.sql download from xwiki.org. I suppose that there is some non-UTF-8 code in it and something failed during the import into the MySql.
I recall there is a way to clear the xwiki database and start from an empty wiki. But I can't found the link anymore. Do you know where that document is? Or do you have a database with UTF-8 code only.
Thanks Patrick Lee
Hi Xavier The blog, Doc.WebHome and the search are all functioning now!! Many, many thanks Patrick Lee On 7/6/05, Xavier MOGHRABI <[email protected]> wrote:
Hi I joined in attachment a database dump which should only have UTF-8 encoded characters.
Regards
-- Xavier MOGHRABI - Consortium ObjectWeb Email: xavier.moghrabi at objectweb.org Phone: +33 4 76 61 52 35 - Skype ID: xavier.moghrabi.bureau
Le Mercredi 6 Juillet 2005 10:40, Patrick Lee a écrit:
Hi Xavier
On 7/6/05, Xavier MOGHRABI <[email protected]> wrote:
xWiki works with UTF-8 Characters after some modifications, live example is on http://zh.objectweb.org
There are some modifications to do in the code of xWiki, you need to configure xWiki to manage UTF-8 and to use a database without non UTF-8 code.
I am using the xwiki-db-0.9.1.sql download from xwiki.org. I suppose that there is some non-UTF-8 code in it and something failed during the import into the MySql.
I recall there is a way to clear the xwiki database and start from an empty wiki. But I can't found the link anymore. Do you know where that document is? Or do you have a database with UTF-8 code only.
Thanks Patrick Lee
Finally, with great help from Xavier, I found the solution. The problem was missing JRCS jars and bad (1.4.2) version of JDK. I'll add a comment on page describing UTF. Best regards, R. On Tue, Jul 05, 2005 at 01:42:14PM +0200, Ryszard Lach wrote:
Hi.
Is anybody able to write/read any non-latin1 documents in xwiki?
I'm trying with XWiki Version 0.9.793 and a new default database. I'v changed default values to UTF-8 in:
WEB-INF/xwiki.cfg: xwiki.encoding WEB-INF/web.xml: init param 'encoding' in 'Set Character Encoding' filter XWiki.Preferences -> Presentation: http-equiv meta header
Tomcat 4.1.30 is using JDK 1.4.2, running in locale en_US.
The behaviour is as follows:
1. I'm creating a new document, entering some non-latin1 chars. 2. I can preview, save and edit this document - all works fine 3. I'm restarting tomcat serving xwiki 4. I cannot display the created document - an exception occurs:
$msg.get("problemoccured")
$msg.get("detailedinformation"):
Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:638) at com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:123) [...] Wrapped Exception: org.hibernate.PropertyAccessException: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:203) at org.hibernate.tuple.PojoTuplizer.setPropertyValues(PojoTuplizer.java:173) at org.hibernate.persister.entity.BasicEntityPersister.setPropertyValues(BasicEntityPersister.java:2900) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:510) [...] Caused by: net.sf.cglib.beans.BulkBeanException: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562792127\n1120562808542\n1.2\n" at com.xpn.xwiki.doc.XWikiDocument$$BulkBeanByCGLIB$$9e5f0943.setPropertyValues() at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:200) ... 63 more
-- "First they ignore you. Then they laugh at you. Then they fight you. Then you win." - Mohandas Gandhi.
Speaking of UTF-8 there is also the change from Stéphane. I think this change is made in Xavier's package.. It would be great to compile exactly what is need to make UTF8 work so that we can commit the changes in the source code and write the actual documentation for it. There is also the following change that Stéphane has made: ---- I have also commented out following lines of "com.xpn.xwiki.web.Utils.java", because content.length() does not return the UTF-8 encoded content length, which shortens the pages… Or converting the string to UTF-8 encoded bytes works fine, but is it absolutely necessary to set the content length of the response? if (context.getResponse() instanceof XWikiServletResponse) { response.setContentLength(content.length()); } ---- Actually it is much better to give the content length when you can because it allows the browser to use the keepalive connection which is more performant. This code needs to be rewrittent to calculate the accurate size regardless of the character set. Ludovic Ryszard Lach a écrit :
Finally, with great help from Xavier, I found the solution.
The problem was missing JRCS jars and bad (1.4.2) version of JDK. I'll add a comment on page describing UTF.
Best regards,
R.
On Tue, Jul 05, 2005 at 01:42:14PM +0200, Ryszard Lach wrote:
Hi.
Is anybody able to write/read any non-latin1 documents in xwiki?
I'm trying with XWiki Version 0.9.793 and a new default database. I'v changed default values to UTF-8 in:
WEB-INF/xwiki.cfg: xwiki.encoding WEB-INF/web.xml: init param 'encoding' in 'Set Character Encoding' filter XWiki.Preferences -> Presentation: http-equiv meta header
Tomcat 4.1.30 is using JDK 1.4.2, running in locale en_US.
The behaviour is as follows:
1. I'm creating a new document, entering some non-latin1 chars. 2. I can preview, save and edit this document - all works fine 3. I'm restarting tomcat serving xwiki 4. I cannot display the created document - an exception occurs:
$msg.get("problemoccured")
$msg.get("detailedinformation"):
Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive com.xpn.xwiki.XWikiException: Error number 3202 in 3: Exception while reading document Sandbox.TestTopic1 Wrapped Exception: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at com.xpn.xwiki.store.XWikiHibernateStore.loadXWikiDoc(XWikiHibernateStore.java:638) at com.xpn.xwiki.store.XWikiCacheStore.loadXWikiDoc(XWikiCacheStore.java:123) [...] Wrapped Exception: org.hibernate.PropertyAccessException: exception setting property value with CGLIB (set hibernate.cglib.use_reflection_optimizer=false for more info) setter of com.xpn.xwiki.doc.XWikiDocument.setArchive at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:203) at org.hibernate.tuple.PojoTuplizer.setPropertyValues(PojoTuplizer.java:173) at org.hibernate.persister.entity.BasicEntityPersister.setPropertyValues(BasicEntityPersister.java:2900) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:510) [...] Caused by: net.sf.cglib.beans.BulkBeanException: Lexical error at line 36, column 10. Encountered: "\u0142" (322), after : "@\n\n\nSandbox\nTestTopic1\n\nen\n0\nSandbox.WebHome\nXWiki.Admin\n1120562792127\n1120562808542\n1.2\n" at com.xpn.xwiki.doc.XWikiDocument$$BulkBeanByCGLIB$$9e5f0943.setPropertyValues() at org.hibernate.tuple.PojoTuplizer.setPropertyValuesWithOptimizer(PojoTuplizer.java:200) ... 63 more
------------------------------------------------------------------------
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
-- Ludovic Dubost XPertNet: http://www.xpertnet.fr/ Blog: http://www.ludovic.org/blog/ XWiki: http://www.xwiki.com Skype: ldubost AIM: nvludo Yahoo: ludovic
participants (4)
-
Ludovic Dubost -
Patrick Lee -
Ryszard Lach -
Xavier MOGHRABI