Re: [xwiki-users] problem with Chinese

Hi Jackson, I'm really sorry we didn't apply your patch. Actually the recommendation is to create a JIRA issue and attach your patch to it. This allows us to not forget anything. See http://www.xwiki.org/xwiki/bin/view/Community/Contributing for more details. If you could do this that would be great as we'll be able to schedule your patch for inclusion. Thanks again! -Vincent On Mar 21, 2007, at 3:55 AM, Jackson Wang wrote: I have translated resource file into Chinese , http://mail-archive.objectweb.org/xwiki-dev/2006-12/msg00092.html and config xwiki with Chinese language http://mail-archive.objectweb.org/xwiki-dev/2006-12/msg00082.html Maybe i put these messages into wrong place, sorry. 2007/3/20, bey &lt;youcef.bey(a)imag.fr&gt;fr>: > > > If you have not found My.cnf, then please create it in both 'mysql/bin' > and 'mysql/data'. > > So If your xwiki is working on windows then you will find a 'bat' file > from which Tomcat start (in the Tomcat directory). Open it and modify. > > ps. You need to create My.cnf only in the case of MySQL. I haven't any > idea on how to do it with HsqlDB - If I understand you have tried both MySql > and HsqlDB. > > Please do the same thing whether your XWiki is working on Linux - for > tomcat modifiy 'catalina.sh' and create My.cnf > > Cheers > -- > Youcef > > ning wang a écrit : > > 2007/3/20, Youcef BEY < youcef.bey(a)imag.fr&gt;gt;: > > > Hi Wang, > > > > This problem have also took for longtime to solve it (in fact, I faced > > it before and hope that it correspond to yours). So I am using multilingual > > data in my XWiki (Arabic, Chinese, English, French and Japanese) and > > everything is working well with the UTF-8 encoding (MySQL, Tomcat, XWiki). > > > > I read all your emails and responses. It seems that someone adviced > > you to check again the database encoding. Yes, the problem is related to the > > encoding of char in the database and it is simple to solve. In fact, it > > seems that MySql display that it's char encoding is in UTF-8 but you need > > also to add the following lines to my.cnf : > > > > In my system, there is only my.ini. So I edit my.ini following your > advise below. > > [client] > > default-character-set=utf8 > > [mysqld] > > default-character-set=utf8 > > character-set-server = utf8 > > collation-server = utf8_general_ci > > > > Also check the following parameters for your Tomcat UTF-8 encoding > > (for linux, check catalina.sh): > > > > I use Tomcat 5.5. I can't find something like start.bat. So I don't > know how to set Tomcat to UTF-8 encoding. May you tell me the detail? > > ----------------------------------------------------------------------------------------------------- > > > > > > echo off > > rem set LANG=fr_FR.ISO8859-1 > > set LANG=en_US.UTF-8 > > set JETTY_PORT=8080 > > set JETTY_HOME=. > > java %JAVA_OPTS% -Djetty.port=%JETTY_PORT% -Djetty.home=%JETTY_HOME% - > > Dfile.encoding=UTF-8 -jar %JETTY_HOME%/start.jar > > > > > > > > Add the following variables to your system > > ----------------------------------------------------------- > > LANG = en-US_UTF8 > > > > Change the XWiki encoding : > > ----------------------------------------- > > (i) XWiki.xml > > encoding = UTF-8 > > (ii) XWiki.cfg > > CharSet = UTF-8 > > > > I do what I can, But there still do not work. "???" still exist. > > If this configuration work well, so we need after that to write (as > > suggested) a document for helping new users. > > > > Please let us know whether it work? > > > > Cheers > > ---------------------------------- > > Youcef Bey > > Graduate School of Education, > > The University of Tokyo, > > ----- Original Message ----- > > *From:* ning wang &lt;daning106(a)gmail.com&gt; > > *To:* xwiki-users(a)objectweb.org > > *Sent:* Friday, March 16, 2007 11:27 AM > > *Subject:* Re: [xwiki-users] problem with Chinese > > > > Hi Jimmy, > > > > I tried again, and I think I found something more. > > > > I use the xwiki without any change about encoding, just using default > > encoding ISO-8859-1. Use Jetty and HsqlDB. I write Chinese twice. Then roll > > back. The words can display correctly. But the only problem is that if I use > > Wiki text editor, I found that Chinese characters show like %D6%D0%CE%C4. > > It's can work in both view and roll-back. Therefore, I think the charaters > > can saved correctly in the database using ISO-8859-1. And it gose alone well > > with the WYSIWYG editor. But it's not convenient at all as I can't see any > > Chinese. So I think the ISO-8859-1 can't display Chinese characters in xwiki > > editor. > > > > But if I instead ISO-8859-1 with UTF-8 int the web.xml xwiki.cfg and > > velocity.properties. Roll-back problem came again. But Chinese show > > properly int xwiki ext editor. > > > > Hope to help some. > > > > Thanks, > > Sincerely, Wang Ning > > > > 2007/3/17, jimmy(a)comkraft.com < jimmy(a)comkraft.com&gt;gt;: > > > > > I tried to creat/modify a couple of pages in Chinese and I didn't > > > see any > > > problems when rolling back. I tried to use both the WYSIWYG editor > > > and the > > > wiki editor but both seems to work for me. > > > > > > I am using 1 Beta 4 on a Windows 2000 computer Sp4 (locale is > > > English). > > > The database is PostgreSQL 8.2.3 > > > I use UTF-8 for the DBs but didn't change any other settings. > > > > > > I am using the following tool to input Chinese > > > (http://www.3d114.com/tools/gb_pinyin.htm ) and then paste them into > > > my > > > xwiki. You may want to try putting in Chinese there and paste them > > > into > > > your wiki to see if there are some other encoding issues in the > > > input > > > phase - as there are several internal formats for Chinese Characters > > > > > > (Big5, GBxxxx, UTF-8 etc). > > > > > > Hope this helps.Bonjour, > > > > > > J'ai parfaitement compris votre explication. > > > > > > Je vous remercie. > > > > > > Cordialement. > > > > > > > > > -Jimmy > > > > > > > On 3/16/07, Gilles Serasset &lt;Gilles.Serasset(a)imag.fr&gt; wrote: > > > >> > > > >> Hi Ning, > > > >> I also have a utf8 instance and I cannot reproduce your bug even > > > when > > > >> mixing French and Japanese characters. I'm using postgres though, > > > but > > > >> mysql > > > >> qhould work also. > > > >> > > > > > > > > However, I can reproduce this, on a fully UTF-8 system: OS, disk, > > > > database, > > > > XWiki settings. > > > > > > > > It seems that when working with the document archive the > > > characters are > > > > broken. In the database everything is OK, so it's something > > > internal. > > > > > > > > When did you set up your wiki to use UTF8 characters ? Maybe you > > > created > > > > the > > > >> database with an ISO Latin 1 encoding and then change your wiki > > > encoding > > > >> after the DB creation ? > > > >> > > > >> What happens when you ask to see the modifications between the > > > current > > > >> version and an older one ? Do you correctly see the chinese chars > > > of the > > > >> older version ? > > > >> > > > >> And by the way, Sergiu, I think the velocity.properties parameter > > > are > > > >> just > > > >> useless (as far as I could test) as the velocity macros are > > > called using > > > >> Strings and not using InputStreams, in fact, it could hurt a > > > little when > > > >> reading the xxx.vm files that are not encoded in UTF-8 (see > > > discussion > > > >> on > > > >> the encoding of the code base...). > > > >> > > > > > > > > Indeed, I did not notice anything when changing this. But I don't > > > know > > > > what > > > > internal stuff might break if I don't set this. > > > > > > > > I think that this only affect the way .vm files stored on the disk > > > are > > > > read, > > > > but these files are plain ASCII. > > > > > > > > Regards, Gilles, > > > >> > > > >> On 16 mars 07, at 16:27, ning wang wrote: > > > >> > > > >> I do everything you said. And I make sure that all encoding is > > > UTF-8 or > > > >> utf8. > > > >> But I'm sad to find that it does not work, still "?????" when > > > roll > > > >> back. > > > >> > > > >> 2007/3/16, Sergiu Dumitriu < sergiu.dumitriu(a)gmail.com&gt;gt;: > > > >> > > > > >> > Some ideas: > > > >> > > > > >> > * Check that the tables are really using UTF-8: get a mysql > > > shell, > > > >> open > > > >> > the xwiki database and type "select charset(xwd_content from > > > >> xwikidoc;". If > > > >> > it does not say UTF-8, then you need to change the encoding. > > > I've > > > >> attached a > > > >> > script that tries to do that (and creates indexes, too) > > > >> > > > > >> > * Check that the mysql connections are set to UTF-8. "show > > > variables > > > >> > like 'character_set%';". If it says something else, you can > > > either > > > >> globally > > > >> > set the mysql encoding to utf-8 (in the mysql configuration > > > file, > > > >> search the > > > >> > mysql documentation), or configure hibernate to request this: > > > >> > <xwiki-dir>/WEB-INF/hibernate.cfg.xml, add > > > >> > <property name="connection.characterEncoding > > > ">utf8</property> > > > >> > after: > > > >> > <property > > > >> name="connection.driver_class">com.mysql.jdbc.Driver </property> > > > >> > > > > >> > * Check that you set all the encoding parameters to UTF-8: > > > >> > ** in WEB-INF/xwiki.cfg: xwiki.encoding=UTF-8 > > > >> > ** in WEB-INF/web.xml: > > > >> > <param-name>encoding</param-name> > > > >> > <param-value>UTF-8</param-value> > > > >> > ** in WEB-INF/velocity.properties: > > > >> > input.encoding=UTF-8 > > > >> > output.encoding=UTF-8 > > > >> > > > > >> > After this, it should work just fine. > > > >> > > > > >> > > > > >