[Opinions needed] Encoding of xwiki source files
Hi all, While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests). But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings. There are 2 solutions: 1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii) Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding) As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs. Can you please tell me which solution you prefer ? Regards, -- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
Option 2, definitely. In the past there were a lot more files written using accented chars. Also, the number of non-ASCII chars should be minimized (see http://jira.xwiki.org/jira/browse/XWIKI-528 ). Sergiu. On 3/8/07, Gilles Serasset <[email protected]> wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the [email protected] list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
+1 for solutions 2. On 3/8/07, Gilles Serasset <[email protected]> wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the [email protected] list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
+1 for solution 2. S.
oops I meant +1 for solution 1 (utf-8) S. 2007/3/10, Sébastien Gaïde <[email protected]>:
+1 for solution 2.
S.
+1 for option 2 On 3/8/07, Gilles Serasset <[email protected]> wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
+1 for option 1 (use of UTF8) On 3/10/07, Marta Girdea <[email protected]> wrote:
+1 for option 2
On 3/8/07, Gilles Serasset <[email protected]> wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
-- François Le Droff http://www.droff.com
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers). -Vincent On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/ xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/ xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ? XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem. S.
-Vincent
On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/FrenchStemmer.java
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java
(Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/xpn/xwiki/content/LinkTest.java
(this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
On Mar 10, 2007, at 7:24 PM, Sébastien Gaïde wrote:
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ?
Because every committer has to make a conscious effort of setting up all his editors to save in utf8 (vi, idea, emacs, etc) and you can be sure some will not do it, by mistake, etc. Look at the rules in the development guide on xwiki.org. They are pretty simple rules but almost nobody uses them... So if we prefer to be safe rather than sorry I think a solution that doesn't mandate some voluntary effort is better. I may be wrong of course :) -Vincent
XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem.
S.
-Vincent On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/ com/xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/ com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/ com/xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http:// www.objectweb.org/wws
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/ wws
On Mar 10, 2007, at 7:41 PM, Vincent Massol wrote:
On Mar 10, 2007, at 7:24 PM, Sébastien Gaïde wrote:
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ?
Because every committer has to make a conscious effort of setting up all his editors to save in utf8 (vi, idea, emacs, etc) and you can be sure some will not do it, by mistake, etc.
I forgot to mention that I don't even know what it means to set one's edit tool to write in utf8. I've never done that in all my life and I suspect others will face the same issues/questions... -Vincent
Look at the rules in the development guide on xwiki.org. They are pretty simple rules but almost nobody uses them...
So if we prefer to be safe rather than sorry I think a solution that doesn't mandate some voluntary effort is better.
I may be wrong of course :)
-Vincent
XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem.
S.
-Vincent On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars: /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/ com/xpn/xwiki/plugin/autotag/FrenchStemmer.java /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/ com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java (Both are in ISO-8859-1) /Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/ com/xpn/xwiki/content/LinkTest.java (this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http:// www.objectweb.org/wws
-- You receive this message as a subscriber of the xwiki- [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http:// www.objectweb.org/wws
On 3/10/07, Sébastien Gaïde <[email protected]> wrote:
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ?
XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem.
Depending on a particular encoding is bad because: - The IDE has to be configured to use the encoding - Building from command line needs extra parameters - Opening a file using a standalone viewer, and not an IDE, will display the file using the system encoding, and we can't tell the user to change this - There are some people still using a shell, and something like midnight commander to open files, and the support for non-ascii chars is not so good in text shells. So the best solution would be not to have any special chars in the source files, so that any encoding works fine. However, there are some modules, like tests and translation files, where special characters are needed. And the only way to correctly display these chars in any environment is to use \u sequences. Still, we should make a full switch to utf8, including the source, so we should switch all the files to utf8. Sergiu S.
-Vincent
On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit
tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars:
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/FrenchStemmer.java
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java
(Both are in ISO-8859-1)
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/xpn/xwiki/content/LinkTest.java
(this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
Agree. +1 to solution 2 (\uXXXX) My system LANG=ru_RU.UTF-8. But I think to see in the code '\u????' is better than something unreadable character because missing fonts. Sergiu Dumitriu wrote:
Depending on a particular encoding is bad because: - The IDE has to be configured to use the encoding - Building from command line needs extra parameters - Opening a file using a standalone viewer, and not an IDE, will display the file using the system encoding, and we can't tell the user to change this - There are some people still using a shell, and something like midnight commander to open files, and the support for non-ascii chars is not so good in text shells.
So the best solution would be not to have any special chars in the source files, so that any encoding works fine. However, there are some modules, like tests and translation files, where special characters are needed. And the only way to correctly display these chars in any environment is to use \u sequences. Still, we should make a full switch to utf8, including the source, so we should switch all the files to utf8.
-- Artem Melentyev, UralSU
Wow ! so many good arguments against everything in UTF8 from you and VIncent. OK I understand better now why UTF8 everywhere may be problem. nevertheless, we have used utf8 for everything for more than a year now (Eclipse, vi, notepad++) with no special configuration for javac, maven etc ... and as of today with no problem. But of course our team is less spread than the XWiki team ... S. 2007/3/10, Sergiu Dumitriu <[email protected]>:
On 3/10/07, Sébastien Gaïde <[email protected]> wrote:
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ?
XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem.
Depending on a particular encoding is bad because: - The IDE has to be configured to use the encoding - Building from command line needs extra parameters - Opening a file using a standalone viewer, and not an IDE, will display the file using the system encoding, and we can't tell the user to change this - There are some people still using a shell, and something like midnight commander to open files, and the support for non-ascii chars is not so good in text shells.
So the best solution would be not to have any special chars in the source files, so that any encoding works fine. However, there are some modules, like tests and translation files, where special characters are needed. And the only way to correctly display these chars in any environment is to use \u sequences. Still, we should make a full switch to utf8, including the source, so we should switch all the files to utf8.
Sergiu
S.
-Vincent
On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit
tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars:
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/FrenchStemmer.java
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java
(Both are in ISO-8859-1)
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/xpn/xwiki/content/LinkTest.java
(this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
--
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
+1 for option 2 Catalin On 3/10/07, Sebastien Gaide <[email protected]> wrote:
Wow ! so many good arguments against everything in UTF8 from you and VIncent. OK I understand better now why UTF8 everywhere may be problem.
nevertheless, we have used utf8 for everything for more than a year now (Eclipse, vi, notepad++) with no special configuration for javac, maven etc ... and as of today with no problem. But of course our team is less spread than the XWiki team ...
S.
2007/3/10, Sergiu Dumitriu <[email protected]>:
On 3/10/07, Sébastien Gaïde <[email protected]> wrote:
Vincent Massol a écrit :
+1 option 2 too (I think option 1 is never going to work because it's too constraining for developers).
why ?
XWiki is the only project that is not encoded in utf8 in my workspace. All the other projects are 100% utf8, with no problem.
Depending on a particular encoding is bad because: - The IDE has to be configured to use the encoding - Building from command line needs extra parameters - Opening a file using a standalone viewer, and not an IDE, will display the file using the system encoding, and we can't tell the user to change this - There are some people still using a shell, and something like midnight commander to open files, and the support for non-ascii chars is not so good in text shells.
So the best solution would be not to have any special chars in the source files, so that any encoding works fine. However, there are some modules, like tests and translation files, where special characters are needed. And the only way to correctly display these chars in any environment is to use \u sequences. Still, we should make a full switch to utf8, including the source, so we should switch all the files to utf8.
Sergiu
S.
-Vincent
On Mar 8, 2007, at 4:01 PM, Gilles Serasset wrote:
Hi all,
While working on encodings problems, I saw that some of the source file do use non ascii chars (which is normal, especially in unit
tests).
But, there is currently no decision on the encoding of the source file, hence, compilers cannot correctly read the files that do use non ascii chars. This leads to tests working on single instances but not on others, only due to compilation settings.
There are 2 solutions:
1. force everybody to use UTF-8 encoding for their source files (it's quite easy to st up most IDE once and for all for this...) and specify the encoding in javac parameters in ant and maven. 2. force everybody to use unicode escapes (\uXXXX) to specify a non ascii char in the sources (easily detectable on build, but harder for developpers who have to use native2ascii)
Current files with non ascii chars:
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/FrenchStemmer.java
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/main/java/com/xpn/xwiki/plugin/autotag/AutoTagPlugin.java
(Both are in ISO-8859-1)
/Users/serasset/dev/xwiki/trunks-users/xwiki/core/src/test/java/com/xpn/xwiki/content/LinkTest.java
(this one seems to be encoded in VISCII (vietnamese encoding)
As in LinkTest.java, the test uses vietnamese characters, it's likely that ISO-8859-1 encoding is not a viable option for the xwiki source encoding. In the mean time, LinkTest.java should use \uXXXX uniocde escapes in order to run correctly in all installs.
Can you please tell me which solution you prefer ?
Regards,
-- Gilles Sérasset GETALP-LIG BP 53 - F-38041 Grenoble Cedex 9 Phone: +33 4 76 51 43 80 Fax: +33 4 76 44 66 75
--
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
-- You receive this message as a subscriber of the [email protected] mailing list. To unsubscribe: mailto:[email protected] For general help: mailto:[email protected]?subject=help ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
participants (10)
-
Artem Melentyev -
Catalin Hritcu -
François Le Droff -
Gilles Serasset -
Marta Girdea -
Phung Nam -
Sebastien Gaide -
Sergiu Dumitriu -
Sébastien Gaïde
-
Vincent Massol