[xwiki-devs] [VOTE] Switch to UTF-8 by default in our distributions
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1 This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release. WDYT? -- Sergiu Dumitriu http://purl.org/net/sergiu/
My giant +1. Note that the official (IANA) encoding name is utf-8 not UTF-8 which is a derivative of Java's (wrong) UTF8. All are accepted as encoding names by Java. paul Le 14-mars-09 à 02:20, Sergiu Dumitriu a écrit :
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1
This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release.
WDYT? -- Sergiu Dumitriu http://purl.org/net/sergiu/ _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Mar 14, 2009, at 2:20 AM, Sergiu Dumitriu wrote:
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1
This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release.
WDYT?
We'd need to put huge disclaimers everywhere since this would cause problems with users (the majority?) who have MySQL or other dbs set in ISO8859-1, or their containers set a ISO8859-1 as the default, right? Is mysql in UTF8 by default? Same question for tomcat or other JEE containers. Thanks -Vincent
Le 14-mars-09 à 19:53, Vincent Massol a écrit :
We'd need to put huge disclaimers everywhere since this would cause problems with users (the majority?) who have MySQL or other dbs set in ISO8859-1, or their containers set a ISO8859-1 as the default, right?
Actually no. My experience is that there's no "encoding mismatch" even with a mysql which is set to iso-8859-1 while xwiki and jdbc is set to utf-8. The only mismatch we endured until we moved to a mysql that's parametrized for utf-8 is the storage inability for high-bytes characters (my test cases was lambda λ and leq ≤) which simply got garbled after the persist and read cycle happened (may take one hour sometimes on xwiki).
Is mysql in UTF8 by default?
It was in my MacOSX MySQL which I downloaded (from mysql.org) about a year ago. Same for my colleague on Windows. Same for our sysadmin with a Linux MySQL downloaded from there. But it was not the case with the mysql coming from a 3-years old SuSE installation.
Same question for tomcat or other JEE containers.
Many many many times the platform encoding is used rather than the containers' encoding. So that's MacRoman on mac! (enjoy!) The re-encoding happens transparently though, there's no commonality between that back-end and front-end. One thing that has bitten me more often than not is that Apache Httpd defaults to iso-8859-1 set explicitly on files with suffix html (so the mime-type is text/html;charset=iso-8859-1 instead of leaving to the meta head element the luxury of specifying that). I believe it should not affect normal XWiki though. paul PS: I think that a sane admin that does an update will diff xwiki.cfg and will see the difference!
Paul Libbrecht wrote:
Le 14-mars-09 à 19:53, Vincent Massol a écrit :
We'd need to put huge disclaimers everywhere since this would cause problems with users (the majority?) who have MySQL or other dbs set in ISO8859-1, or their containers set a ISO8859-1 as the default, right?
Actually no. My experience is that there's no "encoding mismatch" even with a mysql which is set to iso-8859-1 while xwiki and jdbc is set to utf-8. The only mismatch we endured until we moved to a mysql that's parametrized for utf-8 is the storage inability for high-bytes characters (my test cases was lambda λ and leq ≤) which simply got garbled after the persist and read cycle happened (may take one hour sometimes on xwiki).
Yes, the problem is that mysql silently transforms characters it can't store using the current charset into '?'. And since the document is still in the cache, this won't get noticed for a while.
Is mysql in UTF8 by default?
It was in my MacOSX MySQL which I downloaded (from mysql.org) about a year ago. Same for my colleague on Windows. Same for our sysadmin with a Linux MySQL downloaded from there. But it was not the case with the mysql coming from a 3-years old SuSE installation.
mysql set to utf8 on Gentoo.
Same question for tomcat or other JEE containers.
Many many many times the platform encoding is used rather than the containers' encoding. So that's MacRoman on mac! (enjoy!) The re-encoding happens transparently though, there's no commonality between that back-end and front-end.
The only problem I've seen in practice is the PDF export, which uses new String(bytes) and String.getBytes() without specifying the xwiki.cfg encoding. I've opened http://jira.xwiki.org/jira/browse/XWIKI-3361 which isn't hard to fix.
One thing that has bitten me more often than not is that Apache Httpd defaults to iso-8859-1 set explicitly on files with suffix html (so the mime-type is text/html;charset=iso-8859-1 instead of leaving to the meta head element the luxury of specifying that). I believe it should not affect normal XWiki though.
paul
PS: I think that a sane admin that does an update will diff xwiki.cfg and will see the difference!
Absolutely. Unfortunately sane admins are almost as rare as flying pigs. Almost. -- Sergiu Dumitriu http://purl.org/net/sergiu/
Sergiu Dumitriu wrote:
Paul Libbrecht wrote:
Le 14-mars-09 à 19:53, Vincent Massol a écrit :
We'd need to put huge disclaimers everywhere since this would cause problems with users (the majority?) who have MySQL or other dbs set in ISO8859-1, or their containers set a ISO8859-1 as the default, right?
Actually no. My experience is that there's no "encoding mismatch" even with a mysql which is set to iso-8859-1 while xwiki and jdbc is set to utf-8. The only mismatch we endured until we moved to a mysql that's parametrized for utf-8 is the storage inability for high-bytes characters (my test cases was lambda λ and leq ≤) which simply got garbled after the persist and read cycle happened (may take one hour sometimes on xwiki).
Yes, the problem is that mysql silently transforms characters it can't store using the current charset into '?'. And since the document is still in the cache, this won't get noticed for a while.
Is mysql in UTF8 by default?
It was in my MacOSX MySQL which I downloaded (from mysql.org) about a year ago. Same for my colleague on Windows. Same for our sysadmin with a Linux MySQL downloaded from there. But it was not the case with the mysql coming from a 3-years old SuSE installation.
mysql set to utf8 on Gentoo.
Same question for tomcat or other JEE containers.
Many many many times the platform encoding is used rather than the containers' encoding. So that's MacRoman on mac! (enjoy!) The re-encoding happens transparently though, there's no commonality between that back-end and front-end.
The only problem I've seen in practice is the PDF export, which uses new String(bytes) and String.getBytes() without specifying the xwiki.cfg encoding. I've opened http://jira.xwiki.org/jira/browse/XWIKI-3361 which isn't hard to fix.
And now it's fixed, so there's no other problem for switching to utf8, except latin1 databases.
One thing that has bitten me more often than not is that Apache Httpd defaults to iso-8859-1 set explicitly on files with suffix html (so the mime-type is text/html;charset=iso-8859-1 instead of leaving to the meta head element the luxury of specifying that). I believe it should not affect normal XWiki though.
paul
PS: I think that a sane admin that does an update will diff xwiki.cfg and will see the difference!
Absolutely. Unfortunately sane admins are almost as rare as flying pigs. Almost.
-- Sergiu Dumitriu http://purl.org/net/sergiu/
For latin1 databases, or any other limited encoding, a sanity check at storage migration sounds right or? Put lambda, close connection, reopen, read lambda worked? If not add a warning in the log. Paul Le 17-mars-09 à 10:46, Sergiu Dumitriu <[email protected]> a écrit :
The only problem I've seen in practice is the PDF export, which uses new String(bytes) and String.getBytes() without specifying the xwiki.cfg encoding. I've opened http://jira.xwiki.org/jira/browse/XWIKI-3361 which isn't hard to fix.
And now it's fixed, so there's no other problem for switching to utf8, except latin1 databases.
+1 Old issue. I also remember that the default language was (and still is, AFAIK) FR in the start_xwiki.sh and that this was creating issues with XMLRPC and XEclipse on date conversions. It know that Fabio resolved those problems, but shouldn't we switch to en_US as well if we switch to UTF8? My 2 cents. Sergiu Dumitriu wrote:
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1
This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release.
WDYT?
Eduard Moraru wrote:
+1
Old issue.
I also remember that the default language was (and still is, AFAIK) FR in the start_xwiki.sh and that this was creating issues with XMLRPC and XEclipse on date conversions.
It know that Fabio resolved those problems, but shouldn't we switch to en_US as well if we switch to UTF8?
Of course, this goes without saying. I wonder if this is needed at all, since -Dfile.encoding should be enough. Any specific LANG might not be available on the server (in France it is common to have just fr_FR). IIRC, there was a mail about this, I'll try to find it.
My 2 cents.
Sergiu Dumitriu wrote:
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1
This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release.
WDYT?
-- Sergiu Dumitriu http://purl.org/net/sergiu/
Sergiu Dumitriu wrote:
Eduard Moraru wrote:
+1
Old issue.
I also remember that the default language was (and still is, AFAIK) FR in the start_xwiki.sh and that this was creating issues with XMLRPC and XEclipse on date conversions.
It know that Fabio resolved those problems, but shouldn't we switch to en_US as well if we switch to UTF8?
Of course, this goes without saying. I wonder if this is needed at all, since -Dfile.encoding should be enough. Any specific LANG might not be available on the server (in France it is common to have just fr_FR).
IIRC, there was a mail about this, I'll try to find it.
http://n2.nabble.com/LANG-property-in-startup-script-in-standalone-version-t...
My 2 cents.
Sergiu Dumitriu wrote:
Niels Mayer wrote:
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Big +1
This is one of the oldest issues open (http://jira.xwiki.org/jira/browse/XWIKI-94). I guess that since we're nearing 2.0, it would be a good thing to set this on trunk, before the 1.9M1 release.
WDYT?
-- Sergiu Dumitriu http://purl.org/net/sergiu/
I don't know whether I can "+1" on this issue, but if I could, I would definitely "+1" it :-) On Mon, Mar 16, 2009 at 8:23 AM, Sergiu Dumitriu <[email protected]> wrote:
Of course, this goes without saying. I wonder if this is needed at all, since -Dfile.encoding should be enough. Any specific LANG might not be available on the server (in France it is common to have just fr_FR).
I previously identified 6 areas that I hard-coded over to UTF-8. Some of these were voodoo, some of these worked-directly. I believe this part (which also started this thread) was the most relevant: to [email protected], XWiki Users <[email protected]>
date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ;
likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
So what exactly do these 6 areas affect (or side-efect) if set or not set? (1) -Dfile.encoding -> UTF-8
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?).
(2) -Djavax.servlet.request.encoding -> UTF-8
When that alone didn't work, I also added " -Djavax.servlet.request.encoding=UTF-8
(3) -DjavaEncoding -> UTF-8
-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding= UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
(4) LANG -> UTF-8
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script...
(5) com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' -> UTF-8
however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(6) xwiki.encoding -> UTF-8
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
-- Niels http://nielsmayer.com
Niels Mayer wrote:
I previously identified 6 areas that I hard-coded over to UTF-8. Some of these were voodoo, some of these worked-directly. I believe this part (which also started this thread) was the most relevant:
So what exactly do these 6 areas affect (or side-efect) if set or not set?
(1) -Dfile.encoding -> UTF-8
This sets the charset/encoding to use as a default whenever strings (which are always UTF-16, see http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Character.html) are converted to bytes and vice versa: - String.getBytes - new String(byte[]) - InputStream vs. Reader - OutputStream vs. Writer Fortunately, all these allow manually specifying an encoding to use, and if we want to be independent of the JVM encoding, we should always do that. There are still a few places where we use the JVM encoding, but a recent trunk is safe to use without taking care of the -Dfile.encoding.
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?).
(2) -Djavax.servlet.request.encoding -> UTF-8
I can't find anything about this, so I doubt it does something. The servlet specification (2.5) states that when determining the encoding used to parse a request, either it is specified in the request itself in a HTTP Header, or the default encoding of ISO-8859-1 is used. Maybe some containers allow to change this default, but it's not standard. This auto-detection can be overridden by a call to request.setCharacterEncoding("enc").
When that alone didn't work, I also added " -Djavax.servlet.request.encoding=UTF-8
(3) -DjavaEncoding -> UTF-8
I also can't find anything about this. Could be a mistaken name for -Dfile.encoding? There is a Jetty setting named javaEncoding, which affects the encoding used by the JSP compiler to generate Java source files, but we don't have any JSP pages.
-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding= UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
(4) LANG -> UTF-8
This sets an environment variable that is read by the JVM to determine the right value for file.encoding and the default locale. Setting -Dfile.encoding takes precedence, so this makes no sense for XWiki. It still influences the default locale, though, but the OS/JVM must have support for it.
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script...
(5) com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' -> UTF-8
This is set in web.xml, or directly in the Java file if you modify the source and recompile it. This influences how the request is parsed, since request.getParameter transforms bytes into Strings. See the explanation for (2), this always forces our encoding to be used for reading requests. In the future this should be merged with the setting in xwiki.cfg, but this is not possible yet because the servlet filter does not have access to the XWiki object (to call xwiki.getEncoding()), and XWiki does not have access to the filter config. The ugly solution is to manually parse one of these files, but it really is Ugly.
however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(6) xwiki.encoding -> UTF-8
This is the main setting that we use for determining the encoding. This allows us to be independent of the system settings, since hosted users cannot always alter these settings, but they can change their application.
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
There are a few more settings for various containers. For example, Tomcat users must use these (http://tomcat.apache.org/tomcat-6.0-doc/config/http.html): (7) URIEncoding - Tomcat treats different parts of the request differently: - URL path using UTF-8 - URL query using ISO-8859-1 - request body using the specified encoding (see 2 and 5) Previous versions of Tomcat (4.1) used the body encoding also for the query string, but they fixed this (this is in accordance with the specs). So, in order for request.getParameter to work fine, this setting must be specified. (8) useBodyEncodingForURI - see above, this makes tomcat use the body encoding also for the query string. Either (7) or (8) should be set. (9) For Jetty, -Dorg.mortbay.util.URI.charset=UTF-8 can influence the encoding used for parsing the query string. The difference is that by default they use UTF-8 in recent versions. -- Sergiu Dumitriu http://purl.org/net/sergiu/
These are the most fragile parts I find. Provided xwiki fully switches to utf-8, it should care that a form that is submitted via POST or GET is properly received. What I reached with simple xwiki and default settings were pretty poor in this respect (till the fact to recommend no accents in titles). Is there a way to provide at the application level the parsing of URIs? I think that's the critical part, and is mostly fragile for the parameters where user-input appears. paul PS: Looking with envy to clean URLs such as http://fr.wikipedia.org/wiki/été which is properly redirected to: http://fr.wikipedia.org/wiki/été and has translation: http://fa.wikipedia.org/wiki/تابستان Le 17-mars-09 à 15:09, Sergiu Dumitriu a écrit :
There are a few more settings for various containers. For example, Tomcat users must use these (http://tomcat.apache.org/tomcat-6.0-doc/config/http.html):
(7) URIEncoding - Tomcat treats different parts of the request differently: - URL path using UTF-8 - URL query using ISO-8859-1 - request body using the specified encoding (see 2 and 5) Previous versions of Tomcat (4.1) used the body encoding also for the query string, but they fixed this (this is in accordance with the specs). So, in order for request.getParameter to work fine, this setting must be specified.
(8) useBodyEncodingForURI - see above, this makes tomcat use the body encoding also for the query string. Either (7) or (8) should be set.
(9) For Jetty, -Dorg.mortbay.util.URI.charset=UTF-8 can influence the encoding used for parsing the query string. The difference is that by default they use UTF-8 in recent versions. --
participants (5)
-
Eduard Moraru -
Niels Mayer -
Paul Libbrecht -
Sergiu Dumitriu -
Vincent Massol