On 12/07/2012 04:26 PM, Vincent Massol wrote: +1 for UTF-8 only. If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands. +1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs. Thanks, Caleb

On 12/07/2012 04:56 PM, Caleb James DeLisle wrote: I'm afraid we'll get errors if we do that, since indexes are still limited to a total of 1024 bytes, and we're already maxing out with 255-varchar columns + other fields. In short, MySQL sucks for serious projects, but we can't really tell our users "use Postgres, it's better". So I'd rather keep it to the current utf-8, and hope that nobody will need the extended unicode planes, until we find a better solution. To be more specific: we can't switch to 4-byte utf8 until we stop using names as primary key elements. Just tried it, and indeed trying to save characters outside the BMP will fail. Thanks for pointing this out. Except that it doesn't work like that. If you use latin1, you'll get errors with the default XE xar about invalid values in the RCS table. The connector doesn't send bytes, it sends characters, and the database will try to store them, which it can't. Every piece of MySQL has an encoding, which isn't opaque. Pushing characters outside the table's charset will trigger an exception. -- Sergiu Dumitriu http://purl.org/net/sergiu

Hello, I thought xwiki was using UTF-8 by default?! Because in that case, we _only_ need to check the "environment", which means every supported database and containers, and maybe the system language, not sure it matters btw. I can take care of such a documentation if you want. I've heard that postgreSQL uses utf-8 by default, but need to study it anyway. 2013/2/11 Vincent Massol <vincent(a)massol.net>

On 12/07/2012 04:26 PM, Vincent Massol wrote: Non-ASCII-compatible encodings don't work at least because of the way we read components.txt. For other encodings, the problem is that you never know when something will break. Things may appear to work, but enter a non-ASCII character and: - the data might be discarded - the database might throw an exception -- if a non-transactional database engine is used, then all future access to the document or history or object properties will fail - the document might become inaccessible since the URL is decoded incorrectly - The PDF export might break - Escaped XML entities might appear in the browser instead of characters (just from the top of my head) This isn't a proposal to change our rules, it's a proposal to make explicit what we've been doing anyway. There have been many issues and emails where our answer starts with "make sure your <component> encoding is set to UTF-8". On xwiki.org users often try to use non-ASCII characters, and that doesn't work, and so we might be losing potential users if they assume that XWiki simply doesn't support their language. -- Sergiu Dumitriu http://purl.org/net/sergiu