Re: [xwiki-devs] Simple patch to enable/preserve underscore chars in attachment file names

On 04/23/2010 09:44 PM, Caleb James DeLisle wrote: No, utf8 should allow all characters, since UTF-8 allows complete representation of all unicode characters, which cover all existing non-UTF charsets. The workaround that you talk about is actually a different problem, and a common bad practice: storing UTF-8 or some other fixed encoding bytes in a latin1 column, by decomposing and recomposing strings into bytes. This is bad because it breaks sorting and upper/lowercase transformations. It's also bad because it involves another byte split/parse operation, since the data is already transformed once from fake "ISO-8859-1" bytes into a String. It is good because it actually doesn't care about the database encoding, it's independent, and works with (almost) all database encodings transparently. What actually breaks when switching from latin1 to utf8, in this scenario, is that UTF-8 has some intrinsic data validation, meaning that certain bytes and certain byte pairs/triples are not valid UTF-8 strings, thus, trying to push random bytes could sometime fail. Normally, storing UTF-8 bytes in a utf8 column should work, so it fails when storing another encoding in a utf8 column. Fortunately, we don't use this technique, so we're not affected by this problem. Now, on the contrary to what you say, UTF-8 has been the default encoding of the wiki for some time, and it fails if the database (mysql) is NOT in utf8. We actually require the database to be in utf8, since otherwise data will be lost after it gets out of the cache. Why some people still prefer latin1 in a world that is moving more and more towards UTF-8? Well, there are a few disadvantages to utf8, when used inside mysql: - Data is bigger, since latin1 uses exactly 1 byte for each character, while UTF-8 uses 2 for most european languages, and even 3 for Asian and other exotic scripts. I'm speaking here only about storage space needed. - Not only is the storage bigger, but the algorithms are a bit more complex/time consuming: counting how many characters are in a latin1 string is simple, just see how many bytes are in there. In UTF it's more complex, since 1, 2, 3 bytes can form one character, and the rules require full examination of each byte. Thus, length(latin1) is O(1), length(utf8) is O(n). Most other string functions are also affected by this complexity problem. - Moreover, indexes are limited to 1024 (or was it 2048?) BYTES in length, and MySQL assumes the worst case scenario when computing how many bytes a column takes up. So, while it's possible to use 4 small columns (255 chars) combined in an index, if utf8 is used instead, 4*(255*3 bytes per char in the worst case scenario)>1024, thus using utf8 in tables limits the size of indexes. There might be other major disadvantages, but these are the most important that I know of. The big advantage of utf8, when compared to all latinX charsets, is that it can store much more characters. All latinX charsets can store only 256 possible characters (including all the control chars rarely used). And frankly, the entire web is moving towards UTF-8. These disadvantages are not problems with all UTF-8 applications, it is just a very lousy design/implementation in mysql, and it's one of the main reasons why I don't like mysql. I hope that they will realize that they did it all wrong and fix it at some point. -- Sergiu Dumitriu http://purl.org/net/sergiu/