On 12/26/2009 02:29 PM, Vincent Massol wrote:
On Dec 26, 2009, at 2:17 PM, Sergiu Dumitriu wrote:
Hi devs,
The short version:
Should we always use UTF-8 for encoding and decoding URLs,
regardless of
the wiki encoding, for better compliance with web standards?
The long version:
By definition, URLs can only contain ASCII characters, everything else
must be converted to their corresponding bytes and escaped as %XY
escapes. The problem is that "their corresponding bytes" implies a
charset + encoding, and no specification *enforces* a specific pair,
although it is *recommended* to use Unicode + UTF8, to comply with the
modern tendency of the web in general.
Traditionally, XWiki has been using the same encoding as the
configured
global wiki encoding for the URLs, which means that before 1.9, when
we
switched to UTF8 as the default wiki encoding, all URLs were using the
ISO-8859-1 encoding. Since the switch to UTF-8, URLs are also using
the
UTF-8 encoding by default, although the wiki encoding can be changed.
Now, since 2.1, a bugfix accidentally changed the behavior, so that
parsing back URLs always uses the UTF-8 encoding, even though
composing
URLs continues to use the wiki encoding. This is a bug, which prevents
changing the encoding to anything other than UTF-8, and it should be
fixed.
Now, we have two options:
1. Reintroduce the old behavior, so that URLs always use the wiki
encoding. This is a direct bugfix.
2. Also change the encoding part, so that UTF-8 is always used. This
is
an improvement, going towards better compliance with web standards.
Personally I think that the second option is the better one, but it
requires a vote, since it has a few drawbacks.
Advantages:
+ better compliance with web standards, since UTF-8 is the recommended
encoding for URLs (although not imposed)
Is there any reference to this? Some RFC that we could quote in the
code?
The URL RFC predates the wide adoption of UTF, so it does not mention
any encoding (see
http://tools.ietf.org/html/rfc1738#section-2.2 ). This
is why I said that there's no enforcement. However, the URI RFC, which
is a generalization of URLs, enforces UTF-8 (see
http://tools.ietf.org/html/rfc3986#section-2.5 ).
The URI RFC officially *updates* the URL RFC, so we can say that URLs
are currently standardized by the new, UTF-8 enforcing RFC 3986,
although for backwards compatibility URLs can still be used with the RFC
1738 definition.
+ support for
a wider range of document names, since UTF-8 allows
full-unicode document names, while ISO-8859-1 limits names to latin1
characters
+ better support from browsers, since entering accented characters
directly in the address bar encodes the URL sent to the server using
UTF-8,
Is that true for all browsers? Is there a standard?
FF, Opera, Chrome.
Konqueror even displays %E9 as an invalid UTF character <?>, so it
assumes even more that the URL is in UTF-8.
IE6 does not automatically convert %XY escapes to their equivalent
character, so it displays both %E9 and %C3%A9 (the utf encoding for é).
However, entering é in the address bar converts to UTF-8 bytes. Also
note that IE6 predates the RFC 3986 by several years, so it has the
right not to assume UTF-8 in URLs.
and decoding
the URL also assumes UTF-8; this means that a
document named "é" will be printed as .../view/Main/%E9 and will
have to
be entered the same way in the address bar when ISO-8859-1 is used,
and
as .../view/Main/é when UTF-8 is used
Drawbacks:
- by default Tomcat uses ISO-8859-1 as the encoding for URLs, so the
Tomcat configuration will have to be changed as in
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding#HTomcat
Any known issue logged against Tomcat? Any planned fix? If this is
really a web standard why wouldn't Tomcat change this behavior?
According to
http://wiki.apache.org/tomcat/Tomcat/UTF-8 , Tomcat
developers interpret RFC 2616 (the HTTP 1.1 specification) as to
recommend the ISO-8859-1 charset as the default. However, this is not
true. The RFC references the URI RFC as the standard for the requested
address, and the 8859-1 is the default for the response *body* only. The
original URI RFC referenced by the HTTP RFC,
http://www.ietf.org/rfc/rfc2396.txt , does not recommend a default
charset+encoding, although it only mentions UTF-8 as a possible
encoding, and doesn't mention ISO-8859-1 at all. So I'd rather say that
the HTTP 1.1 specification does not recommend any default encoding for
addresses, rather than ISO-8859-1, and it hints towards UTF-8. And that
RFC has been deprecated in favor of RFC 3986, which clearly states that
UTF-8 is used. Perhaps I should send this interpretation to the Tomcat guys.
- some
existing bookmarks will not work anymore once the encoding is
changed
+1 for option 2 from me,
Before voting I'd like to see how "standard" this is. If it's really
standard and we have "official" standard docs to back this then I
agree that we should choose 2.
IMO, RFC 3986 is the current standard and the one we should follow, and
it does specify UTF-8 as the ONLY encoding, not just the default.
--
Sergiu Dumitriu
http://purl.org/net/sergiu/