6 ms·

Accept-Charset Is No More

54 points by charliesome 15y ago

nodata 15y ago
Interesting. Related: Can anyone explain to me how different versions of Unicode are handled?
- jensnockert 15y ago
  It isn't, they are backwards compatible.
- deleted 15y ago
  [deleted]
Eduard 15y ago
Huh, is this a hoax? My installation of Chrome still sends Accept-Charset for every HTTP GET request.
- hoppipolla 15y ago
  It seems like a weird thing to perpetrate a hoax about :) A more reasonable conclusion might be that the author made a mistake when testing Chrome. Having passed on the reports that Chrome still produces the header, the author has verified that this is indeed the case. I guess he will update the article.
shocks 15y ago
Chrome 16.0.912.77 m is sending Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 for me.
- hsivonen 15y ago
  I was wrong to claim Chrome didn't send Accept-Charset. Before I wrote the post, I opened another charset-related test case in Chrome (on two operating systems even) and after opening that URL and the relevant URL in various browsers thought I had opened the relevant test URL in all browsers I was testing, but apparently I hadn't opened it in Chrome on any OS. Very embarrassing. Sorry. So 4/5 OK, 1/5 still to go. Not quite newsworthy yet. :-(
TorKlingberg 15y ago
"There is no way for any non-UTF-8 encoding to represent something more, since everything gets converted to UTF-16 internally." Internally in what? Certainly not servers, but do all major browsers use UTF-16 internally? Also, does Unicode include all characters from various east Asian encodings now?
- hoppipolla 15y ago
  The DOM and ECMAScript both assume that strings are sequences of UTF-16 codepoints. So while a browser could use non-UTF-16 internally it wouldn't help much because you would have to convert to UTF-16 in all the externally-facing APIs anyway.
  - robin_reala 15y ago
    Nearly 100% true, but not quite: http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding
    - obtu 15y ago
      '𝌆'.length == 2 ಠ_ಠ They did the same kludge as the old narrow builds of Python.
  - obtu 15y ago
    JS engines use various strategies depending on actual string content because it's more memory- and cache-friendly.
- FigBug 15y ago
  Windows is UTF-16 internally, unless using the old ASCII functions. NSString on the Mac is UTF-16. I think w_char in Linux is UTF-32. I still think its an awkward statement. It should say Unicode internally, the specific encoding is irrelevant.
  - evmar 15y ago
    Even on Linux WebKit and Gecko use UTF-16.
NelsonMinar 15y ago
Does anyone know the historical justification for charset negotiation in the first place? I guess the idea was that the browser and the server would agree on what encoding they'd use and the server would auto-translate for the web visitor. But I can't imagine that ever being used in practice.
- pilif 15y ago
  The idea wasn't necessarily for the server to translate because back when this was invented, there was no Unicode and no UTF-8, so there might very well have been no way to losslessly translate. The client would send an Accept-Charset header to tell the server what it supports and the server would send either the content in the appropriate encoding (if available) or a "406: Not Acceptable" error.
  - NelsonMinar 15y ago
    I think Unicode predates Accept-Charset. Unicode 1.0 was published late 1991. Accept-Charset was formalized in the HTTP 1.0 RFC (1996). Usage presumably predates that, but it's not described in the HTTP 0.9 docs (which say ASCII only!) http://www.w3.org/Protocols/HTTP/AsImplemented.html http://www.w3.org/Protocols/HTTP/AsImplemented.html But what really matters is widespread acceptance of Unicode and UTF-8 and that was definitely later in coming. Thanks to dchest below for referencing a transcoding server. It's interesting the HTTP/1.0 docs characterize Accept-Encoding as a way to signal the client could handle something other than ASCII and ISO-8859-1. Google started favoring UTF-8 in search result pages somewhere around 2002 or 2003.
- derleth 15y ago
  > Does anyone know the historical justification for charset negotiation in the first place? Back in the Old Days (say, 1995), each OS (family) would have its own variant of 'Extended ASCII': Unix had Latin-1, Windows had its own encoding mostly compatible with Latin-1, and Macintosh computers had Mac OS Roman, aka MacRoman or just macintosh. It's possible to convert most, say, French texts from any of these to any other without losing anything, but if you don't the other person will have garbage on their end. In theory, if a client said it could only handle a charset that didn't include some of the characters in the source document, you could do something like give them an HTML file that used images to represent what they didn't have. Or something.

54 points by charliesome 15y ago

nodata 15y ago

Interesting. Related: Can anyone explain to me how different versions of Unicode are handled?

jensnockert 15y ago
It isn't, they are backwards compatible.
deleted 15y ago
[deleted]

Eduard 15y ago

Huh, is this a hoax? My installation of Chrome still sends Accept-Charset for every HTTP GET request.

hoppipolla 15y ago
It seems like a weird thing to perpetrate a hoax about :) A more reasonable conclusion might be that the author made a mistake when testing Chrome. Having passed on the reports that Chrome still produces the header, the author has verified that this is indeed the case. I guess he will update the article.

shocks 15y ago

Chrome 16.0.912.77 m is sending Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 for me.

hsivonen 15y ago
I was wrong to claim Chrome didn't send Accept-Charset. Before I wrote the post, I opened another charset-related test case in Chrome (on two operating systems even) and after opening that URL and the relevant URL in various browsers thought I had opened the relevant test URL in all browsers I was testing, but apparently I hadn't opened it in Chrome on any OS. Very embarrassing. Sorry. So 4/5 OK, 1/5 still to go. Not quite newsworthy yet. :-(

TorKlingberg 15y ago

"There is no way for any non-UTF-8 encoding to represent something more, since everything gets converted to UTF-16 internally." Internally in what? Certainly not servers, but do all major browsers use UTF-16 internally? Also, does Unicode include all characters from various east Asian encodings now?

hoppipolla 15y ago
The DOM and ECMAScript both assume that strings are sequences of UTF-16 codepoints. So while a browser could use non-UTF-16 internally it wouldn't help much because you would have to convert to UTF-16 in all the externally-facing APIs anyway.
- robin_reala 15y ago
  Nearly 100% true, but not quite: http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding
  - obtu 15y ago
    '𝌆'.length == 2 ಠ_ಠ They did the same kludge as the old narrow builds of Python.
- obtu 15y ago
  JS engines use various strategies depending on actual string content because it's more memory- and cache-friendly.
FigBug 15y ago
Windows is UTF-16 internally, unless using the old ASCII functions. NSString on the Mac is UTF-16. I think w_char in Linux is UTF-32. I still think its an awkward statement. It should say Unicode internally, the specific encoding is irrelevant.
- evmar 15y ago
  Even on Linux WebKit and Gecko use UTF-16.

NelsonMinar 15y ago

Does anyone know the historical justification for charset negotiation in the first place? I guess the idea was that the browser and the server would agree on what encoding they'd use and the server would auto-translate for the web visitor. But I can't imagine that ever being used in practice.

pilif 15y ago
The idea wasn't necessarily for the server to translate because back when this was invented, there was no Unicode and no UTF-8, so there might very well have been no way to losslessly translate. The client would send an Accept-Charset header to tell the server what it supports and the server would send either the content in the appropriate encoding (if available) or a "406: Not Acceptable" error.
- NelsonMinar 15y ago
  I think Unicode predates Accept-Charset. Unicode 1.0 was published late 1991. Accept-Charset was formalized in the HTTP 1.0 RFC (1996). Usage presumably predates that, but it's not described in the HTTP 0.9 docs (which say ASCII only!) http://www.w3.org/Protocols/HTTP/AsImplemented.html http://www.w3.org/Protocols/HTTP/AsImplemented.html But what really matters is widespread acceptance of Unicode and UTF-8 and that was definitely later in coming. Thanks to dchest below for referencing a transcoding server. It's interesting the HTTP/1.0 docs characterize Accept-Encoding as a way to signal the client could handle something other than ASCII and ISO-8859-1. Google started favoring UTF-8 in search result pages somewhere around 2002 or 2003.
derleth 15y ago
> Does anyone know the historical justification for charset negotiation in the first place? Back in the Old Days (say, 1995), each OS (family) would have its own variant of 'Extended ASCII': Unix had Latin-1, Windows had its own encoding mostly compatible with Latin-1, and Macintosh computers had Mac OS Roman, aka MacRoman or just macintosh. It's possible to convert most, say, French texts from any of these to any other without losing anything, but if you don't the other person will have garbage on their end. In theory, if a client said it could only handle a charset that didn't include some of the characters in the source document, you could do something like give them an HTML file that used images to represent what they didn't have. Or something.