11 ms·

Why Unicode Won’t Work on the Internet (2001)

102 points by jordigh 10y ago

TazeTSchnitzel 10y ago
UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. > The current permutation of Unicode gives a theoretical maximum of approximately 65,000 characters No, UTF-16 enables a maximum of 2,097,152 characters (2^21). > Clearly, 32 bits (4 octets) would have been more than adequate if they were a contiguous block. Indeed, "18 bits wide" (262,144 variations) would be enough to address the world’s characters if a contiguous block. UTF-16 provides 21 bits, 3 more than the author wants. Except they're not “in a contiguous block”: > But two separate 16 bit blocks do not solve the problem at all. The author doesn't explain why having multiple blocks is a problem. This works just fine, and has enabled Unicode to accommodate the hundreds of thousands of extra characters the author said it ought to. Though maybe there's a hint in this later comment: > One can easily formulate new standards using 4 octet blocks (ad infinitum) – but piggybacking them on top of Unicode 3.1 simply exacerbates the complexity of font mapping, as Unicode 3.1 has increased the complexity of UCS-2. They would have preferred if backwards-compatibility had been broken and everyone switched to a new format that's like UTF-32/UCS-4, but not called Unicode, I guess?
- masklinn 10y ago
  > UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. Hell UTF-8 was devised in mid-1992 and presented at USENIX in 1993. > No, UTF-16 enables a maximum of 2,097,152 characters (2^21). And until 2003 (and RFC 3629 which neutered it to match UTF-16) UTF-8 enabled 2,147,483,648 characters (2^31).
- dsp1234 10y ago
  UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. UTF-16 was only made into a standard in 2000[0] The first non-BMP characters weren't introduced until 2001[1] From a historical perspective, less than a month after Unicode 3.1 was released officially, it was not clear that Unicode (the standard, not the technical implementation of the encoding) would actually work. [0] - https://tools.ietf.org/html/rfc2781 https://tools.ietf.org/html/rfc2781 [1] - http://www.unicode.org/reports/tr27/tr27-4.html http://www.unicode.org/reports/tr27/tr27-4.html
  - TazeTSchnitzel 10y ago
    > UTF-16 was only made into a standard in 2000 It only got its own IETF RFC in 2000, but the surrogate pair mechanism was first specified in 1996's Unicode 2.0.
- klodolph 10y ago
  The maximum number of characters in Unicode is 1111998 (this excludes non-characters and surrogates). UTF-16 can encode 1112064 code points (this includes non-characters). This is quite a bit smaller than the 21 bits quoted: you really get the 16-bit BMP plus the 20-bit astral planes, minus the surrogates (2048 of them). The "non-characters" number 66.
  - TazeTSchnitzel 10y ago
    Thank you for that correction, I suspected 2^21 might not be quite correct.
  - vorg 10y ago
    > The maximum number of characters ... (this excludes non-characters and surrogates) You should exclude the surrogates but include the non-characters when tallying the characters. According to Unicode's 2013 Corridendum 9, "Noncharacters in the Unicode Standard are intended for internal use and have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange nor do they cause ill-formed Unicode text". See http://www.unicode.org/versions/corrigendum9.html http://www.unicode.org/versions/corrigendum9.html
    - klodolph 10y ago
      I can see why this might be confusing—I chose the word "character" carefully. The annex you cited clarifies that noncharacters are code points that can be used in interchange. However, this does not mean that a noncharacter is a character. The number of code points available for use as characters must, by definition, exclude the noncharacters. So, to expand on the original comment, Unicode defines 1114112 code points, of which 1112064 can be used in interchange, and 1111998 can be defined as characters. UTF-16 can only represent the 1112064 that are valid for interchange, and the 66 noncharacters should generally be avoided (especially U+FFFE).
- jordigh 10y ago
  Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understand most of how UTF-8 works, I still have to read and re-read how codepoints and planes and encodings and decodings work together. It's a pretty complicated beast that could very easily be misunderstood when it was less popular than it is now. It's still widely misunderstood today.
  - TazeTSchnitzel 10y ago
    I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.
    - 0x0 10y ago
      It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.
herge 10y ago
Man, UCS-2 is the pits. I still remember fighting with 'slim-builds' of python back in the day. Any critique of unicode while not assuming UTF-8, which allows for more than 1 million code points) is a bit suspect in my opinion. The biggest point against UTF-8 might be that it takes more space than 'local' encodings for asian languages.
- achamayou 10y ago
  Which really isn't a very serious concern at this point, considering how cheap storage and compression are. What's expensive to store are images and sound, from an ever increasing number of devices at an ever higher resolution. The production and storage of text barely registers in comparison.
  - imagist 10y ago
    Text size is still relevant due to bandwidth and reliability of transmission. Not everyone has a gigabit internet connection--large portions of the world are still operating on 2G wireless, or even dialup.
    - ssalazar 10y ago
      Definitely. I would be interested to know of any good statistics on data size of real-world non-European language webpages in UTF-8 vs. UTF-16 and also with/without compression. Much of the markup will be smaller in UTF-8 but actual text content would be smaller in UTF-16.
      - creshal 10y ago
        And both end up being fed to optimized-deflate-encoder-of-the-week and have negligible size differences once compressed.
      - dom0 10y ago
        See comment above parent: > This happens for pure text[nb 2] but actual documents often contain enough spaces and line terminators, numbers (digits 0–9), and HTML or XML or wiki markup characters, that they are shorter in UTF-8. For example, both the Japanese UTF-8 and the Hindi Unicode articles on Wikipedia take more space in UTF-16 than in UTF-8.[nb 3]
mcaruso 10y ago
More like, "Why UCS-2 Won’t Work on the Internet".
oconnor663 10y ago
> Thus is can be said that Hiragana can form pictures but Katakana can only form sounds That sounds really weird to me. Does that sound right to any native Japanese speakers here?
- xigency 10y ago
  This writing is sort of a strange metaphor, but I guess the point the author makes is that kanji can be transliterated as hiragana but not katakana. The writer goes on to talk about traumatic brain injuries so I guess he's aiming at the cultural value of each syllabary. I'm not a native speaker, but if I were to make an equally strange metaphor as the author, katakana feels like writing in all capital letters.
  - achamayou 10y ago
    Kanji can be transliterated either way, and both forms are lossy since there are so many homonyms and kana only encodes sounds. It's traditional to annotate difficult Kanji pronunciation with small Hiragana called Furigana, for example in children's books. But it could be done all the same in Katakana. Modern Chinese words that Japanese borrows are usually translated in Katakana for example.
    - klodolph 10y ago
      Kana mostly contain just sounds, but do contain some morphological information—there are homonyms in kana as well, after all. This is a bit rare, however.
    - xigency 10y ago
      While possible phonetically, it would be strange to see kanji transliterated as katakana. But I guess there are some Japanese sources that are written all in katakana.
  - kalleboo 10y ago
    There are situations where Kanji get transliterated to katakana - many name input forms will ask for the reading of your kanji name in katakana
lcuff 10y ago
This article was written before UTF-8 became the de-facto standard. According to Wikipedia, UTF-8 encodes each of the 1,112,064 valid code points. Much more than Goundry's (the author's) 170,000. Goundry's only complaint against UTF-8 is that at the time, it was one of three possible encoding formats that might work. Since it has now been widely embraced, the complaint is no longer valid. In short, Unicode will work just fine on the internet in 2016 as far as encoding all the characters goes. Problems having to do with how ordinal numbers are used, right-to-left languages, upper-case/lower-case anomalies, different glyphs being used for the same letter depending on the letter's position in the word (and many other realities of language and script differences) all need to be in the forefront of a developer's mind when trying to build a multi-lingual site.
- WayneBro 10y ago
  > In short, Unicode will work just fine... > Problems............all need to be in the forefront of a developer's mind when trying to build a multi-lingual site. It will work. Just fine though? It sounds like way too much work!
  - Dylan16807 10y ago
    Unicode handily solves the problem of storing text. Manipulating text, though, is inherently nightmarish. No format can prevent that.
  - deleted 10y ago
    [deleted]
- syncsynchalt 10y ago
  This article is written years after the 16-bit problem was solved, yet seems to be completely unaware of the solution. (It's unclear if the mentions of adding "octet blocks" is a misunderstanding of the solution, or a dismissal.) It mentions Unicode 3.1, yet surrogate pairs were introduced in Unicode 2.0 five years before the doc was written. In actuality we don't concern ourselves overmuch with planes, "octet blocks", and so on these days. There's just a single numbered list of characters* that we call "Unicode", and a few fairly minimal algorithms for transforming a list of characters into a bytestream ("UTF-8", "UTF-16", etc). Still I love it as a historically interesting document. It's jarring to see "Oriental" used, I suspect the same author would never use that word in a professional context today. [*] I use "characters" here when "code points" would be more accurate, but the details of that comparison aren't meaningful to this argument. I also don't go into normalization and so on, as this article seems to be more about the feasibility of fitting >100K code points into actual-bits-on-the-wire.
jimjimjim 10y ago
http://utf8everywhere.org/ http://utf8everywhere.org/ a very useful site especially when having to explain what utf8 is to other devs when working in a windows shop.
- wmccullough 10y ago
  'working in a windows shop.' Surely you're flattering yourself.
  - jimjimjim 10y ago
    no seriously. I'm a windows application dev, and I have been for >decade. If all you see around you is wchar_t and LPCWSTR then that is what unicode means.
- yuhong 10y ago
  Reminds me of: https://blogs.msdn.microsoft.com/oldnewthing/20161007-00/?p=94475#comment-1267826 https://blogs.msdn.microsoft.com/oldnewthing/20161007-00/?p=...
reality_czech 10y ago
Hilarious, a document from 2001 talking about why Unicode is unsuitable to "the orient." At the end, I half expected to read that "Negroes have also proved to be most unfavorable to it."
- mark_edward 10y ago
  Maybe a foreign language speaker? The term didn't get the same valences in places that didn't colonize it
- darkengine 10y ago
  Simply because of his use of an outdated synonym of "Asia"? If anything, I got the impression that the author was critical of Westerners for being insensitive to the needs of Asian computer users. I think this is because of Han Unification, but he does not mention it by name.
  - sundarurfriend 10y ago
    In this author's case, I agree that the intent doesn't appear to have been malicious. But calling it just "an outdated synonym of Asia" is like saying Negro is just an outdated synonym for Black (after all, negro is just Spanish for black). In both cases, it ignores heavy historical and cultural implications that come with the words.
    - hueving 10y ago
      What are the cultural implications of using Orient as a synonym for Asia?
khaled 10y ago
An extensive and very informative, though a bit sarcastic, rebuttal (from 2001 as well): https://features.slashdot.org/story/01/06/06/0132203/why-unicode-will-work-on-the-internet https://features.slashdot.org/story/01/06/06/0132203/why-uni... (via https://twitter.com/FakeUnicode/status/786324531828838400 https://twitter.com/FakeUnicode/status/786324531828838400).
jbmorgado 10y ago
Why "640K ought to be enough to everyone"
- Dylan16807 10y ago
  The opposite, really. It's arguing why 16 bits is not enough. But it does this while casually dismissing the solutions we already had.
zoom6628 10y ago
The paper is far more interesting for its informative background on the the use of the character sets in CJK region.
david90 10y ago
Good to see we've had a breakthrough after 15+ years.
bombless 10y ago
Chinese developers have happily adopted the utf8 solution but I believe it is not that popular in Japan and Taiwan today.
jrcii 10y ago
Oriental is not an offensive word, anyone who thinks it is should grow a pair and get a life. The hypersensitive PC social justice crap is so wearisome.
- luxpir 10y ago
  Not sure you're contributing a lot to the conversation there. Probably annoying a bunch of folks in the process. You probably know all that though. But you're right. We love being called orientals. It's just that warm feeling of being categorised and separated from the occidentals in that lovely nostalgic way the PC guys just don't get.
  - cafard 10y ago
    And the odd thing is that orient = Asia doesn't make sense in an American context...
- eyelidlessness 10y ago
  There are no offensive words, per se. The term "oriental" is problematic because it represents a world view that nearly no one possesses. It represents a colonial mindset that is hilariously ignorant of reality, which no one from the originating culture uses anymore. It's a plainly stupid word, with no meaning besides its ignorance. You can use the word "cunt" if you want to, but if you don't wish to be ignorant, you should use it carefully. If you're upset that the language you use has been scrutinized, perhaps you should familiarize yourself with the scrutiny and understand why it doesn't carry much meaning to your audience. No one is censoring you! If your language is stupid, you're being told it's stupid. That's not political correctness, it's just correctness. Language criticism isn't about sensitivity. It's about expression: clarity of meaning, intent. In every supposedly "PC" community, you'll find reclaiming of terms otherwise used ignorantly. You mentioned "growing a pair" in your comment. Your usage is clearly a reference to testicles (though many who would reclaim the turn of phrase would reference ovaries as well). Your usage doesn't express any meaning. What does it mean to have a "pair"? What do you have that the targets of your vitriol lack? Do you absorb callousness with your balls? You clearly can't handle genuine discussion of how the realities of "words" and what they represent affect other human beings. Are you so tiny that you have to shove your balls into every conversation about things that don't fucking concern you?

102 points by jordigh 10y ago

UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. > The current permutation of Unicode gives a theoretical maximum of approximately 65,000 characters No, UTF-16 enables a maximum of 2,097,152 characters (2^21). > Clearly, 32 bits (4 octets) would have been more than adequate if they were a contiguous block. Indeed, "18 bits wide" (262,144 variations) would be enough to address the world’s characters if a contiguous block. UTF-16 provides 21 bits, 3 more than the author wants. Except they're not “in a contiguous block”: > But two separate 16 bit blocks do not solve the problem at all. The author doesn't explain why having multiple blocks is a problem. This works just fine, and has enabled Unicode to accommodate the hundreds of thousands of extra characters the author said it ought to. Though maybe there's a hint in this later comment: > One can easily formulate new standards using 4 octet blocks (ad infinitum) – but piggybacking them on top of Unicode 3.1 simply exacerbates the complexity of font mapping, as Unicode 3.1 has increased the complexity of UCS-2. They would have preferred if backwards-compatibility had been broken and everyone switched to a new format that's like UTF-32/UCS-4, but not called Unicode, I guess?

masklinn 10y ago
> UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. Hell UTF-8 was devised in mid-1992 and presented at USENIX in 1993. > No, UTF-16 enables a maximum of 2,097,152 characters (2^21). And until 2003 (and RFC 3629 which neutered it to match UTF-16) UTF-8 enabled 2,147,483,648 characters (2^31).
dsp1234 10y ago
UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. UTF-16 was only made into a standard in 2000[0] The first non-BMP characters weren't introduced until 2001[1] From a historical perspective, less than a month after Unicode 3.1 was released officially, it was not clear that Unicode (the standard, not the technical implementation of the encoding) would actually work. [0] - https://tools.ietf.org/html/rfc2781 https://tools.ietf.org/html/rfc2781 [1] - http://www.unicode.org/reports/tr27/tr27-4.html http://www.unicode.org/reports/tr27/tr27-4.html
- TazeTSchnitzel 10y ago
  > UTF-16 was only made into a standard in 2000 It only got its own IETF RFC in 2000, but the surrogate pair mechanism was first specified in 1996's Unicode 2.0.
klodolph 10y ago
The maximum number of characters in Unicode is 1111998 (this excludes non-characters and surrogates). UTF-16 can encode 1112064 code points (this includes non-characters). This is quite a bit smaller than the 21 bits quoted: you really get the 16-bit BMP plus the 20-bit astral planes, minus the surrogates (2048 of them). The "non-characters" number 66.
- TazeTSchnitzel 10y ago
  Thank you for that correction, I suspected 2^21 might not be quite correct.
- vorg 10y ago
  > The maximum number of characters ... (this excludes non-characters and surrogates) You should exclude the surrogates but include the non-characters when tallying the characters. According to Unicode's 2013 Corridendum 9, "Noncharacters in the Unicode Standard are intended for internal use and have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange nor do they cause ill-formed Unicode text". See http://www.unicode.org/versions/corrigendum9.html http://www.unicode.org/versions/corrigendum9.html
  - klodolph 10y ago
    I can see why this might be confusing—I chose the word "character" carefully. The annex you cited clarifies that noncharacters are code points that can be used in interchange. However, this does not mean that a noncharacter is a character. The number of code points available for use as characters must, by definition, exclude the noncharacters. So, to expand on the original comment, Unicode defines 1114112 code points, of which 1112064 can be used in interchange, and 1111998 can be defined as characters. UTF-16 can only represent the 1112064 that are valid for interchange, and the 66 noncharacters should generally be avoided (especially U+FFFE).
jordigh 10y ago
Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understand most of how UTF-8 works, I still have to read and re-read how codepoints and planes and encodings and decodings work together. It's a pretty complicated beast that could very easily be misunderstood when it was less popular than it is now. It's still widely misunderstood today.
- TazeTSchnitzel 10y ago
  I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.
  - 0x0 10y ago
    It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.

herge 10y ago

Man, UCS-2 is the pits. I still remember fighting with 'slim-builds' of python back in the day. Any critique of unicode while not assuming UTF-8, which allows for more than 1 million code points) is a bit suspect in my opinion. The biggest point against UTF-8 might be that it takes more space than 'local' encodings for asian languages.

achamayou 10y ago
Which really isn't a very serious concern at this point, considering how cheap storage and compression are. What's expensive to store are images and sound, from an ever increasing number of devices at an ever higher resolution. The production and storage of text barely registers in comparison.
- imagist 10y ago
  Text size is still relevant due to bandwidth and reliability of transmission. Not everyone has a gigabit internet connection--large portions of the world are still operating on 2G wireless, or even dialup.
  - ssalazar 10y ago
    Definitely. I would be interested to know of any good statistics on data size of real-world non-European language webpages in UTF-8 vs. UTF-16 and also with/without compression. Much of the markup will be smaller in UTF-8 but actual text content would be smaller in UTF-16.
    - creshal 10y ago
      And both end up being fed to optimized-deflate-encoder-of-the-week and have negligible size differences once compressed.
    - dom0 10y ago
      See comment above parent: > This happens for pure text[nb 2] but actual documents often contain enough spaces and line terminators, numbers (digits 0–9), and HTML or XML or wiki markup characters, that they are shorter in UTF-8. For example, both the Japanese UTF-8 and the Hindi Unicode articles on Wikipedia take more space in UTF-16 than in UTF-8.[nb 3]

mcaruso 10y ago

More like, "Why UCS-2 Won’t Work on the Internet".

oconnor663 10y ago

> Thus is can be said that Hiragana can form pictures but Katakana can only form sounds That sounds really weird to me. Does that sound right to any native Japanese speakers here?

xigency 10y ago
This writing is sort of a strange metaphor, but I guess the point the author makes is that kanji can be transliterated as hiragana but not katakana. The writer goes on to talk about traumatic brain injuries so I guess he's aiming at the cultural value of each syllabary. I'm not a native speaker, but if I were to make an equally strange metaphor as the author, katakana feels like writing in all capital letters.
- achamayou 10y ago
  Kanji can be transliterated either way, and both forms are lossy since there are so many homonyms and kana only encodes sounds. It's traditional to annotate difficult Kanji pronunciation with small Hiragana called Furigana, for example in children's books. But it could be done all the same in Katakana. Modern Chinese words that Japanese borrows are usually translated in Katakana for example.
  - klodolph 10y ago
    Kana mostly contain just sounds, but do contain some morphological information—there are homonyms in kana as well, after all. This is a bit rare, however.
  - xigency 10y ago
    While possible phonetically, it would be strange to see kanji transliterated as katakana. But I guess there are some Japanese sources that are written all in katakana.
- kalleboo 10y ago
  There are situations where Kanji get transliterated to katakana - many name input forms will ask for the reading of your kanji name in katakana

lcuff 10y ago

This article was written before UTF-8 became the de-facto standard. According to Wikipedia, UTF-8 encodes each of the 1,112,064 valid code points. Much more than Goundry's (the author's) 170,000. Goundry's only complaint against UTF-8 is that at the time, it was one of three possible encoding formats that might work. Since it has now been widely embraced, the complaint is no longer valid. In short, Unicode will work just fine on the internet in 2016 as far as encoding all the characters goes. Problems having to do with how ordinal numbers are used, right-to-left languages, upper-case/lower-case anomalies, different glyphs being used for the same letter depending on the letter's position in the word (and many other realities of language and script differences) all need to be in the forefront of a developer's mind when trying to build a multi-lingual site.

WayneBro 10y ago
> In short, Unicode will work just fine... > Problems............all need to be in the forefront of a developer's mind when trying to build a multi-lingual site. It will work. Just fine though? It sounds like way too much work!
- Dylan16807 10y ago
  Unicode handily solves the problem of storing text. Manipulating text, though, is inherently nightmarish. No format can prevent that.
- deleted 10y ago
  [deleted]
syncsynchalt 10y ago
This article is written years after the 16-bit problem was solved, yet seems to be completely unaware of the solution. (It's unclear if the mentions of adding "octet blocks" is a misunderstanding of the solution, or a dismissal.) It mentions Unicode 3.1, yet surrogate pairs were introduced in Unicode 2.0 five years before the doc was written. In actuality we don't concern ourselves overmuch with planes, "octet blocks", and so on these days. There's just a single numbered list of characters* that we call "Unicode", and a few fairly minimal algorithms for transforming a list of characters into a bytestream ("UTF-8", "UTF-16", etc). Still I love it as a historically interesting document. It's jarring to see "Oriental" used, I suspect the same author would never use that word in a professional context today. [*] I use "characters" here when "code points" would be more accurate, but the details of that comparison aren't meaningful to this argument. I also don't go into normalization and so on, as this article seems to be more about the feasibility of fitting >100K code points into actual-bits-on-the-wire.

jimjimjim 10y ago

http://utf8everywhere.org/ http://utf8everywhere.org/ a very useful site especially when having to explain what utf8 is to other devs when working in a windows shop.

wmccullough 10y ago
'working in a windows shop.' Surely you're flattering yourself.
- jimjimjim 10y ago
  no seriously. I'm a windows application dev, and I have been for >decade. If all you see around you is wchar_t and LPCWSTR then that is what unicode means.
yuhong 10y ago
Reminds me of: https://blogs.msdn.microsoft.com/oldnewthing/20161007-00/?p=94475#comment-1267826 https://blogs.msdn.microsoft.com/oldnewthing/20161007-00/?p=...

reality_czech 10y ago

Hilarious, a document from 2001 talking about why Unicode is unsuitable to "the orient." At the end, I half expected to read that "Negroes have also proved to be most unfavorable to it."

mark_edward 10y ago
Maybe a foreign language speaker? The term didn't get the same valences in places that didn't colonize it
darkengine 10y ago
Simply because of his use of an outdated synonym of "Asia"? If anything, I got the impression that the author was critical of Westerners for being insensitive to the needs of Asian computer users. I think this is because of Han Unification, but he does not mention it by name.
- sundarurfriend 10y ago
  In this author's case, I agree that the intent doesn't appear to have been malicious. But calling it just "an outdated synonym of Asia" is like saying Negro is just an outdated synonym for Black (after all, negro is just Spanish for black). In both cases, it ignores heavy historical and cultural implications that come with the words.
  - hueving 10y ago
    What are the cultural implications of using Orient as a synonym for Asia?

khaled 10y ago

An extensive and very informative, though a bit sarcastic, rebuttal (from 2001 as well): https://features.slashdot.org/story/01/06/06/0132203/why-unicode-will-work-on-the-internet https://features.slashdot.org/story/01/06/06/0132203/why-uni... (via https://twitter.com/FakeUnicode/status/786324531828838400 https://twitter.com/FakeUnicode/status/786324531828838400).

jbmorgado 10y ago

Why "640K ought to be enough to everyone"

Dylan16807 10y ago
The opposite, really. It's arguing why 16 bits is not enough. But it does this while casually dismissing the solutions we already had.

zoom6628 10y ago

The paper is far more interesting for its informative background on the the use of the character sets in CJK region.

david90 10y ago

Good to see we've had a breakthrough after 15+ years.

bombless 10y ago

Chinese developers have happily adopted the utf8 solution but I believe it is not that popular in Japan and Taiwan today.

jrcii 10y ago

Oriental is not an offensive word, anyone who thinks it is should grow a pair and get a life. The hypersensitive PC social justice crap is so wearisome.

luxpir 10y ago
Not sure you're contributing a lot to the conversation there. Probably annoying a bunch of folks in the process. You probably know all that though. But you're right. We love being called orientals. It's just that warm feeling of being categorised and separated from the occidentals in that lovely nostalgic way the PC guys just don't get.
- cafard 10y ago
  And the odd thing is that orient = Asia doesn't make sense in an American context...
eyelidlessness 10y ago
There are no offensive words, per se. The term "oriental" is problematic because it represents a world view that nearly no one possesses. It represents a colonial mindset that is hilariously ignorant of reality, which no one from the originating culture uses anymore. It's a plainly stupid word, with no meaning besides its ignorance. You can use the word "cunt" if you want to, but if you don't wish to be ignorant, you should use it carefully. If you're upset that the language you use has been scrutinized, perhaps you should familiarize yourself with the scrutiny and understand why it doesn't carry much meaning to your audience. No one is censoring you! If your language is stupid, you're being told it's stupid. That's not political correctness, it's just correctness. Language criticism isn't about sensitivity. It's about expression: clarity of meaning, intent. In every supposedly "PC" community, you'll find reclaiming of terms otherwise used ignorantly. You mentioned "growing a pair" in your comment. Your usage is clearly a reference to testicles (though many who would reclaim the turn of phrase would reference ovaries as well). Your usage doesn't express any meaning. What does it mean to have a "pair"? What do you have that the targets of your vitriol lack? Do you absorb callousness with your balls? You clearly can't handle genuine discussion of how the realities of "words" and what they represent affect other human beings. Are you so tiny that you have to shove your balls into every conversation about things that don't fucking concern you?