5 ms·

It always baffled me as a native Chinese speaker that 16-bit was ever considered enough to “be plenty of room to encode every script in modern use, and then som

It always baffled me as a native Chinese speaker that 16-bit was ever considered enough to “be plenty of room to encode every script in modern use, and then some”. Common local encodings like Big5 (also 16-bit fixwidth) was already suffering from serious lack of code points at that time. It would be obvious if they consulted literally any experienced programmer from East Asia. Yeah, that was 1990 so it’s not an easy thing to do by any means, but you’d expect more from someone aspiring to create such thing.

khuey 7y ago
This is actually covered in the original Unicode paper. See section 2.1 of https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf. The reasoning is not totally crazy; the Table of General Standard Chinese Characters even today only has about 8000 characters.
- unscaled 7y ago
  I think what they did not realize is that once a universal encoding won the battle of standards, it would have to be all-inclusive. Operating systems and platforms like Java settled on Unicode so they won't have to deal with a myriad of different encodings, but the moment you support only one encoding, you'll eventually have to accommodate everyone, and it's not enough to include all commonly used characters in the world's major languages. You'll have to make a go at the long tail. You can't leave out people who need to encode rare languages on computers underserved, so for all the scholars recording Egyptian and Luwian hieroglyphs, Sumerian and Akkadian cuneiform and Tangut script. You'd probably have to eventually support as-of-yet unencoded scripts such as Classical Yi, Jurchen, Aztec hieroglyphs and so on. You'd have to add alphabets and syllabaries for small minority languages such as Lisu, Coptic, Cherokee and Vai. And you'd have to support the rather widely used writing systems of India, each with its own 128 characters. Combine this with the over 28,000 CJK characters Unicode did end up allocating, the 11,184 discrete codepoints they Unicode did end up adding for Koran Hangul [1], you've already used 40,000 characters. Add all the long-tail scripts and you're past 65,535 well before we even reach Emoji or extremely rare CJKV characters from Mojikyo[1]. "In other words, given that the limitation to 65,536 character codes genuinely does satisfy all the world's modern communication needs with a safety factor of about four, then one can decide up-front that preserving a pure 16-bit architecture has a higher design priority than publicly encoding every extinct or obscure character form. Then the sufficiency of 16 bits for the writing technology of the future becomes a matter of our active intention, rather than passive victimization by writing systems of the past." The engineers behind Unicode made the classic software engineering mistake of looking at their needs today and taking a factor of safety that's way too small. It's the same thing that happened with the IPv4 address range. A simple back-of-the-envelope calculation would have told us it's not going to be enough, but it was easy to forget back in the 1980s that one day children in Africa would also have personal computers connected to the grand Internet, that would need an IP address. The fact that by version 1.0.1 (the first version of Unicode to include the CJK characters), they already used almost half of their space when they were planning to use less than a quarter, should have already prepared them to the future. No, the reasoning behind 16 bits was always crazy. You can't design future-proof standards with a low safety factor and condescending estimation of usage (Yi Syllabary? Who would ever use that?). But we should probably be happy about this. A 32-bit encoding would have certainly been rejected by just about everyone as too wasteful, and software engineers back in the 80's and 90's were still unable to give up merging their understanding of codepoints with glyhs, so a multi-byte encoding (which is what we ended up having eventually with UTF-8 and UTF-16) wasn't a popular idea either. Fortunately enough, Unicode is still much better than the mess it preceded, and the fix for the 16-bit limitation, as painful as it was, has proven far easier than IPv6. [1] a.k.a The "Korean Mess": http://unicode.org/pipermail/unicode/2015-June/002038.html http://unicode.org/pipermail/unicode/2015-June/002038.html [2] https://en.wikipedia.org/wiki/Mojikyo https://en.wikipedia.org/wiki/Mojikyo
Tuna-Fish 7y ago
> It would be obvious if they consulted literally any experienced programmer from East Asia. Yeah, that was 1990 so it’s not an easy thing to do by any means They did consult not just experienced East Asian programmers, but Chinese, Korean, Japanese and Vietnamese professors of Linguistics and the relevant local authorities, and had them form the Ideographic Rapporteur Group, which advised them that all the East Asian languages can be encoded in 20940 code points. Unicode is not centrally run like so many people seem to expect. Rather, the Unicode Consortium itself acts just as the standards body that mates together local standards into a single complete whole. The Consortium makes no decisions about how Chinese is represented, other than allocating them code pages; all the relevant decisions about any language is done by local experts. The problem was that the early 90's were an era of boundless optimism and futurism, which in East Asia resulted in the Han Unification project, with the idea of cutting down the amount of symbols in use, and unifying the representation of all the languages that used a Han-derived alphabet. That... did not go so well.
- SloopJon 7y ago
  Thanks for that background. I was thinking that the (for lack of a better word) politics of CJKV and Han unification likely had a lot to do with it. To my eyes, the letter "A" in English (Latin), Greek, and Russian (Cyrillic) looks identical. Why does it need three separate code points? Maybe it gets tricky when the relationship between uppercase and lowercase diverges. In fact, that does appear to be one of the arguments in this technical note: https://www.unicode.org/notes/tn26/ https://www.unicode.org/notes/tn26/
  - amaccuish 7y ago
    One thing I can think of is that when I write in Russian, and I put text into "italics", it goes into the cursive script. So I guess the letter "a" is different at least from the roman letter.
    - deleted 7y ago
      [deleted]
  - 7y ago
deleted 7y ago
[deleted]
lifthrasiir 7y ago
> Common local encodings like Big5 (also 16-bit fixwidth) was already suffering from serious lack of code points at that time. They are not 16 bits wide, but mostly 14 bits wide and separated from the single byte portion with most significant bits of bytes set. As far as I can tell there are "only" about 33,000 code points [1] in the Unihan database that are known to be officially encoded in some sort of encodings throughout the entire East Asia. This does not include unofficial but de facto encodings like ETEN extensions to Big5 but you get the idea. This and the Han unification made it pretty reasonable to expect that the modern characters can be encoded in 16 bits. [1] I've counted distinct code points in the "other mappings" table from the Unihan database.
garmaine 7y ago
There are not more than 65,000 Chinese characters in common use today. Indeed, a college graduate from this region would probably only recognize 1/10th that number, even from Taiwan or Hong Kong (where a greater number of traditional characters are used). If the goal is simply to encode one point for each character in use today, 16 bits is very likely sufficient. That's not what Unicode does though. Unicode encodes many duplicate characters (because they are duplicated in a national encoding standard and the round-trip compatibility requirement). Unicode also includes every possible variant that is of cultural or academic interest, including dead languages and so-called "ghost characters" that are actually printing errors in old paper dictionaries or census data, but now allocated a code point. And this is before we say anything about emoji... So from today's perspective 16 bit encoding space is laughably small. But it is true that basically any message meant to be consumed by the general population, including CJKV languages but notably not emoji, can be fully represented with characters found on the BMP, which can be faithfully encoded in UCS-2.
Koshkin 7y ago
But, but... 64K code points should be enough for everybody!

khuey 7y ago

This is actually covered in the original Unicode paper. See section 2.1 of https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf. The reasoning is not totally crazy; the Table of General Standard Chinese Characters even today only has about 8000 characters.

unscaled 7y ago
I think what they did not realize is that once a universal encoding won the battle of standards, it would have to be all-inclusive. Operating systems and platforms like Java settled on Unicode so they won't have to deal with a myriad of different encodings, but the moment you support only one encoding, you'll eventually have to accommodate everyone, and it's not enough to include all commonly used characters in the world's major languages. You'll have to make a go at the long tail. You can't leave out people who need to encode rare languages on computers underserved, so for all the scholars recording Egyptian and Luwian hieroglyphs, Sumerian and Akkadian cuneiform and Tangut script. You'd probably have to eventually support as-of-yet unencoded scripts such as Classical Yi, Jurchen, Aztec hieroglyphs and so on. You'd have to add alphabets and syllabaries for small minority languages such as Lisu, Coptic, Cherokee and Vai. And you'd have to support the rather widely used writing systems of India, each with its own 128 characters. Combine this with the over 28,000 CJK characters Unicode did end up allocating, the 11,184 discrete codepoints they Unicode did end up adding for Koran Hangul [1], you've already used 40,000 characters. Add all the long-tail scripts and you're past 65,535 well before we even reach Emoji or extremely rare CJKV characters from Mojikyo[1]. "In other words, given that the limitation to 65,536 character codes genuinely does satisfy all the world's modern communication needs with a safety factor of about four, then one can decide up-front that preserving a pure 16-bit architecture has a higher design priority than publicly encoding every extinct or obscure character form. Then the sufficiency of 16 bits for the writing technology of the future becomes a matter of our active intention, rather than passive victimization by writing systems of the past." The engineers behind Unicode made the classic software engineering mistake of looking at their needs today and taking a factor of safety that's way too small. It's the same thing that happened with the IPv4 address range. A simple back-of-the-envelope calculation would have told us it's not going to be enough, but it was easy to forget back in the 1980s that one day children in Africa would also have personal computers connected to the grand Internet, that would need an IP address. The fact that by version 1.0.1 (the first version of Unicode to include the CJK characters), they already used almost half of their space when they were planning to use less than a quarter, should have already prepared them to the future. No, the reasoning behind 16 bits was always crazy. You can't design future-proof standards with a low safety factor and condescending estimation of usage (Yi Syllabary? Who would ever use that?). But we should probably be happy about this. A 32-bit encoding would have certainly been rejected by just about everyone as too wasteful, and software engineers back in the 80's and 90's were still unable to give up merging their understanding of codepoints with glyhs, so a multi-byte encoding (which is what we ended up having eventually with UTF-8 and UTF-16) wasn't a popular idea either. Fortunately enough, Unicode is still much better than the mess it preceded, and the fix for the 16-bit limitation, as painful as it was, has proven far easier than IPv6. [1] a.k.a The "Korean Mess": http://unicode.org/pipermail/unicode/2015-June/002038.html http://unicode.org/pipermail/unicode/2015-June/002038.html [2] https://en.wikipedia.org/wiki/Mojikyo https://en.wikipedia.org/wiki/Mojikyo

Tuna-Fish 7y ago

> It would be obvious if they consulted literally any experienced programmer from East Asia. Yeah, that was 1990 so it’s not an easy thing to do by any means They did consult not just experienced East Asian programmers, but Chinese, Korean, Japanese and Vietnamese professors of Linguistics and the relevant local authorities, and had them form the Ideographic Rapporteur Group, which advised them that all the East Asian languages can be encoded in 20940 code points. Unicode is not centrally run like so many people seem to expect. Rather, the Unicode Consortium itself acts just as the standards body that mates together local standards into a single complete whole. The Consortium makes no decisions about how Chinese is represented, other than allocating them code pages; all the relevant decisions about any language is done by local experts. The problem was that the early 90's were an era of boundless optimism and futurism, which in East Asia resulted in the Han Unification project, with the idea of cutting down the amount of symbols in use, and unifying the representation of all the languages that used a Han-derived alphabet. That... did not go so well.

SloopJon 7y ago
Thanks for that background. I was thinking that the (for lack of a better word) politics of CJKV and Han unification likely had a lot to do with it. To my eyes, the letter "A" in English (Latin), Greek, and Russian (Cyrillic) looks identical. Why does it need three separate code points? Maybe it gets tricky when the relationship between uppercase and lowercase diverges. In fact, that does appear to be one of the arguments in this technical note: https://www.unicode.org/notes/tn26/ https://www.unicode.org/notes/tn26/
- amaccuish 7y ago
  One thing I can think of is that when I write in Russian, and I put text into "italics", it goes into the cursive script. So I guess the letter "a" is different at least from the roman letter.
  - deleted 7y ago
    [deleted]
- 7y ago

deleted 7y ago

[deleted]

lifthrasiir 7y ago

> Common local encodings like Big5 (also 16-bit fixwidth) was already suffering from serious lack of code points at that time. They are not 16 bits wide, but mostly 14 bits wide and separated from the single byte portion with most significant bits of bytes set. As far as I can tell there are "only" about 33,000 code points [1] in the Unihan database that are known to be officially encoded in some sort of encodings throughout the entire East Asia. This does not include unofficial but de facto encodings like ETEN extensions to Big5 but you get the idea. This and the Han unification made it pretty reasonable to expect that the modern characters can be encoded in 16 bits. [1] I've counted distinct code points in the "other mappings" table from the Unihan database.

garmaine 7y ago

There are not more than 65,000 Chinese characters in common use today. Indeed, a college graduate from this region would probably only recognize 1/10th that number, even from Taiwan or Hong Kong (where a greater number of traditional characters are used). If the goal is simply to encode one point for each character in use today, 16 bits is very likely sufficient. That's not what Unicode does though. Unicode encodes many duplicate characters (because they are duplicated in a national encoding standard and the round-trip compatibility requirement). Unicode also includes every possible variant that is of cultural or academic interest, including dead languages and so-called "ghost characters" that are actually printing errors in old paper dictionaries or census data, but now allocated a code point. And this is before we say anything about emoji... So from today's perspective 16 bit encoding space is laughably small. But it is true that basically any message meant to be consumed by the general population, including CJKV languages but notably not emoji, can be fully represented with characters found on the BMP, which can be faithfully encoded in UCS-2.

Koshkin 7y ago

But, but... 64K code points should be enough for everybody!