4 ms·
> The underlying issue is that Unicode was run by people who thought 16 bits was enough[0] It really depends on how you look at it. I wouldn't fault the origin
by hnfong 2y ago
> The underlying issue is that Unicode was run by people who thought 16 bits was enough[0]
It really depends on how you look at it. I wouldn't fault the original designers of unicode who thought 16 bits was enough.
In modern unicode there are apparently 90k+ CJK characters, but honestly "nobody" actually uses more than the 10k. The thing is the unicode codepoints proliferate:
One for Japan, one for simplified and one for traditional. That's potentially 3x. Korea and Vietnam wants some too.
Then you have this thing called the Kangxi dictionary which was the most comprehensive Chinese character dictionary that includes all obscure characters people could find in the classical literature. Probably half of the characters in Kangxi have this description: "Variant of <more common character>". (I'm pretty sure a significant portion of these variants are just "typos" in old books)
If they were more "economical" in the usage of codepoints, i.e. don't create new code points for each country (just let them use the right fonts), don't include all the frivolous characters in Kangxi, etc... I'm pretty sure it's technically possible for CJK to use less than 10k characters in total. (which, if it did, will let unicode fit within 16 bits)
But this is a political nightmare and nobody wants this compromise. And the Unicode Consortium isn't going to prescribe some solution to some Asian countries on how their languages should be used.
So while the 16-bit limit was a bit tight even assuming optimal conditions, I really wouldn't fault the people who designed this limit to realize Asian cultural politics was so complicated. Heck, AFAIK, before CJK-unification, ALL East Asian codecs were 16-bit, and it was sufficient for each country respectively.