10 ms·
One big obstacle to UTF8 uptake is in 2003 Unicode was clipped down to about a million codepoints, instead of the 2.1 billion originally specified by Pike and T
by vorg 12y ago
One big obstacle to UTF8 uptake is in 2003 Unicode was clipped down to about a million codepoints, instead of the 2.1 billion originally specified by Pike and Thompson. Only 137,000 of those million are available for Private Use, and if a user recognizes the ConScript allocations such as the Kinya Syllables in F0000..F0E6F, those 137,000 are cut down quickly. There's only room left for a few dozen more Hangul-style syllabaries!
- TazeTSchnitzel 12y agoYeah, I do wonder how long those 21 bits will last. They clipped it down to ensure it'd remain UTF-16 compatible, but I worry if we might need to extend UTF-8 in future to support more codepoints, and add some horrible new set of surrogate pairs, encoding using surrogate pairs, to UTF-16.
- vorg 12y agoAnyone can extend UTF-16 to the UTF-8 repertoire of codepoints by using the top 2 private use planes as high and low 2nd-tier surrogates respectively. Hopefully, by the time UTF-8 (and UTF-32) need to be extended, UTF-16 will have fallen into general disuse and almost no-one will need to use those 2 levels of surrogates. Java/Android and .NET strings aren't being broken anytime soon, however, so it seems to me people should start talking about the details of such a 2nd-tier surrogate system.