5 ms·

> The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed Unicode 2.0 came out in 1996 increasing the c

by vorg 12y ago

> The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed Unicode 2.0 came out in 1996 increasing the codepoint count from about 65,000 to 2.1 billion, but in 2003 came a bigger misstep: those 2.1 billion were reduced back down to about 1 million. And of those million, only 137,000 are for private use. Neither short-sightedness nor scope-creep are to blame, IMO, but power plays and office politics.

jrochkind1 12y ago
I'm interested in reading more about this, can you give a cite to any URL talking more about this reduction in codepoint space? Haven't been succesful finding info on google.
- vorg 12y ago
  http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.
  - jrochkind1 12y ago
    Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?
    - vorg 12y ago
      Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!

by vorg 12y ago

jrochkind1 12y ago

I'm interested in reading more about this, can you give a cite to any URL talking more about this reduction in codepoint space? Haven't been succesful finding info on google.

vorg 12y ago
http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.
- jrochkind1 12y ago
  Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?
  - vorg 12y ago
    Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!