4 ms·

http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoin

by vorg 12y ago

http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.

jrochkind1 12y ago
Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?
- vorg 12y ago
  Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!

by vorg 12y ago

jrochkind1 12y ago

Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?

vorg 12y ago
Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!