4 ms·

Something that often gets missed out of the Unicode story was that originally there were two groups. The first was the Unicode consortium, who wanted to combine

Something that often gets missed out of the Unicode story was that originally there were two groups. The first was the Unicode consortium, who wanted to combine all the world's existing character encodings, and had picked 16-bit units as a comfortable representation, which would have been more than enough for their stated goal. When Unicode 1.0 came out, there were a bunch of people forming the ISO 10646 committee to produce a character encoding that would cover every human-written character ever, even the ones that weren't already part of an existing encoding, but 16 bits would definitely not be sufficient for that. On the other hand, creating two entirely separate standards wouldn't be a great idea either, so they joined forces and created Unicode 2.0 with astral planes and surrogate pairs and all that expansion business. The point is, we shouldn't blame the Unicode consortium for short-sightedness, we should blame scope-creep.

jrochkind1 12y ago
Hm, except I think the scope broadening was a _great_ idea. A standard that only covered the human-written characters that had encodings created for them at that time would have been much less useful than what we've got. Thanks for your explanation of the history, that it wasn't exactly short-sightedness. Still, I wouldn't "blame" scope-creep -- maybe it's just another unusual example of standards-makers involved here managing to make the right decision at almost every point, even when it involved 'competition' between standards bodies. The UCS 2 leftover stuff is one of the biggest problems in practical unicode at the moment, alas.
- thristian 12y ago
  Oh, certainly. In an ideal world we would have had the ISO10646 scope from the start, combined with maybe UTF-8. I do occaionally come across people "explaining" UTF-16 by saying the Unicode consortium couldn't count, which I feel is unfair even if it's a lie-to-children.
vorg 12y ago
> The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed Unicode 2.0 came out in 1996 increasing the codepoint count from about 65,000 to 2.1 billion, but in 2003 came a bigger misstep: those 2.1 billion were reduced back down to about 1 million. And of those million, only 137,000 are for private use. Neither short-sightedness nor scope-creep are to blame, IMO, but power plays and office politics.
- jrochkind1 12y ago
  I'm interested in reading more about this, can you give a cite to any URL talking more about this reduction in codepoint space? Haven't been succesful finding info on google.
  - vorg 12y ago
    http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.
    - jrochkind1 12y ago
      Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?
      - vorg 12y ago
        Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!

jrochkind1 12y ago

Hm, except I think the scope broadening was a _great_ idea. A standard that only covered the human-written characters that had encodings created for them at that time would have been much less useful than what we've got. Thanks for your explanation of the history, that it wasn't exactly short-sightedness. Still, I wouldn't "blame" scope-creep -- maybe it's just another unusual example of standards-makers involved here managing to make the right decision at almost every point, even when it involved 'competition' between standards bodies. The UCS 2 leftover stuff is one of the biggest problems in practical unicode at the moment, alas.

thristian 12y ago
Oh, certainly. In an ideal world we would have had the ISO10646 scope from the start, combined with maybe UTF-8. I do occaionally come across people "explaining" UTF-16 by saying the Unicode consortium couldn't count, which I feel is unfair even if it's a lie-to-children.

vorg 12y ago

> The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed Unicode 2.0 came out in 1996 increasing the codepoint count from about 65,000 to 2.1 billion, but in 2003 came a bigger misstep: those 2.1 billion were reduced back down to about 1 million. And of those million, only 137,000 are for private use. Neither short-sightedness nor scope-creep are to blame, IMO, but power plays and office politics.

jrochkind1 12y ago
I'm interested in reading more about this, can you give a cite to any URL talking more about this reduction in codepoint space? Haven't been succesful finding info on google.
- vorg 12y ago
  http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.
  - jrochkind1 12y ago
    Thanks. Do you have a need today for more than 1 million codepoints for some reason? Or you are just predicting that we collectively will eventually? Or you just miss the theoretical elegance? Or what?
    - vorg 12y ago
      Only 137,000 of the 1 million Unicode points are categorized as Private Use, and the library I wrote addresses the need for more private use codepoints. A suggested path for people needing private use codepoints is: * Use the 6400 codepoints in the BMP * Begin using plane 0xF beginning at U+F0000 * After using the 32,000 codepoints in the bottom half of 0xF up to U+F7FFF, if you're certain you won't need more than 137,000 codepoints, continue using plane 0xF from U+F8000 onwards * Otherwise, skip the top half of plane 0xF, and go straight to plane 0x10 beginning at U+100000, and start using the surrogates when you get to U+110000 As for needing more than a million codepoints, I've been looking at whether it's possible to represent CJK Unihan characters compositionally in the 6-byte codepoints (U+4000000 to U+7FFFFFFF) in the same way Korean Hangul has been represented since Unicode 2.0, thus creating many more Unihan characters that don't presently exist. Perhaps there'll be a demand for them one day!