3 ms·
Unicode is just about the most technically successful standard I have ever seen, it's pretty amazing. The weird and complicated parts are all a result of the w
by jrochkind1 12y ago
Unicode is just about the most technically successful standard I have ever seen, it's pretty amazing.
The weird and complicated parts are all a result of the weirdness and complexity of the domain -- the universe of human written language. All the solutions are amazingly elegant for the domain they are in -- including solutions to legacy backwards compatibility where possible, which have made unicode as successful at catching on as it has been. The decisions on compromises between practial legacy compatibility and pure elegance were _just right_.
The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed, sadly now stuck in Java and making proper unicode support in Java way harder than it should be.
But in general, if only all our standards dealing with very complex problems could be as elegantly designed and executed as unicode.
- masklinn 12y ago> sadly now stuck in Java If only it were only java… It's also encoded in the JS spec, and throughout the Windows ecosystem… and anything spawned from the latter which is why bloody UEFI uses UCS-2.
- thristian 12y agoSomething that often gets missed out of the Unicode story was that originally there were two groups. The first was the Unicode consortium, who wanted to combine all the world's existing character encodings, and had picked 16-bit units as a comfortable representation, which would have been more than enough for their stated goal. When Unicode 1.0 came out, there were a bunch of people forming the ISO 10646 committee to produce a character encoding that would cover every human-written character ever, even the ones that weren't already part of an existing encoding, but 16 bits would definitely not be sufficient for that. On the other hand, creating two entirely separate standards wouldn't be a great idea either, so they joined forces and created Unicode 2.0 with astral planes and surrogate pairs and all that expansion business. The point is, we shouldn't blame the Unicode consortium for short-sightedness, we should blame scope-creep.
- jrochkind1 12y agoHm, except I think the scope broadening was a _great_ idea. A standard that only covered the human-written characters that had encodings created for them at that time would have been much less useful than what we've got. Thanks for your explanation of the history, that it wasn't exactly short-sightedness. Still, I wouldn't "blame" scope-creep -- maybe it's just another unusual example of standards-makers involved here managing to make the right decision at almost every point, even when it involved 'competition' between standards bodies. The UCS 2 leftover stuff is one of the biggest problems in practical unicode at the moment, alas.
- thristian 12y agoOh, certainly. In an ideal world we would have had the ISO10646 scope from the start, combined with maybe UTF-8. I do occaionally come across people "explaining" UTF-16 by saying the Unicode consortium couldn't count, which I feel is unfair even if it's a lie-to-children.
- vorg 12y ago> The only major mis-step was the "UCS-2" mistake, before they realized more bytes really were going to be needed Unicode 2.0 came out in 1996 increasing the codepoint count from about 65,000 to 2.1 billion, but in 2003 came a bigger misstep: those 2.1 billion were reduced back down to about 1 million. And of those million, only 137,000 are for private use. Neither short-sightedness nor scope-creep are to blame, IMO, but power plays and office politics.
- jrochkind1 12y agoI'm interested in reading more about this, can you give a cite to any URL talking more about this reduction in codepoint space? Haven't been succesful finding info on google.
- vorg 12y agohttp://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description describes Ken Thompson's original proposal for UTF-8 with codepoints up to U+7FFFFFFF, and how the upper limit was reduced to U+10FFFF in 2003. I've actually written a package for Go https://github.com/gavingroovygrover/utf88 https://github.com/gavingroovygrover/utf88 that uses the 2 private use planes to implement a surrogate system that restores that upper limit to UTF-8, which could easily also be applied to UTF-16 and UTF-32. It's all part of looking forward to the day UTF-8 as originally specified is reinstated.