12 ms·
It doesn't have anything to do with UTF-8. It has to do with Unicode‡ growing from a 16-bit address space, into a ~20.1-bit address space; 17 16-bit "planes"‡‡
by LukeShu 10y ago
It doesn't have anything to do with UTF-8. It has to do with Unicode‡ growing from a 16-bit address space, into a ~20.1-bit address space; 17 16-bit "planes"‡‡ (log₂(17×2¹⁶)≈20.1bits).
This expansion happened sort-of gradually. Unicode 3.1 (2001) assigned the Unicode 3.0 character set to be "plane 0", and added 14 additional planes‡‡‡ (for a total address space of log₂(15×2¹⁶)≈19.9bits). I'm not sure exactly which version between 3.1 and 9.0 the additional two planes in.
That is to say, Unicode 3.1 solved the address-space problem.
So why does the article have a section "Why Unicode 3.1 Does Not Solve the Problem"? Well, there are two answers: the one that the article suggests, and the one giving the author the benefit of the doubt.
The way the section is written, it seems that the author thinks the the address space of 16 bits + 16 bits is 2×2¹⁶ instead of 2^(2×16); because they are in separate "16 bit blocks". They seem to think that maybe 1 bit was added, but think that that just grew the encoding size by 16 bits. It honestly read to me like it was written by a linguist who only had a rudimentary grasp of programming; I was surprised when it said the author was a programmer at the bottom.
Giving him the benefit of the doubt that the argument was only poorly expressed, not poorly thought: Unicode 3.1 expanded the address space by another 917,504 encodable characters; more than enough! However, it didn't actually define 900,000 more characters; it only defined 44,946 more characters (as the author noted). But it wasn't limited to 16 bits anymore; it had a full 19 (more actually; 19.9-ish!) to work with. The author even mentions that 18 bits would have been plenty. Well Unicode 3.1 got them! They just weren't allocated yet.
That said, Unicode 9.0 (2016) still only has about 128,000 characters defined in it. A far cry from the author's claim of 170,000 characters needed to satisfy asian languages.
‡: To say otherwise would be to confuse Unicode with its encodings, a mistake that only leads to confusion.
‡‡: The term "plane" comes from ISO/IEC standards dealing with character sets. Unicode 3.0 corresponded to the ISO/IEC "Basic Multilingual Plane"; so each 16-bit group of characters got its own cutesy name as a "Plane" to match.
‡‡‡: Why 15 planes, then 17? It has to do with what was encodable with existing encodings. That isn't to say that Unicode was limited by the encodings; but that it was informed by them. The growth beyond plane 0 meant that UCS-2 had to be phased out for UTF-16 (its successor), as UCS-2 couldn't encode anything but plane 0. However, seeing that UTF-16 could only encode 17×2¹⁶ characters, it made sense to limit the number of planes to 17 if there isn't a pressing need for more; as doing so would require obsoleting UTF-16. And given that the current address space is only about 12% utilized, there's no reason to mandate phasing out UTF-16 yet.
- eridius 10y agoI just want to say, I absolutely love that you used a whole bunch of unicode characters for math. I'm not sure I've ever actually seen someone use the superscript characters in a legitimate context (well, beyond my own very occasional usage). And you even threw in stuff like U+00D7 MULTIPLICATION SIGN (×)! As a side note, if anybody here is on OS X and wants to be able to type these characters, years ago I wrote a DefaultKeyBinding.dict file that adds bindings for these and a lot more (including the greek alphabet). You can see the file at https://gist.github.com/kballard/7584246fa5d5fcb684e996ff0957965a https://gist.github.com/kballard/7584246fa5d5fcb684e996ff095..., and if you want to use it, just put this in ~/Library/KeyBindings with the name DefaultKeyBinding.dict. Any running apps may have to be restarted to notice this.
- LukeShu 10y agoTo be honest, the only reason I used × was that it was easier than figuring out how to make HN not treat asterisk as italic!
- Pxtl 10y agoSpeaking of, I know it's way out of scope for Unicode, but am I the only one that is sad that Unicode only provides the symbols for math and not the layout? The poor standardization of mathematical formulae in various document editors is a constant frustration. You can happily move Word documents to Google Docs, but all the formulas vanish because there are a zillion standards for formulas and all of them are terrible.
- reacweb 10y agoI know many math teachers who are using latex because there was too many problems with word (bad editor and incompatibilities with previous versions with no upgrade path).
- witty_username 10y agoLatex works across many websites. There's an extension for Google Docs for Latex BTW.
- 10y ago