27 ms·

UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. > The current permutation of Unicode gives a theoret

UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. > The current permutation of Unicode gives a theoretical maximum of approximately 65,000 characters No, UTF-16 enables a maximum of 2,097,152 characters (2^21). > Clearly, 32 bits (4 octets) would have been more than adequate if they were a contiguous block. Indeed, "18 bits wide" (262,144 variations) would be enough to address the world’s characters if a contiguous block. UTF-16 provides 21 bits, 3 more than the author wants. Except they're not “in a contiguous block”: > But two separate 16 bit blocks do not solve the problem at all. The author doesn't explain why having multiple blocks is a problem. This works just fine, and has enabled Unicode to accommodate the hundreds of thousands of extra characters the author said it ought to. Though maybe there's a hint in this later comment: > One can easily formulate new standards using 4 octet blocks (ad infinitum) – but piggybacking them on top of Unicode 3.1 simply exacerbates the complexity of font mapping, as Unicode 3.1 has increased the complexity of UCS-2. They would have preferred if backwards-compatibility had been broken and everyone switched to a new format that's like UTF-32/UCS-4, but not called Unicode, I guess?

masklinn 10y ago
> UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. Hell UTF-8 was devised in mid-1992 and presented at USENIX in 1993. > No, UTF-16 enables a maximum of 2,097,152 characters (2^21). And until 2003 (and RFC 3629 which neutered it to match UTF-16) UTF-8 enabled 2,147,483,648 characters (2^31).
dsp1234 10y ago
UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. UTF-16 was only made into a standard in 2000[0] The first non-BMP characters weren't introduced until 2001[1] From a historical perspective, less than a month after Unicode 3.1 was released officially, it was not clear that Unicode (the standard, not the technical implementation of the encoding) would actually work. [0] - https://tools.ietf.org/html/rfc2781 https://tools.ietf.org/html/rfc2781 [1] - http://www.unicode.org/reports/tr27/tr27-4.html http://www.unicode.org/reports/tr27/tr27-4.html
- TazeTSchnitzel 10y ago
  > UTF-16 was only made into a standard in 2000 It only got its own IETF RFC in 2000, but the surrogate pair mechanism was first specified in 1996's Unicode 2.0.
klodolph 10y ago
The maximum number of characters in Unicode is 1111998 (this excludes non-characters and surrogates). UTF-16 can encode 1112064 code points (this includes non-characters). This is quite a bit smaller than the 21 bits quoted: you really get the 16-bit BMP plus the 20-bit astral planes, minus the surrogates (2048 of them). The "non-characters" number 66.
- TazeTSchnitzel 10y ago
  Thank you for that correction, I suspected 2^21 might not be quite correct.
- vorg 10y ago
  > The maximum number of characters ... (this excludes non-characters and surrogates) You should exclude the surrogates but include the non-characters when tallying the characters. According to Unicode's 2013 Corridendum 9, "Noncharacters in the Unicode Standard are intended for internal use and have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange nor do they cause ill-formed Unicode text". See http://www.unicode.org/versions/corrigendum9.html http://www.unicode.org/versions/corrigendum9.html
  - klodolph 10y ago
    I can see why this might be confusing—I chose the word "character" carefully. The annex you cited clarifies that noncharacters are code points that can be used in interchange. However, this does not mean that a noncharacter is a character. The number of code points available for use as characters must, by definition, exclude the noncharacters. So, to expand on the original comment, Unicode defines 1114112 code points, of which 1112064 can be used in interchange, and 1111998 can be defined as characters. UTF-16 can only represent the 1112064 that are valid for interchange, and the 66 noncharacters should generally be avoided (especially U+FFFE).
jordigh 10y ago
Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understand most of how UTF-8 works, I still have to read and re-read how codepoints and planes and encodings and decodings work together. It's a pretty complicated beast that could very easily be misunderstood when it was less popular than it is now. It's still widely misunderstood today.
- TazeTSchnitzel 10y ago
  I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.
  - 0x0 10y ago
    It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.

masklinn 10y ago

> UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. Hell UTF-8 was devised in mid-1992 and presented at USENIX in 1993. > No, UTF-16 enables a maximum of 2,097,152 characters (2^21). And until 2003 (and RFC 3629 which neutered it to match UTF-16) UTF-8 enabled 2,147,483,648 characters (2^31).

dsp1234 10y ago

UTF-16, and non-BMP planes, were devised in 1996. The author seems to have been 5 years late to the party. UTF-16 was only made into a standard in 2000[0] The first non-BMP characters weren't introduced until 2001[1] From a historical perspective, less than a month after Unicode 3.1 was released officially, it was not clear that Unicode (the standard, not the technical implementation of the encoding) would actually work. [0] - https://tools.ietf.org/html/rfc2781 https://tools.ietf.org/html/rfc2781 [1] - http://www.unicode.org/reports/tr27/tr27-4.html http://www.unicode.org/reports/tr27/tr27-4.html

TazeTSchnitzel 10y ago
> UTF-16 was only made into a standard in 2000 It only got its own IETF RFC in 2000, but the surrogate pair mechanism was first specified in 1996's Unicode 2.0.

klodolph 10y ago

The maximum number of characters in Unicode is 1111998 (this excludes non-characters and surrogates). UTF-16 can encode 1112064 code points (this includes non-characters). This is quite a bit smaller than the 21 bits quoted: you really get the 16-bit BMP plus the 20-bit astral planes, minus the surrogates (2048 of them). The "non-characters" number 66.

TazeTSchnitzel 10y ago
Thank you for that correction, I suspected 2^21 might not be quite correct.
vorg 10y ago
> The maximum number of characters ... (this excludes non-characters and surrogates) You should exclude the surrogates but include the non-characters when tallying the characters. According to Unicode's 2013 Corridendum 9, "Noncharacters in the Unicode Standard are intended for internal use and have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange nor do they cause ill-formed Unicode text". See http://www.unicode.org/versions/corrigendum9.html http://www.unicode.org/versions/corrigendum9.html
- klodolph 10y ago
  I can see why this might be confusing—I chose the word "character" carefully. The annex you cited clarifies that noncharacters are code points that can be used in interchange. However, this does not mean that a noncharacter is a character. The number of code points available for use as characters must, by definition, exclude the noncharacters. So, to expand on the original comment, Unicode defines 1114112 code points, of which 1112064 can be used in interchange, and 1111998 can be defined as characters. UTF-16 can only represent the 1112064 that are valid for interchange, and the 66 noncharacters should generally be avoided (especially U+FFFE).

jordigh 10y ago

Maybe the errors in the article are more a statement of how complicated and improperly communicated Unicode was... and mostly still is! While I think I understand most of how UTF-8 works, I still have to read and re-read how codepoints and planes and encodings and decodings work together. It's a pretty complicated beast that could very easily be misunderstood when it was less popular than it is now. It's still widely misunderstood today.

TazeTSchnitzel 10y ago
I'm not sure that's fair, Unicode's encodings are pretty straightforward, particularly compared to some other character sets. Most of the complexity comes above the encoding level.
- 0x0 10y ago
  It also doesn't help that the classic LAMP stack has very confusing defaults and badly named functions: * PHP has functions named "utf8_encode()" and "utf8_decode()", when they should have been called "latin1_to_utf8_transcode()" and "utf8_to_latin1_transcode()" * MySQL for the longest time used latin1 as a default character set, then introduced an insufficient character set called "utf8" which only allows up to 3 bytes, not enough for all possible utf8 encoded codepoints, then introduced a proper implementation called "utf8mb4". * mysql connectors and client libraries often default their "client character set" setting to latin1, causing "silent" transcodes against the "server character set" and table column character sets. Also, because their "latin1" charset is more or less a binary-safe encoding, it is very easy to get double latin1-to-utf8 transcoded data in the database, something that often goes by unnoticed as long as data is merely received-inserted-selected-output to a browser, until you start to work on substrings or case insensitive searches etc. * In Java, there are tons of methods that work on the boundary between bytes and characters that allows not specifying an encoding, which then silenty falls back to an almost randomly set system encoding * Many languages such as Java, JavaScript and the unicode variants of win32 were unfortunately designed at a time where unicode characters could fit into 16bits, with the devastating result that the data type "char" is too small to store a single unicode character. It also plays hell on substring indexing. In short, the APIs are stacked against the beginning programmer and doesn't make it obvious that when you go from working with abstract "characters" to byte streams, there is ALWAYS an encoding involved.