4 ms·
That was set up when Microsoft and Java had standardized on UTF-16. So this can represent the UTF-16 subset of Unicode, which is Unicode Plane 0, the Basic Mult
by Animats 5y ago
That was set up when Microsoft and Java had standardized on UTF-16. So this can represent the UTF-16 subset of Unicode, which is Unicode Plane 0, the Basic Multilingual Plane (BMP). The higher-numbered "astral planes" of UTF-8 were rarely used. All modern languages with a significant user base are covered in Plane 0. Plane 1, the Supplementary Multilingual Plane, with Cretan Linear B, Egyptian hieroglyphics, and such, was seldom needed. Few people had fonts for those, anyway.
Because of the way UTF-8 is encoded, it takes 3 bytes to represent the UTF-16 set. That's because it only takes one byte for ASCII characters. Hence, 3-byte MySQL data.
Emoji, though, were put in Unicode Plane 1. That's where mass demand for the astral planes came from. More four byte UTF-8 characters started showing up in data.
- loulouxiv 5y agoUTF-16 can represent the full range of Unicode codepoints by using couples of surrogates
- Animats 5y agoSort of. Applications using UTF-16 have to be aware of pairs at the application level. Many are not.
- int_19h 5y agoThis isn't a consequence of using UTF-16 as such - Java, .NET etc could totally have an API around UTF-16 strings that handles surrogate pairs. The problem, rather, is that those languages introduced a 16-bit type that they called "character", even though it wasn't even a Unicode codepoint. And then used that type throughout all string APIs, including strings themselves (indexing etc). In .NET land you're now supposed to use https://docs.microsoft.com/en-us/dotnet/api/system.text.rune https://docs.microsoft.com/en-us/dotnet/api/system.text.rune instead. It transparently handles surrogate pairs, so the app needn't be aware of anything - and yet the internal encoding is still UTF-16.
- mort96 5y agoYou're right. Replace UTF-16 with UCS-2 and the comment sounds at least slightly more correct.
- chrismorgan 5y agoTechnical amendment: UTF-16 can represent the full range of Unicode scalar values with surrogate pairs. Code points includes the surrogates U+D800–U+DFFF, scalar values don’t. Like all other Unicode encodings, UTF-16 cannot represent surrogates. That’s where the real problem lies: almost nothing that uses UTF-16 actually uses UTF-16, but rather potentially ill-formed UTF-16.
- tremon 5y agoA UTF-16 "subset" of Unicode doesn't exist. all UTF-* encodings are just that: encodings. They can all represent the entire Unicode character set. (Except for UTF-7, which never was an official standard). What you're referring to is UCS-2 [0], which is UTF-16 without support for the high surrogates that make up the rest of the Unicode character set. But to imply that UTF-16 is a subset of Unicode is just not true. [0] https://en.wikipedia.org/wiki/UCS-2 https://en.wikipedia.org/wiki/UCS-2
- Dylan16807 5y agoThat they can all represent all of unicode is true, but they're not "just" encodings either. The current code space of unicode is built around UTF-16. That's why there are 17 planes.
- chrismorgan 5y agoAnd why surrogates are a thing, and why code points and scalar values are different concepts. I hate UTF-16. It’s the worst thing that ever happened to Unicode. It ruined Unicode.
- est 5y agoCode points, scalar values, glype, surrogates, encodings Unicode should be called multicode really. If we want something "uni-" then all of them should better be unified to one encoding. Hopefully UTF-8 do the job.
- light_hue_1 5y agoThere's so much wrong with this answer. It completely confuses Unicode, with Unicode encodings, and with non-Unicode encodings. > So this can represent the UTF-16 subset of Unicode There is no UTF-16 subset of Unicode. UTF-16 has surrogate pairs, it can represent all of Unicode. You are talking about UCS-2, a format which has no surrogate pairs. > That was set up when Microsoft and Java had standardized on UTF-16 Java standardized on UCS-2 initially (which is what you're talking about). In the mid 2000s they moved to actual UTF-16. > The higher-numbered "astral planes" of UTF-8 "astral planes" are not a UTF-8 concept. They're a Unicode concept. A Unicode code point has 6 hex digits. The plane is just the first 2 of those 6 digits. So code points 0000– FFFF are in the BMP (Basic Multilingual Plane), etc. People nickname planes above the 00 plane, BMP, astral planes. > Plane 1 ... was seldom needed. Few people had fonts for those, anyway. So "rarely", that the majority of humanity writes in languages that aren't in BMP. CJK (Chinese, Japanese, Korean and sometimes Vietnamese) languages need characters outside of BMP. > Emoji, though, were put in Unicode Plane 1. That's where mass demand for the astral planes came from. More four byte UTF-8 characters started showing up in data. That's totally untrue. The demand always existed by the majority of humanity, you just never cared about this massive problem until it impacted you.
- Dylan16807 5y ago> That's totally untrue. The demand always existed by the majority of humanity, you just never cared about this massive problem until it impacted you. I think you swung so far in the other direction you also landed in untrue territory. Originally the intent was to encode all characters in "modern use" and 16 bits was probably enough to do that for the entire world. Also CJK is about one quarter of the world population.
- light_hue_1 5y ago"originally" is 20 years ago. People realized this mistake almost immediately before fixing it in Unicode 2.0. The mistake existed for something like 5 years last millennium. There's no excuse for not taking seriously a large part of the world's communication needs in 2022 as something esoteric that no one but people in 5000BC needed.