4 ms·
Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters https://www.unicode.org/faq
by layer8 5y ago
Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters https://www.unicode.org/faq/private_use.html#noncharacters
Because of modifier characters, control characters like for bidi, stuff like soft-hyphens and ligatures, locale-dependent semantics (upper/lowercase, collation etc.), the general discordance between glyphs and characters, and so on and so forth, Unicode is so complex, and in general always requires careful processing of code point (or code unit) sequences, that honestly the surrogate encoding doesn’t make that much of a difference. It’s just an additional wrinkle in a sea of wrinkles.
- Aardwolf 5y agoI still find the surrogates different. Bidi, private use, ligatures, ... are script or locale related. Unicode uses numeric values from 0 to 1112063. You can invent all kinds of methods to encode numbers from 0 to 1112063 (variable length, fixed length, decimal, hexadecimal, anything else). But most ways I can think of to encode these numbers, including variable length ones that would use 8 bit or 16 bit primitives, don't require me to actually reserve some of those to-be-encoded numbers themselves for a special meaning. Yet for UTF-16 they managed to do it. Imagine that all other encodings out there would also want to reserve some Unicode values for their own purpose!
- layer8 5y agoYou always have to work with sequences of code units anyway (instead of just single code points), so the individual reasons for that doesn’t make much of a difference. It seems your rejection is more on aesthetic than on practical grounds.
- chrismorgan 5y agoAs far as valid Unicode is concerned, you care about the distinction between code points and scalar values, and surrogates are the only difference. Noncharacters can be represented in any Unicode encoding. Surrogates code points cannot, but can be found in unvalidated UTF-16 (which is most UTF-16). Dealing with Unicode text semantically requires that you be aware of a great many factors such as those you name, but you don’t need to be aware of those for just storing and transferring Unicode text. But with surrogates, UTF-16 managed to break it for everyone: any part of the system that uses unvalidated UTF-16 can introduce errors that you must care about. Hence surrogates are a special kind of atrocity.
- account42 5y ago> unvalidated UTF-16 (which is most UTF-16). I prefer the name WTF-16 [0] [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
- layer8 5y agoHow is that different from unvalidated UTF-8? Both are variable-length encodings that require decoding and have failure cases.
- chrismorgan 5y agoTo begin with, the difference is one of convention. UTF-16 is almost never checked for well-formedness: not a single one of the major users of UTF-16 that I am familiar with validates (Windows wide API functions, JavaScript, Qt); I don’t know of any system that exposes UTF-16 strings as some kind of main string type that does check well-formedness. UTF-8, on the other hand, is normally validated. Not always, to be sure, but normally. (Probably the most significant “largely UTF-8 but not necessarily” thing is *nix paths and other related syscalls—which is not dissimilar from the Windows situation, except that one uses 16-bit code units and the other 8-. This would seem to weaken my point, but my point is again strengthened by the number of languages that can’t equanimously interact with non-UTF-8 paths on Linux, which is not such a problem for non-UTF-16 paths on Windows, for all that the latter are very much rarer.) But beyond that, the difference lies in Unicode having been compromised for the sake of one particular encoding, and now there are all kinds of places that have to deal with potentially ill-formed UTF-16, more that potentially ill-formed UTF-8, I would say. Look at what Python 3 did: its string type is not a Unicode string, but rather a sequence of Unicode code points. That is, it can encode surrogates. Blech.
- layer8 5y agoI’ll grant that the surrogate code points prevent round-tripping of arbitrary code point sequences (though BOM characters arguably do as well). > Unicode having been compromised for the sake of one particular encoding They really didn’t have much choice though, because UCS-2 existed and was in major use at the time Unicode was extended beyond 16 bits (which wasn’t part of the original Unicode design), and too many code points were already used up to turn it into an UTF-8-like encoding. Not supporting the additional planes with the 16-bit encoding would have been a nonstarter. Switching to 32-bit in-memory representation (UCS-4) even less so, given the resource constraints of the time. The reason platforms like Windows and Java didn’t add strict input validation was due to backwards compatibility, because they previously were UCS-2 where arbitrary sequences are valid. One can lament the chronology that lead to those technical choices, but they each were reasonable and appropriate choices under the constraints present at each time. More importantly, surrogate characters rarely create any actual problems in practice. I haven’t encountered any in the past 20+ years (other than missing support for characters beyond U+FFFF). Other parts of Unicode cause significantly more complexity.