3 ms·

As far as valid Unicode is concerned, you care about the distinction between code points and scalar values, and surrogates are the only difference. Noncharacte

As far as valid Unicode is concerned, you care about the distinction between code points and scalar values, and surrogates are the only difference. Noncharacters can be represented in any Unicode encoding. Surrogates code points cannot, but can be found in unvalidated UTF-16 (which is most UTF-16). Dealing with Unicode text semantically requires that you be aware of a great many factors such as those you name, but you don’t need to be aware of those for just storing and transferring Unicode text. But with surrogates, UTF-16 managed to break it for everyone: any part of the system that uses unvalidated UTF-16 can introduce errors that you must care about. Hence surrogates are a special kind of atrocity.

account42 5y ago
> unvalidated UTF-16 (which is most UTF-16). I prefer the name WTF-16 [0] [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
layer8 5y ago
How is that different from unvalidated UTF-8? Both are variable-length encodings that require decoding and have failure cases.
- chrismorgan 5y ago
  To begin with, the difference is one of convention. UTF-16 is almost never checked for well-formedness: not a single one of the major users of UTF-16 that I am familiar with validates (Windows wide API functions, JavaScript, Qt); I don’t know of any system that exposes UTF-16 strings as some kind of main string type that does check well-formedness. UTF-8, on the other hand, is normally validated. Not always, to be sure, but normally. (Probably the most significant “largely UTF-8 but not necessarily” thing is *nix paths and other related syscalls—which is not dissimilar from the Windows situation, except that one uses 16-bit code units and the other 8-. This would seem to weaken my point, but my point is again strengthened by the number of languages that can’t equanimously interact with non-UTF-8 paths on Linux, which is not such a problem for non-UTF-16 paths on Windows, for all that the latter are very much rarer.) But beyond that, the difference lies in Unicode having been compromised for the sake of one particular encoding, and now there are all kinds of places that have to deal with potentially ill-formed UTF-16, more that potentially ill-formed UTF-8, I would say. Look at what Python 3 did: its string type is not a Unicode string, but rather a sequence of Unicode code points. That is, it can encode surrogates. Blech.
  - layer8 5y ago
    I’ll grant that the surrogate code points prevent round-tripping of arbitrary code point sequences (though BOM characters arguably do as well). > Unicode having been compromised for the sake of one particular encoding They really didn’t have much choice though, because UCS-2 existed and was in major use at the time Unicode was extended beyond 16 bits (which wasn’t part of the original Unicode design), and too many code points were already used up to turn it into an UTF-8-like encoding. Not supporting the additional planes with the 16-bit encoding would have been a nonstarter. Switching to 32-bit in-memory representation (UCS-4) even less so, given the resource constraints of the time. The reason platforms like Windows and Java didn’t add strict input validation was due to backwards compatibility, because they previously were UCS-2 where arbitrary sequences are valid. One can lament the chronology that lead to those technical choices, but they each were reasonable and appropriate choices under the constraints present at each time. More importantly, surrogate characters rarely create any actual problems in practice. I haven’t encountered any in the past 20+ years (other than missing support for characters beyond U+FFFF). Other parts of Unicode cause significantly more complexity.
    - chrismorgan 5y ago
      Missing support for characters beyond U+FFFF is the main problem caused by surrogates (their existence, even if indirect)—it normally comes of some kind of UCS-2/UTF-16 confusion. It’s not fair to disqualify them. The only (class of) case that I’m aware of for a long time where it’s not linked to that is with MySQL’s idiotic utf8 → utf8mb3 type. You may not have encountered such bugs, but I’m very familiar with surrogate-related bugs, because I use a Compose key extensively. I haven’t been using Windows for the last year, but from time to time I would definitely encounter bugs that are certainly due to surrogates. On the web, I found bugs a few times, all but once in Rust WebAssembly things, such as https://github.com/Pauan/rust-dominator/issues/10 https://github.com/Pauan/rust-dominator/issues/10. And even now I’m back on Linux, I know of one almost certainly surrogate-related bug: I can’t type astral plane characters in Zoom at all; pretty sure I had this problem back on Windows, too. Copy and paste, sure, but type, no, they become REPLACEMENT CHARACTER. The history is unfortunate but I strongly refute that they had not much choice. UCS-2 should have been abandoned as a failed experiment. Certainly there had been significant investment into it in the last few years, but with the benefit of hindsight, switching to UTF-8 (which was invented before they decided on surrogates) would have made everyone’s life much easier, especially given its ASCII-compatibility, which would have allowed the Windows API to retire the misbegotten A/W split and return to sanity a few decades early. Ah, BOMs. Haven’t seen one in years. Good riddance.

account42 5y ago

> unvalidated UTF-16 (which is most UTF-16). I prefer the name WTF-16 [0] [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

layer8 5y ago

How is that different from unvalidated UTF-8? Both are variable-length encodings that require decoding and have failure cases.

chrismorgan 5y ago
To begin with, the difference is one of convention. UTF-16 is almost never checked for well-formedness: not a single one of the major users of UTF-16 that I am familiar with validates (Windows wide API functions, JavaScript, Qt); I don’t know of any system that exposes UTF-16 strings as some kind of main string type that does check well-formedness. UTF-8, on the other hand, is normally validated. Not always, to be sure, but normally. (Probably the most significant “largely UTF-8 but not necessarily” thing is *nix paths and other related syscalls—which is not dissimilar from the Windows situation, except that one uses 16-bit code units and the other 8-. This would seem to weaken my point, but my point is again strengthened by the number of languages that can’t equanimously interact with non-UTF-8 paths on Linux, which is not such a problem for non-UTF-16 paths on Windows, for all that the latter are very much rarer.) But beyond that, the difference lies in Unicode having been compromised for the sake of one particular encoding, and now there are all kinds of places that have to deal with potentially ill-formed UTF-16, more that potentially ill-formed UTF-8, I would say. Look at what Python 3 did: its string type is not a Unicode string, but rather a sequence of Unicode code points. That is, it can encode surrogates. Blech.
- layer8 5y ago
  I’ll grant that the surrogate code points prevent round-tripping of arbitrary code point sequences (though BOM characters arguably do as well). > Unicode having been compromised for the sake of one particular encoding They really didn’t have much choice though, because UCS-2 existed and was in major use at the time Unicode was extended beyond 16 bits (which wasn’t part of the original Unicode design), and too many code points were already used up to turn it into an UTF-8-like encoding. Not supporting the additional planes with the 16-bit encoding would have been a nonstarter. Switching to 32-bit in-memory representation (UCS-4) even less so, given the resource constraints of the time. The reason platforms like Windows and Java didn’t add strict input validation was due to backwards compatibility, because they previously were UCS-2 where arbitrary sequences are valid. One can lament the chronology that lead to those technical choices, but they each were reasonable and appropriate choices under the constraints present at each time. More importantly, surrogate characters rarely create any actual problems in practice. I haven’t encountered any in the past 20+ years (other than missing support for characters beyond U+FFFF). Other parts of Unicode cause significantly more complexity.
  - chrismorgan 5y ago
    Missing support for characters beyond U+FFFF is the main problem caused by surrogates (their existence, even if indirect)—it normally comes of some kind of UCS-2/UTF-16 confusion. It’s not fair to disqualify them. The only (class of) case that I’m aware of for a long time where it’s not linked to that is with MySQL’s idiotic utf8 → utf8mb3 type. You may not have encountered such bugs, but I’m very familiar with surrogate-related bugs, because I use a Compose key extensively. I haven’t been using Windows for the last year, but from time to time I would definitely encounter bugs that are certainly due to surrogates. On the web, I found bugs a few times, all but once in Rust WebAssembly things, such as https://github.com/Pauan/rust-dominator/issues/10 https://github.com/Pauan/rust-dominator/issues/10. And even now I’m back on Linux, I know of one almost certainly surrogate-related bug: I can’t type astral plane characters in Zoom at all; pretty sure I had this problem back on Windows, too. Copy and paste, sure, but type, no, they become REPLACEMENT CHARACTER. The history is unfortunate but I strongly refute that they had not much choice. UCS-2 should have been abandoned as a failed experiment. Certainly there had been significant investment into it in the last few years, but with the benefit of hindsight, switching to UTF-8 (which was invented before they decided on surrogates) would have made everyone’s life much easier, especially given its ASCII-compatibility, which would have allowed the Windows API to retire the misbegotten A/W split and return to sanity a few decades early. Ah, BOMs. Haven’t seen one in years. Good riddance.