3 ms·
I’ll grant that the surrogate code points prevent round-tripping of arbitrary code point sequences (though BOM characters arguably do as well). > Unicode havin
by layer8 5y ago
I’ll grant that the surrogate code points prevent round-tripping of arbitrary code point sequences (though BOM characters arguably do as well).
> Unicode having been compromised for the sake of one particular encoding
They really didn’t have much choice though, because UCS-2 existed and was in major use at the time Unicode was extended beyond 16 bits (which wasn’t part of the original Unicode design), and too many code points were already used up to turn it into an UTF-8-like encoding. Not supporting the additional planes with the 16-bit encoding would have been a nonstarter. Switching to 32-bit in-memory representation (UCS-4) even less so, given the resource constraints of the time. The reason platforms like Windows and Java didn’t add strict input validation was due to backwards compatibility, because they previously were UCS-2 where arbitrary sequences are valid.
One can lament the chronology that lead to those technical choices, but they each were reasonable and appropriate choices under the constraints present at each time.
More importantly, surrogate characters rarely create any actual problems in practice. I haven’t encountered any in the past 20+ years (other than missing support for characters beyond U+FFFF). Other parts of Unicode cause significantly more complexity.
- chrismorgan 5y agoMissing support for characters beyond U+FFFF is the main problem caused by surrogates (their existence, even if indirect)—it normally comes of some kind of UCS-2/UTF-16 confusion. It’s not fair to disqualify them. The only (class of) case that I’m aware of for a long time where it’s not linked to that is with MySQL’s idiotic utf8 → utf8mb3 type. You may not have encountered such bugs, but I’m very familiar with surrogate-related bugs, because I use a Compose key extensively. I haven’t been using Windows for the last year, but from time to time I would definitely encounter bugs that are certainly due to surrogates. On the web, I found bugs a few times, all but once in Rust WebAssembly things, such as https://github.com/Pauan/rust-dominator/issues/10 https://github.com/Pauan/rust-dominator/issues/10. And even now I’m back on Linux, I know of one almost certainly surrogate-related bug: I can’t type astral plane characters in Zoom at all; pretty sure I had this problem back on Windows, too. Copy and paste, sure, but type, no, they become REPLACEMENT CHARACTER. The history is unfortunate but I strongly refute that they had not much choice. UCS-2 should have been abandoned as a failed experiment. Certainly there had been significant investment into it in the last few years, but with the benefit of hindsight, switching to UTF-8 (which was invented before they decided on surrogates) would have made everyone’s life much easier, especially given its ASCII-compatibility, which would have allowed the Windows API to retire the misbegotten A/W split and return to sanity a few decades early. Ah, BOMs. Haven’t seen one in years. Good riddance.