2 ms·
> UTF-8 should only concern itself with encoding arbitrary 24-bit integers I mostly agree, but situations in which there are two or more ways to encode the sam
by kazinator 4y ago
> UTF-8 should only concern itself with encoding arbitrary 24-bit integers
I mostly agree, but situations in which there are two or more ways to encode the same integer should use a canonical representation; so that is to say, it is well and good that overlong forms are banned in UTF-8.
So there is the rub; those pesky surrogate pairs create the same problem: a pair of code is used to produce a single character. When you're converting to UTF-8, a valid surrogate pair which encodes a character should just produce that character, and not the individual codes. The separate characters are kind of like an overlong form.
If a surrogate occurs by itself, though, and not part of a pair, that means the original string is not valid Unicode. That is the case here with the "\udead" string.
The choices are: encode that U+DEAD code point into UTF-8, thereby punting the problem to whoever/whatever decodes it, or else flag it here.
- account42 4y agoOverlong forms would break a lot of useful properties of UTF-8 that rely on the encoding being unique, most important of all perhaps being compatibility with ASCII tools. Surrogates however have only one encoding (except for overlong forms) so this is not the same problem at all. You could say there is a disambiguity with decoding matching surrogate pairs in WTF-16 but this is a WTF-16 issue and would not exist if "UTF-16" would not have been built by requiring to carve out unicode code points. > The choices are: encode that U+DEAD code point into UTF-8, thereby punting the problem to whoever/whatever decodes it, or else flag it here. And encoding it in WTF-8 is the better choice because it retains more information, allowing you to perfectly round trip back to original WTF-16 string. But either way, handling unmatched surrogate pairs is not really a concern of the UTF-8/WTF-8 encoding (they are just code points as far as it is concerned) but of the UTF-16/WTF-16 decoding.