4 ms·

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with th

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value. If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.

planede 3y ago
WTF-8 is bridging the gap here. I don't know of a UTF-32 equivalent, maybe it's impossible.
- kps 3y ago
  The UTF-32 equivalent is just the original UCS-4 — simply not enforcing any restrictions on the 32-bit value. Probably most code using UTF-32 does this, at least internally. (I can understand using high bits for metadata or non-Unicode points and have done so, but I don't see any reason for testing for surrogates outside of encoding/decoding UTF-16; they are indeed an abomination.)
infinitedata 3y ago
‘Would that it were only so.’ What?
- chrismorgan 3y ago
  “I wish it was that simple, that it was only an implementation detail.” I suppose “would that—” isn’t a particularly common form these days.
cylemons 3y ago
I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.
- kps 3y ago
  > Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.
  - cylemons 3y ago
    Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.
    - kps 3y ago
      The main issue is that it adds validation code (if one is sticking to the standard) for things that don't care about UTF-16 at all. It does occupy 1/32 of the BMP, displaying a couple thousand potential actual characters (making them take an extra byte in UTF-8, and an extra two in UTF-16).
    - chrismorgan 3y ago
      As a hole, it would only be annoying and a performance penalty for validation. But by its very design, it will leak, and it does in such ways that it became the worst thing to ever happen to Unicode. I don’t know of a single language or library that uses UTF-16 for strings that validates strings: every last one actually uses sequences of UTF-16 code units, potentially ill-formed, and has APIs that guarantee this will leak to other systems. This has caused a lot of trouble for environments that then try to work with the vastly more sensible UTF-8 (the only credible alternative for interchange). Servo, for example, wanted to work in UTF-8, for massive memory savings and performance improvements, but the web has built on and depends on UTF-16 code unit semantics so much that they had to invent WTF-8, which is basically “UTF-8 but with that hole filled in” (well, actually it’s more complicated: half filled in, permitting only unpaired surrogates, so that you still have only one representation). So: the problem is that the Unicode standard was compromised for the sake of a buggy encoding (they should instead have written UCS-2 off as a failed experiment), and every implementation that uses that buggy encoding is itself buggy, and that bugginess has made it into many other standards (e.g. ECMAScript).
RcouF1uZ4gsC 3y ago
> Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, UTF-16 is a hack. Unicode originally thought 65,535 values should be enough to represent all human languages and so 16-bit fixed size characters would work. However, that proved incorrect. UTF-16 was a hack to try retrofit this onto systems that had already adopted this 16 bit character (Java, Windows NT, etc). An entertaining summary of the situation is here: https://www.theregister.com/2013/10/04/verity_stob_unicode/ https://www.theregister.com/2013/10/04/verity_stob_unicode/

planede 3y ago

WTF-8 is bridging the gap here. I don't know of a UTF-32 equivalent, maybe it's impossible.

kps 3y ago
The UTF-32 equivalent is just the original UCS-4 — simply not enforcing any restrictions on the 32-bit value. Probably most code using UTF-32 does this, at least internally. (I can understand using high bits for metadata or non-Unicode points and have done so, but I don't see any reason for testing for surrogates outside of encoding/decoding UTF-16; they are indeed an abomination.)

infinitedata 3y ago

‘Would that it were only so.’ What?

chrismorgan 3y ago
“I wish it was that simple, that it was only an implementation detail.” I suppose “would that—” isn’t a particularly common form these days.

cylemons 3y ago

I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.

kps 3y ago
> Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.
- cylemons 3y ago
  Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.
  - kps 3y ago
    The main issue is that it adds validation code (if one is sticking to the standard) for things that don't care about UTF-16 at all. It does occupy 1/32 of the BMP, displaying a couple thousand potential actual characters (making them take an extra byte in UTF-8, and an extra two in UTF-16).
  - chrismorgan 3y ago
    As a hole, it would only be annoying and a performance penalty for validation. But by its very design, it will leak, and it does in such ways that it became the worst thing to ever happen to Unicode. I don’t know of a single language or library that uses UTF-16 for strings that validates strings: every last one actually uses sequences of UTF-16 code units, potentially ill-formed, and has APIs that guarantee this will leak to other systems. This has caused a lot of trouble for environments that then try to work with the vastly more sensible UTF-8 (the only credible alternative for interchange). Servo, for example, wanted to work in UTF-8, for massive memory savings and performance improvements, but the web has built on and depends on UTF-16 code unit semantics so much that they had to invent WTF-8, which is basically “UTF-8 but with that hole filled in” (well, actually it’s more complicated: half filled in, permitting only unpaired surrogates, so that you still have only one representation). So: the problem is that the Unicode standard was compromised for the sake of a buggy encoding (they should instead have written UCS-2 off as a failed experiment), and every implementation that uses that buggy encoding is itself buggy, and that bugginess has made it into many other standards (e.g. ECMAScript).

RcouF1uZ4gsC 3y ago

> Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, UTF-16 is a hack. Unicode originally thought 65,535 values should be enough to represent all human languages and so 16-bit fixed size characters would work. However, that proved incorrect. UTF-16 was a hack to try retrofit this onto systems that had already adopted this 16 bit character (Java, Windows NT, etc). An entertaining summary of the situation is here: https://www.theregister.com/2013/10/04/verity_stob_unicode/ https://www.theregister.com/2013/10/04/verity_stob_unicode/