5 ms·

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the

I'm not a fan of "everything you know about X is wrong" articles. Very often they try to present some little tidbit of knowledge as a revelation and mislead the reader in the process. In this case, the tidbit is: "grapheme clusters exist and they are useful". The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units. UTF-32 code units are Unicode code points. This is a general Unicode concept that exists in all Unicode encodings. UTF-16 code units, on the other hand, are an implementation detail of UTF-16. It is wrong to present them as equally arbitrary concepts.

masklinn 3y ago
> UTF-32 code units are Unicode code points. They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. > It is wrong to present them as equally arbitrary concepts. Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.
- xeeeeeeeeeeenu 3y ago
  >They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings. BTW, Python strings can store surrogates. >Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium. If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all. Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.
chrismorgan 3y ago
> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value. If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.
- planede 3y ago
  WTF-8 is bridging the gap here. I don't know of a UTF-32 equivalent, maybe it's impossible.
  - kps 3y ago
    The UTF-32 equivalent is just the original UCS-4 — simply not enforcing any restrictions on the 32-bit value. Probably most code using UTF-32 does this, at least internally. (I can understand using high bits for metadata or non-Unicode points and have done so, but I don't see any reason for testing for surrogates outside of encoding/decoding UTF-16; they are indeed an abomination.)
- infinitedata 3y ago
  ‘Would that it were only so.’ What?
  - chrismorgan 3y ago
    “I wish it was that simple, that it was only an implementation detail.” I suppose “would that—” isn’t a particularly common form these days.
- cylemons 3y ago
  I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.
  - kps 3y ago
    > Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.
avgcorrection 3y ago
> I'm not a fan of "everything you know about X is wrong" articles. But it’s not. That style is about tone and the article doesn’t exude that kind of tone. Do you see the author scolding programmers for being ignorant Americans, for having unknown unknowns, or for not being “professionals”? Well, me neither.

masklinn 3y ago

> UTF-32 code units are Unicode code points. They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. > It is wrong to present them as equally arbitrary concepts. Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.

xeeeeeeeeeeenu 3y ago
>They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid. This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings. BTW, Python strings can store surrogates. >Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium. If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all. Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.

chrismorgan 3y ago

> UTF-16 code units, on the other hand, are an implementation detail of UTF-16. Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32. UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value. If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.

planede 3y ago
WTF-8 is bridging the gap here. I don't know of a UTF-32 equivalent, maybe it's impossible.
- kps 3y ago
  The UTF-32 equivalent is just the original UCS-4 — simply not enforcing any restrictions on the 32-bit value. Probably most code using UTF-32 does this, at least internally. (I can understand using high bits for metadata or non-Unicode points and have done so, but I don't see any reason for testing for surrogates outside of encoding/decoding UTF-16; they are indeed an abomination.)
infinitedata 3y ago
‘Would that it were only so.’ What?
- chrismorgan 3y ago
  “I wish it was that simple, that it was only an implementation detail.” I suppose “would that—” isn’t a particularly common form these days.
cylemons 3y ago
I don't see what's inherently wrong with UTF-16 surrogates. If I am not wrong, a given UTF-16 codeunit is unambigously either a complete code point, a first surrogate, or a second surrogate. Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.
- kps 3y ago
  > Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.

avgcorrection 3y ago

> I'm not a fan of "everything you know about X is wrong" articles. But it’s not. That style is about tone and the article doesn’t exude that kind of tone. Do you see the author scolding programmers for being ignorant Americans, for having unknown unknowns, or for not being “professionals”? Well, me neither.