7 ms·

Indeed, representing text as a sequence of Unicode code points is fundamentally wrong. There are no operations on sequences of Unicode code points that are mor

Indeed, representing text as a sequence of Unicode code points is fundamentally wrong. There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. (Everyone's favourite example, length, actually becomes less correct—a byte array's length at least corresponds to the amount of space one might have to allocate for it in a particular encoding. A length in codepoints is absolutely meaningless both technically and linguistically. And this is, for what little it's worth, close to the only operation you can do on a string without imposing additional restrictions about its context.)

orf 6y ago
> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. That’s ridiculous. Uppercasing/lowercasing, slicing, “startswith”, splitting, etc etc. Your statement is correct if you only care about ascii.
- naniwaduni 6y ago
  None of those operations are correct on Unicode codepoints. Your statement is only just barely tenable if you only care about well-edited and normalized formal prose in common Western languages. Even then, upper/lower-casing is iffy.
- hvdijk 6y ago
  Uppercasing/lowercasing cannot be done on Unicode code points, because that fails to handle things like ﬁ -> FI where the uppercased version does not consist of the same number of Unicode code points. Slicing and splitting cannot be done on Unicode code points because it may separate a character from a subsequent combining character. "startswith" cannot be done on Unicode code points because some distinct code points need to be treated as equivalent. These are pretty much the same problems you also have when you perform those same operations on bytes. You might encounter those problems in fewer cases when you perform operations on code points rather than on bytes, but you won't have solved the problems entirely.
  - naniwaduni 6y ago
    Worse, you'll have pushed the problematic cases out of the realm of obviously wrong and not sensible to do, into subtly wrong and will break down the line in ways that will be hard to recognize and debug.
lolc 6y ago
> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. Wow. I wonder how you arrived at this point. You can't, for example, truncate a UTF-8 byte array without the risk of producing a broken string. But this is only the start. Here are two strings, six letters each, one in NFC, the other in NFD, and their byte-length in UTF-8: "Åström" is 8 bytes in UTF-8 "Åström" is 10 bytes in UTF-8 If your software tells the user that one is eight and the other is 10 letters long, it is not "less correct". It is incorrect. Further, if searching for "Åström" won't find "Åström", your software is less useful than it could be if it knew Unicode. (And it's sad how often software gets this wrong.)
- naniwaduni 6y ago
  You can't truncate a sequence of Unicode codepoints without the risk of producing a broken string, either. What do you get if you truncate "Åström" after the first "o"? What do you get if you truncate 🇨🇦 after the first codepoint? Normalization is not a real solution unless you restrict yourself to working with well-edited formal prose in common Western languages. This is not a claim made from ignorance.
  - lolc 6y ago
    Sorry, we're mixing two layers. Of course, if I truncate a string, it may lose its meaning. And having accents fall off is problematic. But it's not the same as truncating a byte-array, because then an invalid sequence of bytes may result. Stop treating these cases as equivalent. They're not.
    - naniwaduni 6y ago
      They are equivalent. The only reason you find it problematic that a sequence of bytes is "invalid" (read: can't be decoded in your preferred encoding) is because you've manufactured the problem. In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer, and just being valid utf-8 isn't good enough for it either.

orf 6y ago

> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. That’s ridiculous. Uppercasing/lowercasing, slicing, “startswith”, splitting, etc etc. Your statement is correct if you only care about ascii.

naniwaduni 6y ago
None of those operations are correct on Unicode codepoints. Your statement is only just barely tenable if you only care about well-edited and normalized formal prose in common Western languages. Even then, upper/lower-casing is iffy.
hvdijk 6y ago
Uppercasing/lowercasing cannot be done on Unicode code points, because that fails to handle things like ﬁ -> FI where the uppercased version does not consist of the same number of Unicode code points. Slicing and splitting cannot be done on Unicode code points because it may separate a character from a subsequent combining character. "startswith" cannot be done on Unicode code points because some distinct code points need to be treated as equivalent. These are pretty much the same problems you also have when you perform those same operations on bytes. You might encounter those problems in fewer cases when you perform operations on code points rather than on bytes, but you won't have solved the problems entirely.
- naniwaduni 6y ago
  Worse, you'll have pushed the problematic cases out of the realm of obviously wrong and not sensible to do, into subtly wrong and will break down the line in ways that will be hard to recognize and debug.

lolc 6y ago

> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. Wow. I wonder how you arrived at this point. You can't, for example, truncate a UTF-8 byte array without the risk of producing a broken string. But this is only the start. Here are two strings, six letters each, one in NFC, the other in NFD, and their byte-length in UTF-8: "Åström" is 8 bytes in UTF-8 "Åström" is 10 bytes in UTF-8 If your software tells the user that one is eight and the other is 10 letters long, it is not "less correct". It is incorrect. Further, if searching for "Åström" won't find "Åström", your software is less useful than it could be if it knew Unicode. (And it's sad how often software gets this wrong.)

naniwaduni 6y ago
You can't truncate a sequence of Unicode codepoints without the risk of producing a broken string, either. What do you get if you truncate "Åström" after the first "o"? What do you get if you truncate 🇨🇦 after the first codepoint? Normalization is not a real solution unless you restrict yourself to working with well-edited formal prose in common Western languages. This is not a claim made from ignorance.
- lolc 6y ago
  Sorry, we're mixing two layers. Of course, if I truncate a string, it may lose its meaning. And having accents fall off is problematic. But it's not the same as truncating a byte-array, because then an invalid sequence of bytes may result. Stop treating these cases as equivalent. They're not.
  - naniwaduni 6y ago
    They are equivalent. The only reason you find it problematic that a sequence of bytes is "invalid" (read: can't be decoded in your preferred encoding) is because you've manufactured the problem. In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer, and just being valid utf-8 isn't good enough for it either.