4 ms·

> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. Wow. I wonder how you arrived at this

by lolc 6y ago

> There are no operations on sequences of Unicode code points that are more correct than an analogous operation on bytes. Wow. I wonder how you arrived at this point. You can't, for example, truncate a UTF-8 byte array without the risk of producing a broken string. But this is only the start. Here are two strings, six letters each, one in NFC, the other in NFD, and their byte-length in UTF-8: "Åström" is 8 bytes in UTF-8 "Åström" is 10 bytes in UTF-8 If your software tells the user that one is eight and the other is 10 letters long, it is not "less correct". It is incorrect. Further, if searching for "Åström" won't find "Åström", your software is less useful than it could be if it knew Unicode. (And it's sad how often software gets this wrong.)

naniwaduni 6y ago
You can't truncate a sequence of Unicode codepoints without the risk of producing a broken string, either. What do you get if you truncate "Åström" after the first "o"? What do you get if you truncate 🇨🇦 after the first codepoint? Normalization is not a real solution unless you restrict yourself to working with well-edited formal prose in common Western languages. This is not a claim made from ignorance.
- lolc 6y ago
  Sorry, we're mixing two layers. Of course, if I truncate a string, it may lose its meaning. And having accents fall off is problematic. But it's not the same as truncating a byte-array, because then an invalid sequence of bytes may result. Stop treating these cases as equivalent. They're not.
  - naniwaduni 6y ago
    They are equivalent. The only reason you find it problematic that a sequence of bytes is "invalid" (read: can't be decoded in your preferred encoding) is because you've manufactured the problem. In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer, and just being valid utf-8 isn't good enough for it either.
    - lolc 6y ago
      > In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer Ok that explains how we ended up here. I'm considering some other common uses! A search-index for example greatly profits from being able to normalize representations and split words.
      - naniwaduni 6y ago
        Search index use cases probably also benefit from normalizing inputs across encodings, that's no shining example of utf-8-onlyism. You can still best-effort split words! You can do a pretty good job splitting words without ensuring that the words decode in your preferred encoding.
        lolc 6y ago
        Here's the thing: I don't want to work in UTF8. I want to work in Unicode. Big difference. Because tracking the encoding of my strings would increase complexity. So at the earliest convenience, I validate my assumptions about encoding and let a lower layer handle it from then on. I understand you're arguing about some sort of equivalency between byte-arrays and Unicode strings. Sure there are half-baked ways to do word-splitting on a byte-array. But why do you consider that a viable option? Under what circumstances would you do that?
        naniwaduni 6y ago
        Every circumstance. Why do you consider it unviable? What problems do you think having a Unicode sequence solves?
        lolc 6y ago
        Convince me. Here's a little library function that turns text into a set of words: def keywords(text): return set(filter(None, re.split("\W+", unicodedata.normalize("NFKC", input_str).lower()))) How would this look if strings were byte-arrays? How would `normalize()`, `lower()`, and `split()` know what encoding to use? The way I see it: If the encoding is implicit, you have global state. If it's explicit, you have to pass the encoding. Both is extra state to worry about. When the passed value is a Unicode string, this question doesn't come up.
  - Jasper_ 6y ago
    So in one case, the text becomes corrupted and unreadable (i.e. loses its meaning), and in the other, it becomes corrupted and unreadable. What's the difference? Having "accents fall off" has gotten people murdered [0]. Accents aren't things peppered in for effect, they turn letters into different letters, spelling different words. Analogously, imagine that a bunch of software accidentally turned every "d" into a "c" because some committee halfway around the world decided "d" should be composed of the "c" and "|" glyphs. That's the kind of text corruption that regularly happens in other languages when dealing with text at the code point layer. [0] https://languagelog.ldc.upenn.edu/nll/?p=73 https://languagelog.ldc.upenn.edu/nll/?p=73 . Note that this is Turkish, which has the "dotted i" problem, meaning that this was more than likely a .toupper() gone wrong rather than a truncation issue.
    - lolc 6y ago
      The difference is that for truncating, I can work within Unicode to deal with the situation. I can accept the possibility of mutilated letters, I can convert to NFC, I can truncate on word-boundaries, I have choice. If I have an byte-array, I can do none of these things short of implementing a good chunk of Unicode. If I truncate, I risk ending up with an invalid UTF-8 string. End of story.
      - tsimionescu 6y ago
        And what is wrong with an invalid UTF-8 string? Why were you truncating the string in the first place? Basically, I believe the point here is that a Unicode aware truncation should be done in a Unicode aware truncate method. There is no good reason to parse a string as UTF-8 ahead of time - just keep it as a blob of bytes until you need to do some something "texty" with it. It is the truncate-at-word-boundaries() method that should interpret the bytes as UTF-8 and fail if they are not valid. Why parse it sooner?
      - Jasper_ 6y ago
        > If I have an byte-array, I can do none of these things short of implementing a good chunk of Unicode. If I truncate, I risk ending up with an invalid UTF-8 string. Yes, and? You can have an invalid sequence of Unicode code points too, such as an unpaired surrogate (something Python's text model actually abuses to store "invalid Unicode" in a special, non-standard way). If you truncate at the byte level, you are just truncating "between code points"; it's a closer granularity than at the code point layer, so you can also convert to NFC, truncate on word boundaries, etc. You just need to ignore the parts of the UTF-8 string that are invalid; which isn't difficult, because UTF-8 is self-synchronizing.
tsimionescu 6y ago
> If your software tells the user that one is eight and the other is 10 letters long, it is not "less correct". It is incorrect. In fact, if the software tells you that either of the strings is either 8 or 10 letters wrong, then either way the software is incorrect - those are both obviously 6 letter strings. Now, does UTF8 help you discover they are 6 letter strings better than other representations? There are certainly text-oriented libraries that can do that, but not those that simply count the UTF8 code points - they must have an understanding of all of Unicode. Even worse, the question "how many letters does this string have" is not generally meaningful - there are plenty of perfectly valid unicode strings for which this question doesn't have a meaningful answer. However, the question "how many unicode code points does this string have" is almost never of interest. You either care about some notion of unique glyphs, or you care about byte lengths.
- lolc 6y ago
  > then either way the software is incorrect - those are both obviously 6 letter strings. What I wanted to get at is that in Unicode, I have a chance to count letters to some useful degree. Why should I consider starting at byte-arrays? > there are plenty of perfectly valid Unicode strings for which this question doesn't have a meaningful answer. I don't get it. Why does the existence of degenerate cases invalidate the usefulness of a Unicode lib? If I want to know how many letters are in a string, I can probably get a useful answer from a Unicode lib. Not for all edge-cases, but I can decide on the trade-offs. If I have a byte-array, I start at a lower level.
  - a1369209993 6y ago
    > What I wanted to get at is that in Unicode, I have a chance to count letters to some useful degree. You do not. You merely happen to get the right answer by coincidence in some cases, same as bytes-that-probably-are(n't)-ASCII. To throw your own words back at you: "Åström" is 6 code points in Unicode "Åström" is 8 code points in Unicode If your software tells the user that one is six and the other is 8 letters long, it is not "less correct". It is incorrect. Further, if searching for "Åström" won't find "Åström", your software is less useful than it could be if it knew text. (And it's sad how often software gets this wrong.)

by lolc 6y ago

naniwaduni 6y ago

You can't truncate a sequence of Unicode codepoints without the risk of producing a broken string, either. What do you get if you truncate "Åström" after the first "o"? What do you get if you truncate 🇨🇦 after the first codepoint? Normalization is not a real solution unless you restrict yourself to working with well-edited formal prose in common Western languages. This is not a claim made from ignorance.

lolc 6y ago
Sorry, we're mixing two layers. Of course, if I truncate a string, it may lose its meaning. And having accents fall off is problematic. But it's not the same as truncating a byte-array, because then an invalid sequence of bytes may result. Stop treating these cases as equivalent. They're not.
- naniwaduni 6y ago
  They are equivalent. The only reason you find it problematic that a sequence of bytes is "invalid" (read: can't be decoded in your preferred encoding) is because you've manufactured the problem. In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer, and just being valid utf-8 isn't good enough for it either.
  - lolc 6y ago
    > In the end, the only layer at which it really matters whether your byte sequence can be decoded is the font renderer Ok that explains how we ended up here. I'm considering some other common uses! A search-index for example greatly profits from being able to normalize representations and split words.
    - naniwaduni 6y ago
      Search index use cases probably also benefit from normalizing inputs across encodings, that's no shining example of utf-8-onlyism. You can still best-effort split words! You can do a pretty good job splitting words without ensuring that the words decode in your preferred encoding.
      - lolc 6y ago
        Here's the thing: I don't want to work in UTF8. I want to work in Unicode. Big difference. Because tracking the encoding of my strings would increase complexity. So at the earliest convenience, I validate my assumptions about encoding and let a lower layer handle it from then on. I understand you're arguing about some sort of equivalency between byte-arrays and Unicode strings. Sure there are half-baked ways to do word-splitting on a byte-array. But why do you consider that a viable option? Under what circumstances would you do that?
        naniwaduni 6y ago
        Every circumstance. Why do you consider it unviable? What problems do you think having a Unicode sequence solves?
        lolc 6y ago
        Convince me. Here's a little library function that turns text into a set of words: def keywords(text): return set(filter(None, re.split("\W+", unicodedata.normalize("NFKC", input_str).lower()))) How would this look if strings were byte-arrays? How would `normalize()`, `lower()`, and `split()` know what encoding to use? The way I see it: If the encoding is implicit, you have global state. If it's explicit, you have to pass the encoding. Both is extra state to worry about. When the passed value is a Unicode string, this question doesn't come up.
- Jasper_ 6y ago
  So in one case, the text becomes corrupted and unreadable (i.e. loses its meaning), and in the other, it becomes corrupted and unreadable. What's the difference? Having "accents fall off" has gotten people murdered [0]. Accents aren't things peppered in for effect, they turn letters into different letters, spelling different words. Analogously, imagine that a bunch of software accidentally turned every "d" into a "c" because some committee halfway around the world decided "d" should be composed of the "c" and "|" glyphs. That's the kind of text corruption that regularly happens in other languages when dealing with text at the code point layer. [0] https://languagelog.ldc.upenn.edu/nll/?p=73 https://languagelog.ldc.upenn.edu/nll/?p=73 . Note that this is Turkish, which has the "dotted i" problem, meaning that this was more than likely a .toupper() gone wrong rather than a truncation issue.
  - lolc 6y ago
    The difference is that for truncating, I can work within Unicode to deal with the situation. I can accept the possibility of mutilated letters, I can convert to NFC, I can truncate on word-boundaries, I have choice. If I have an byte-array, I can do none of these things short of implementing a good chunk of Unicode. If I truncate, I risk ending up with an invalid UTF-8 string. End of story.
    - tsimionescu 6y ago
      And what is wrong with an invalid UTF-8 string? Why were you truncating the string in the first place? Basically, I believe the point here is that a Unicode aware truncation should be done in a Unicode aware truncate method. There is no good reason to parse a string as UTF-8 ahead of time - just keep it as a blob of bytes until you need to do some something "texty" with it. It is the truncate-at-word-boundaries() method that should interpret the bytes as UTF-8 and fail if they are not valid. Why parse it sooner?
    - Jasper_ 6y ago
      > If I have an byte-array, I can do none of these things short of implementing a good chunk of Unicode. If I truncate, I risk ending up with an invalid UTF-8 string. Yes, and? You can have an invalid sequence of Unicode code points too, such as an unpaired surrogate (something Python's text model actually abuses to store "invalid Unicode" in a special, non-standard way). If you truncate at the byte level, you are just truncating "between code points"; it's a closer granularity than at the code point layer, so you can also convert to NFC, truncate on word boundaries, etc. You just need to ignore the parts of the UTF-8 string that are invalid; which isn't difficult, because UTF-8 is self-synchronizing.

tsimionescu 6y ago

> If your software tells the user that one is eight and the other is 10 letters long, it is not "less correct". It is incorrect. In fact, if the software tells you that either of the strings is either 8 or 10 letters wrong, then either way the software is incorrect - those are both obviously 6 letter strings. Now, does UTF8 help you discover they are 6 letter strings better than other representations? There are certainly text-oriented libraries that can do that, but not those that simply count the UTF8 code points - they must have an understanding of all of Unicode. Even worse, the question "how many letters does this string have" is not generally meaningful - there are plenty of perfectly valid unicode strings for which this question doesn't have a meaningful answer. However, the question "how many unicode code points does this string have" is almost never of interest. You either care about some notion of unique glyphs, or you care about byte lengths.

lolc 6y ago
> then either way the software is incorrect - those are both obviously 6 letter strings. What I wanted to get at is that in Unicode, I have a chance to count letters to some useful degree. Why should I consider starting at byte-arrays? > there are plenty of perfectly valid Unicode strings for which this question doesn't have a meaningful answer. I don't get it. Why does the existence of degenerate cases invalidate the usefulness of a Unicode lib? If I want to know how many letters are in a string, I can probably get a useful answer from a Unicode lib. Not for all edge-cases, but I can decide on the trade-offs. If I have a byte-array, I start at a lower level.
- a1369209993 6y ago
  > What I wanted to get at is that in Unicode, I have a chance to count letters to some useful degree. You do not. You merely happen to get the right answer by coincidence in some cases, same as bytes-that-probably-are(n't)-ASCII. To throw your own words back at you: "Åström" is 6 code points in Unicode "Åström" is 8 code points in Unicode If your software tells the user that one is six and the other is 8 letters long, it is not "less correct". It is incorrect. Further, if searching for "Åström" won't find "Åström", your software is less useful than it could be if it knew text. (And it's sad how often software gets this wrong.)