4 ms·

Search index use cases probably also benefit from normalizing inputs across encodings, that's no shining example of utf-8-onlyism. You can still best-effort sp

Search index use cases probably also benefit from normalizing inputs across encodings, that's no shining example of utf-8-onlyism. You can still best-effort split words! You can do a pretty good job splitting words without ensuring that the words decode in your preferred encoding.

lolc 6y ago
Here's the thing: I don't want to work in UTF8. I want to work in Unicode. Big difference. Because tracking the encoding of my strings would increase complexity. So at the earliest convenience, I validate my assumptions about encoding and let a lower layer handle it from then on. I understand you're arguing about some sort of equivalency between byte-arrays and Unicode strings. Sure there are half-baked ways to do word-splitting on a byte-array. But why do you consider that a viable option? Under what circumstances would you do that?
- naniwaduni 6y ago
  Every circumstance. Why do you consider it unviable? What problems do you think having a Unicode sequence solves?
  - lolc 6y ago
    Convince me. Here's a little library function that turns text into a set of words: def keywords(text): return set(filter(None, re.split("\W+", unicodedata.normalize("NFKC", input_str).lower()))) How would this look if strings were byte-arrays? How would `normalize()`, `lower()`, and `split()` know what encoding to use? The way I see it: If the encoding is implicit, you have global state. If it's explicit, you have to pass the encoding. Both is extra state to worry about. When the passed value is a Unicode string, this question doesn't come up.
    - naniwaduni 6y ago
      It looks pretty much the some, except that you assume the input is already in your library's canonical encoding (probably utf-8 nowadays). I realize this sounds like a total cop-out, but when the use-case is destructively best-effort tokenizing an input string using library functions, it doesn't really matter whether your internal encoding is utf-32 or utf-8. I mean, under the hood, normalize still has to map arbitrary-length sequences to arbitrary-length sequences even when working with utf-32 (see: unicodedata.normalize("NFKC", "a\u0301 ﬃ") == "\xe1 ffi"). So on the happy path, you don't see much of a difference. The main observable difference is that if you take input without decoding it explicitly, then the always-decode approach has already crashed long before reaching this function, while the assume-the-encoding approach probably spouts gibberish at this point. And sure, there are plenty of plausible scenarios where you'd rather get the crash than subtly broken behaviour. But ... I don't see this reasonably being one of them, considering that you're apparently okay with discarding all \W+.
      - suzuki 6y ago
        I agree with you. I wish Python 3 had strings as byte sequences mainly in UTF-8 as Python 2 had once and Go has now. Then things would be kept simple in Japan. Python 3 feels cumbersome. To handle a raw input as a string, you must decode it in some encoding first. It is a fragile process. It would be adequate to treat the input bytes transparently and put an optional stage to convert other encodings to UTF-8 if necessary.
      - lolc 6y ago
        I know this from PHP, where I have to be aware of the encoding the strings are in. I still don't see what the advantage should be of that.

lolc 6y ago

Here's the thing: I don't want to work in UTF8. I want to work in Unicode. Big difference. Because tracking the encoding of my strings would increase complexity. So at the earliest convenience, I validate my assumptions about encoding and let a lower layer handle it from then on. I understand you're arguing about some sort of equivalency between byte-arrays and Unicode strings. Sure there are half-baked ways to do word-splitting on a byte-array. But why do you consider that a viable option? Under what circumstances would you do that?

naniwaduni 6y ago
Every circumstance. Why do you consider it unviable? What problems do you think having a Unicode sequence solves?
- lolc 6y ago
  Convince me. Here's a little library function that turns text into a set of words: def keywords(text): return set(filter(None, re.split("\W+", unicodedata.normalize("NFKC", input_str).lower()))) How would this look if strings were byte-arrays? How would `normalize()`, `lower()`, and `split()` know what encoding to use? The way I see it: If the encoding is implicit, you have global state. If it's explicit, you have to pass the encoding. Both is extra state to worry about. When the passed value is a Unicode string, this question doesn't come up.
  - naniwaduni 6y ago
    It looks pretty much the some, except that you assume the input is already in your library's canonical encoding (probably utf-8 nowadays). I realize this sounds like a total cop-out, but when the use-case is destructively best-effort tokenizing an input string using library functions, it doesn't really matter whether your internal encoding is utf-32 or utf-8. I mean, under the hood, normalize still has to map arbitrary-length sequences to arbitrary-length sequences even when working with utf-32 (see: unicodedata.normalize("NFKC", "a\u0301 ﬃ") == "\xe1 ffi"). So on the happy path, you don't see much of a difference. The main observable difference is that if you take input without decoding it explicitly, then the always-decode approach has already crashed long before reaching this function, while the assume-the-encoding approach probably spouts gibberish at this point. And sure, there are plenty of plausible scenarios where you'd rather get the crash than subtly broken behaviour. But ... I don't see this reasonably being one of them, considering that you're apparently okay with discarding all \W+.
    - suzuki 6y ago
      I agree with you. I wish Python 3 had strings as byte sequences mainly in UTF-8 as Python 2 had once and Go has now. Then things would be kept simple in Japan. Python 3 feels cumbersome. To handle a raw input as a string, you must decode it in some encoding first. It is a fragile process. It would be adequate to treat the input bytes transparently and put an optional stage to convert other encodings to UTF-8 if necessary.
    - lolc 6y ago
      I know this from PHP, where I have to be aware of the encoding the strings are in. I still don't see what the advantage should be of that.