3 ms·

I've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a be

I've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a better choice than what Python 3 went with. Sure, you can argue that iterating over bytes is less "correct" than iterating over Unicode code-points. But iterating over code-points isn't very useful either -- maybe that code-point represents the letter B, but maybe it represents the third of seven characters and modifiers and zero-width joiners that make up one family emoji.

pansa2 6y ago
This is the worst part. 12 years to change the language’s Unicode model and, after all that, the new model turns out to be a bad design because in the meantime, everyone else has standardised on UTF-8.
dehrmann 6y ago
> I actually think it's a better choice What should `str.reverse()` do? It's very different for byte strings and UTF-8 strings. Or `str[4]`? `str[1:]` can corrupt a string for one definition of []. What about Windows users? There's a chance Powershell is using Windows-1252.
- csande17 6y ago
  That's the thing, though -- even in Python 3's model, neither of those operations produce the expected result in all cases. If I reverse "réal", I might get "laér" as expected, or I might get "láer" -- it depends on if é is represented as one single character or as two characters using "latin letter e" + "combining acute accent". The same thing applies to indexing and slicing. You might get an actual character, or you might get something invalid by slicing halfway into a multi-code-point sequence. So Python 3's model isn't actually that much better than Python 2's at handling these cases. But it comes at a significant cost, introducing "unicode vs bytes" confusion, and making it more difficult to write correct programs for platforms like Unix where filenames aren't necessarily valid Unicode.
  - takeda 6y ago
    Actually it is. What you suggesting would completely break those operations. Compared just not getting what you want. In any way you can tell python how to behave through normalization of the unicode.
    - csande17 6y ago
      > What you suggesting would completely break those operations. Compared just not getting what you want. I reversed a string several times in Python 2 and not once did my computer catch fire! I just got a string back that, in certain situations, didn't contain the characters I was expecting. And then my father had to have a talk with me about Unicode, and how computers sometimes use multiple bytes to represent a single character, and so if you rearrange the individual bytes you get different, usually more broken-looking characters. And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk. I just need to use the word "codepoint" instead of the word "byte". The situation is basically the same--if you know enough about Unicode to understand Python 3's behavior on certain non-English text, you know enough to understand Python 2's behavior.
      - takeda 6y ago
        > And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk What were you expecting to get when doing reverse on 🇨🇦? The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island. In python2 you get a total garbage. The flag representation is not part of unicode, unicode only specifies that you can encode country codes using ISO 3166-1 alpha-2 and it is up to the application how to present it.
        csande17 6y ago
        > What were you expecting to get when doing reverse on 🇨🇦? My future child sees only one character in that string, an emoji. (If you put the cursor to the right of 🇨🇦 and press backspace, the whole flag goes away, so that must mean it's only one character, right?) If they reverse that string, they'd expect to get the same result back. > The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island. Right, this is exactly what I'll have to explain! Just like how in UTF-8, "é" is encoded using two bytes: 0xC3 0xA9. So if you reverse that, you get bytes 0xA9 0xC3, which do not represent "é". It's the exact same problem, you've just moved up one level of abstraction. (The Python 3 version of the problem isn't unique to flags -- there are all kinds of modifiers and ZWJ sequences and combining characters that get messed up if you rearrange individual codepoints.) A few languages have tried to solve this problem in other ways, like how Swift considers each character of a string to be a "grapheme cluster", but that comes with a lot of its own drawbacks.

pansa2 6y ago

This is the worst part. 12 years to change the language’s Unicode model and, after all that, the new model turns out to be a bad design because in the meantime, everyone else has standardised on UTF-8.

dehrmann 6y ago

> I actually think it's a better choice What should `str.reverse()` do? It's very different for byte strings and UTF-8 strings. Or `str[4]`? `str[1:]` can corrupt a string for one definition of []. What about Windows users? There's a chance Powershell is using Windows-1252.

csande17 6y ago
That's the thing, though -- even in Python 3's model, neither of those operations produce the expected result in all cases. If I reverse "réal", I might get "laér" as expected, or I might get "láer" -- it depends on if é is represented as one single character or as two characters using "latin letter e" + "combining acute accent". The same thing applies to indexing and slicing. You might get an actual character, or you might get something invalid by slicing halfway into a multi-code-point sequence. So Python 3's model isn't actually that much better than Python 2's at handling these cases. But it comes at a significant cost, introducing "unicode vs bytes" confusion, and making it more difficult to write correct programs for platforms like Unix where filenames aren't necessarily valid Unicode.
- takeda 6y ago
  Actually it is. What you suggesting would completely break those operations. Compared just not getting what you want. In any way you can tell python how to behave through normalization of the unicode.
  - csande17 6y ago
    > What you suggesting would completely break those operations. Compared just not getting what you want. I reversed a string several times in Python 2 and not once did my computer catch fire! I just got a string back that, in certain situations, didn't contain the characters I was expecting. And then my father had to have a talk with me about Unicode, and how computers sometimes use multiple bytes to represent a single character, and so if you rearrange the individual bytes you get different, usually more broken-looking characters. And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk. I just need to use the word "codepoint" instead of the word "byte". The situation is basically the same--if you know enough about Unicode to understand Python 3's behavior on certain non-English text, you know enough to understand Python 2's behavior.
    - takeda 6y ago
      > And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk What were you expecting to get when doing reverse on 🇨🇦? The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island. In python2 you get a total garbage. The flag representation is not part of unicode, unicode only specifies that you can encode country codes using ISO 3166-1 alpha-2 and it is up to the application how to present it.
      - csande17 6y ago
        > What were you expecting to get when doing reverse on 🇨🇦? My future child sees only one character in that string, an emoji. (If you put the cursor to the right of 🇨🇦 and press backspace, the whole flag goes away, so that must mean it's only one character, right?) If they reverse that string, they'd expect to get the same result back. > The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island. Right, this is exactly what I'll have to explain! Just like how in UTF-8, "é" is encoded using two bytes: 0xC3 0xA9. So if you reverse that, you get bytes 0xA9 0xC3, which do not represent "é". It's the exact same problem, you've just moved up one level of abstraction. (The Python 3 version of the problem isn't unique to flags -- there are all kinds of modifiers and ZWJ sequences and combining characters that get messed up if you rearrange individual codepoints.) A few languages have tried to solve this problem in other ways, like how Swift considers each character of a string to be a "grapheme cluster", but that comes with a lot of its own drawbacks.