5 ms·

> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs. There are hundreds if no

> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs. There are hundreds if not thousands of pages of unicode documentation. If anything the problem is that there is way too much for any normal person to actually understand it. For most people though, knowing when to normalize strings to NFC versus NFKC is all they need to not unknowingly accumulate tons of technical debt.

epistasis 6y ago
You are right, there's tons of Unicode documentation out there. The problem is how to use encodings in Python. What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? For example, I coded along in Python 3, not changing and IO handling for the longest time, but when I Tried to use the gzip module I suddenly had to learn a whole bunch more knowledge about Unicode. I looked around for about 10 minutes for a some sort of best practices guide, but ij lieu of that just mucked with encode/decode functions until it worked. I don't know if I did conversions at the right spot, or where the guidelines are for changing between bytes and characters, but it didn't matter for my data at the time and o had a deadline so I couldn't really justify spending another hour at the time to figure it out. Maybe I'm just trying to justify my behavior to myself, but I think this sort of problem solving is the norm. I tend to try to adopt best practices and educate myself more than the other programmers I have worked with, if not more, and am the one introducing flake8 etc. to code reviews, so I'm not the worst sort of programmer, though I certainly wouldn't claim to be top-tier either.
- dehrmann 6y ago
  > What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? This is coming from someone who's written a lot of Java (Java's had the byte/char distinction since forever); the way Python 2 pretended byte strings are strings really was broken and not what you want. What the changes in Python 3 mean are that you have to actually think about if something is bytes or a string, and you have to be intentional in choosing encodings.
  - csande17 6y ago
    I've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a better choice than what Python 3 went with. Sure, you can argue that iterating over bytes is less "correct" than iterating over Unicode code-points. But iterating over code-points isn't very useful either -- maybe that code-point represents the letter B, but maybe it represents the third of seven characters and modifiers and zero-width joiners that make up one family emoji.
    - pansa2 6y ago
      This is the worst part. 12 years to change the language’s Unicode model and, after all that, the new model turns out to be a bad design because in the meantime, everyone else has standardised on UTF-8.
    - dehrmann 6y ago
      > I actually think it's a better choice What should `str.reverse()` do? It's very different for byte strings and UTF-8 strings. Or `str[4]`? `str[1:]` can corrupt a string for one definition of []. What about Windows users? There's a chance Powershell is using Windows-1252.
      - csande17 6y ago
        That's the thing, though -- even in Python 3's model, neither of those operations produce the expected result in all cases. If I reverse "réal", I might get "laér" as expected, or I might get "láer" -- it depends on if é is represented as one single character or as two characters using "latin letter e" + "combining acute accent". The same thing applies to indexing and slicing. You might get an actual character, or you might get something invalid by slicing halfway into a multi-code-point sequence. So Python 3's model isn't actually that much better than Python 2's at handling these cases. But it comes at a significant cost, introducing "unicode vs bytes" confusion, and making it more difficult to write correct programs for platforms like Unix where filenames aren't necessarily valid Unicode.

epistasis 6y ago

You are right, there's tons of Unicode documentation out there. The problem is how to use encodings in Python. What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? For example, I coded along in Python 3, not changing and IO handling for the longest time, but when I Tried to use the gzip module I suddenly had to learn a whole bunch more knowledge about Unicode. I looked around for about 10 minutes for a some sort of best practices guide, but ij lieu of that just mucked with encode/decode functions until it worked. I don't know if I did conversions at the right spot, or where the guidelines are for changing between bytes and characters, but it didn't matter for my data at the time and o had a deadline so I couldn't really justify spending another hour at the time to figure it out. Maybe I'm just trying to justify my behavior to myself, but I think this sort of problem solving is the norm. I tend to try to adopt best practices and educate myself more than the other programmers I have worked with, if not more, and am the one introducing flake8 etc. to code reviews, so I'm not the worst sort of programmer, though I certainly wouldn't claim to be top-tier either.

dehrmann 6y ago
> What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? This is coming from someone who's written a lot of Java (Java's had the byte/char distinction since forever); the way Python 2 pretended byte strings are strings really was broken and not what you want. What the changes in Python 3 mean are that you have to actually think about if something is bytes or a string, and you have to be intentional in choosing encodings.
- csande17 6y ago
  I've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a better choice than what Python 3 went with. Sure, you can argue that iterating over bytes is less "correct" than iterating over Unicode code-points. But iterating over code-points isn't very useful either -- maybe that code-point represents the letter B, but maybe it represents the third of seven characters and modifiers and zero-width joiners that make up one family emoji.
  - pansa2 6y ago
    This is the worst part. 12 years to change the language’s Unicode model and, after all that, the new model turns out to be a bad design because in the meantime, everyone else has standardised on UTF-8.
  - dehrmann 6y ago
    > I actually think it's a better choice What should `str.reverse()` do? It's very different for byte strings and UTF-8 strings. Or `str[4]`? `str[1:]` can corrupt a string for one definition of []. What about Windows users? There's a chance Powershell is using Windows-1252.
    - csande17 6y ago
      That's the thing, though -- even in Python 3's model, neither of those operations produce the expected result in all cases. If I reverse "réal", I might get "laér" as expected, or I might get "láer" -- it depends on if é is represented as one single character or as two characters using "latin letter e" + "combining acute accent". The same thing applies to indexing and slicing. You might get an actual character, or you might get something invalid by slicing halfway into a multi-code-point sequence. So Python 3's model isn't actually that much better than Python 2's at handling these cases. But it comes at a significant cost, introducing "unicode vs bytes" confusion, and making it more difficult to write correct programs for platforms like Unix where filenames aren't necessarily valid Unicode.