4 ms·

Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we wan

by Latty 8y ago

Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we want. People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.

josefx 8y ago
The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.
slavik81 8y ago
I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise: Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout. My bet is very few people would write that program correctly.
- avian 8y ago
  Ok, I'll bite. This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial: https://gist.github.com/avian2/e3968bf933212ee561410b15495e626f https://gist.github.com/avian2/e3968bf933212ee561410b15495e6... What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
  - slavik81 8y ago
    I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe C:\Users\local\tmp>python --version Python 3.7.1 C:\Users\local\tmp>python foo.py Diet Parliament Traceback (most recent call last): File "foo.py", line 7, in <module> text = f.read() File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char acter maps to <undefined> Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?
    - ubernostrum 8y ago
      The way I usually put it is that Python 3 shifted its priorities. Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things. If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like. Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts. But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).

by Latty 8y ago

josefx 8y ago

The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.

slavik81 8y ago

I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise: Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout. My bet is very few people would write that program correctly.

avian 8y ago
Ok, I'll bite. This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial: https://gist.github.com/avian2/e3968bf933212ee561410b15495e626f https://gist.github.com/avian2/e3968bf933212ee561410b15495e6... What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
- slavik81 8y ago
  I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe C:\Users\local\tmp>python --version Python 3.7.1 C:\Users\local\tmp>python foo.py Diet Parliament Traceback (most recent call last): File "foo.py", line 7, in <module> text = f.read() File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char acter maps to <undefined> Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?
  - ubernostrum 8y ago
    The way I usually put it is that Python 3 shifted its priorities. Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things. If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like. Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts. But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).