5 ms·

Python 3 is worth it just for Unicode handling. As someone using a language that is uses non-ASCII characters, thank you if you're using Python 3 and made your

Python 3 is worth it just for Unicode handling. As someone using a language that is uses non-ASCII characters, thank you if you're using Python 3 and made your software more internationally compatible.

saurik 8y ago
...or you could just use the unicode type in Python 2; like: all of my Python 2 software handled languages in other character sets correctly (and yes: this was stress tested often as I have had many many users from China and the Middle East). There was absolutely nothing wrong with Python 2 and Unicode, and if anything Python 3 just made a bunch of stuff impossible to do correctly (in particular encodings for filenames) until years later (I think Python 3.4 or something is when they finally got their Unicode shit together; meanwhile, Python 2 worked great).
- guitarbill 8y ago
  Good for you, but that isn't how it usually worked. Python 3 has made my life easier, but if you really think "There was absolutey nothing wrong with Python 2 and Unicode", I doubt I'll be able to convince you. So I'll simply say again: thank you Python 3 maintainers and developers, it's a huge help to me.
  - bayesian_horse 8y ago
    I was always able to work out my unicode trouble in Python 2 ... eventually. It's better now, less space to screw up.
- Latty 8y ago
  Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we want. People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.
  - josefx 8y ago
    The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.
  - slavik81 8y ago
    I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise: Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout. My bet is very few people would write that program correctly.
    - avian 8y ago
      Ok, I'll bite. This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial: https://gist.github.com/avian2/e3968bf933212ee561410b15495e626f https://gist.github.com/avian2/e3968bf933212ee561410b15495e6... What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
      - slavik81 8y ago
        I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe C:\Users\local\tmp>python --version Python 3.7.1 C:\Users\local\tmp>python foo.py Diet Parliament Traceback (most recent call last): File "foo.py", line 7, in <module> text = f.read() File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char acter maps to <undefined> Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?
- hnauz 8y ago
  You have to be kidding. I don't even need to elaborate, people who actually used Python 2 know what I'm talking about.
  - mehrdadn 8y ago
    You're suggesting the parent is lying about having used Python 2 and/or having felt that way?
    - hnauz 8y ago
      It's completely impossible to have used Python 2 and not having felt miserable under a pile of UnicodeDecodeErrors.
      - mehrdadn 8y ago
        Let's not claim people are liars just because their experiences are different.
  - deleted 8y ago
    [deleted]
- mrweasel 8y ago
  We where able to delete so much code when we switched to Python 3. When you integrate with especially Windows system converting unknown text was tricky at best. It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works. Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.
  - the_mitsuhiko 8y ago
    I think this is largely a second system result. If you want correct unicode handling it's not harder in 2.x than in 3.x. The main reason unicode support was more complex in 2.x for the most part is that many APIs supported working with Bytes and on 3.x that burden is pushed to the API consumer. > Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works. You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-surrogate-handling http://click.palletsprojects.com/en/7.x/python3/#python-3-su...
    - mrweasel 8y ago
      >on 3.x that burden is pushed to the API consumer So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.
    - wbond 8y ago
      At first blush you would think so, but I know for a fact that Unicode on Windows is not possible to properly handle below Python 3.4. Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6. In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug. I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.

saurik 8y ago

...or you could just use the unicode type in Python 2; like: all of my Python 2 software handled languages in other character sets correctly (and yes: this was stress tested often as I have had many many users from China and the Middle East). There was absolutely nothing wrong with Python 2 and Unicode, and if anything Python 3 just made a bunch of stuff impossible to do correctly (in particular encodings for filenames) until years later (I think Python 3.4 or something is when they finally got their Unicode shit together; meanwhile, Python 2 worked great).

guitarbill 8y ago
Good for you, but that isn't how it usually worked. Python 3 has made my life easier, but if you really think "There was absolutey nothing wrong with Python 2 and Unicode", I doubt I'll be able to convince you. So I'll simply say again: thank you Python 3 maintainers and developers, it's a huge help to me.
- bayesian_horse 8y ago
  I was always able to work out my unicode trouble in Python 2 ... eventually. It's better now, less space to screw up.
Latty 8y ago
Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we want. People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.
- josefx 8y ago
  The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.
- slavik81 8y ago
  I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise: Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout. My bet is very few people would write that program correctly.
  - avian 8y ago
    Ok, I'll bite. This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial: https://gist.github.com/avian2/e3968bf933212ee561410b15495e626f https://gist.github.com/avian2/e3968bf933212ee561410b15495e6... What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
    - slavik81 8y ago
      I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe C:\Users\local\tmp>python --version Python 3.7.1 C:\Users\local\tmp>python foo.py Diet Parliament Traceback (most recent call last): File "foo.py", line 7, in <module> text = f.read() File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char acter maps to <undefined> Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?
hnauz 8y ago
You have to be kidding. I don't even need to elaborate, people who actually used Python 2 know what I'm talking about.
- mehrdadn 8y ago
  You're suggesting the parent is lying about having used Python 2 and/or having felt that way?
  - hnauz 8y ago
    It's completely impossible to have used Python 2 and not having felt miserable under a pile of UnicodeDecodeErrors.
    - mehrdadn 8y ago
      Let's not claim people are liars just because their experiences are different.
- deleted 8y ago
  [deleted]
mrweasel 8y ago
We where able to delete so much code when we switched to Python 3. When you integrate with especially Windows system converting unknown text was tricky at best. It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works. Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.
- the_mitsuhiko 8y ago
  I think this is largely a second system result. If you want correct unicode handling it's not harder in 2.x than in 3.x. The main reason unicode support was more complex in 2.x for the most part is that many APIs supported working with Bytes and on 3.x that burden is pushed to the API consumer. > Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works. You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-surrogate-handling http://click.palletsprojects.com/en/7.x/python3/#python-3-su...
  - mrweasel 8y ago
    >on 3.x that burden is pushed to the API consumer So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.
  - wbond 8y ago
    At first blush you would think so, but I know for a fact that Unicode on Windows is not possible to properly handle below Python 3.4. Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6. In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug. I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.