5 ms·

Interestingly, I had exactly the opposite situation: I always had codec problems when using Python 2, and they were all solved with Python 3. Perhaps the main

Interestingly, I had exactly the opposite situation: I always had codec problems when using Python 2, and they were all solved with Python 3. Perhaps the main difference is the native language of the developer and his/her clients/customers: If you and your customers are native english speakers, encoding problems were dramatically less probable before because all letters in use were ASCII anyway - and the UTF-8 positions of ASCII characters stayed the same (e.g. 'A' is still 0x41)- so buggy encoding didn't tend to break things for them. So even older Windows versions encoding filenames in latin1 etc. were never a problem to begin with - something that was always a hazzle for me (natively German speaking, with umlauts etc.). Python 3 forces UTF-8 everywhere, which solved oodles of quirks for everyone but natively english speaking people. And thankfully encoding bugs now tend to break things mostly on startup, not just if a customer uploads a funky file from an old OS, two months into deployment.

slavik81 7y ago
> Python 3 forces UTF-8 everywhere If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only. Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file. It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.
- bildung 7y ago
  Yes, I omitted that my Python 2 (mostly Django or bottle) ran on Linux with UTF-8. I suspect your piping case could be due to either the OS not using UTF-8 as locale or the application behind the pipe ignoring the OS' locale. These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.
  - slavik81 7y ago
    Just to elaborate, on a fresh Windows 7 install with en_CA locale, this was the behaviour: python3 -c 'print("é")' é python3 -c 'print("é")' > out.txt (UnicodeEncodeError) 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128) I felt that was unacceptable behaviour for my program, but it was beyond my ability to fix. Though, I think this has been addressed in more recent Python releases.
    - mehrdadn 7y ago
      I thought you might have fun with this -- try running my example here: https://news.ycombinator.com/item?id=21842471 https://news.ycombinator.com/item?id=21842471
oefrha 7y ago
Yeah, I recall doing import codecs sys.stdout = codecs.getwriter('utf-8')(sys.stdout) sys.stderr = codecs.getwriter('utf-8')(sys.stderr) to fix Unicode woes in Python 2, which opened another can of worms. Not fun.

slavik81 7y ago

> Python 3 forces UTF-8 everywhere If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only. Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file. It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.

bildung 7y ago
Yes, I omitted that my Python 2 (mostly Django or bottle) ran on Linux with UTF-8. I suspect your piping case could be due to either the OS not using UTF-8 as locale or the application behind the pipe ignoring the OS' locale. These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.
- slavik81 7y ago
  Just to elaborate, on a fresh Windows 7 install with en_CA locale, this was the behaviour: python3 -c 'print("é")' é python3 -c 'print("é")' > out.txt (UnicodeEncodeError) 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128) I felt that was unacceptable behaviour for my program, but it was beyond my ability to fix. Though, I think this has been addressed in more recent Python releases.
  - mehrdadn 7y ago
    I thought you might have fun with this -- try running my example here: https://news.ycombinator.com/item?id=21842471 https://news.ycombinator.com/item?id=21842471

oefrha 7y ago

Yeah, I recall doing import codecs sys.stdout = codecs.getwriter('utf-8')(sys.stdout) sys.stderr = codecs.getwriter('utf-8')(sys.stderr) to fix Unicode woes in Python 2, which opened another can of worms. Not fun.