3 ms·

> Bad things only happen sys.stdout.encoding isn't utf-8 Yes, so your code will break when the caller or user sets it to something else. > which is rare on *

> Bad things only happen sys.stdout.encoding isn't utf-8 Yes, so your code will break when the caller or user sets it to something else. > which is rare on * nix systems "UNIX only" is not exactly what I meant by "portable"... > It doesn't come for free but it's definitely not "excruciatingly difficult". It's not when you're ignoring the difficult parts! > Encoding on windows is always a headache (not specific to Python at all) It's a headache in Windows, but in some cases encoding is worse in Python 3. Sadly I've never sat down to make a library of all the examples I come across, but here's one: Try predicting what this shell script should print on each platform. (It's Bash/Batch cross-compatible, so once you tell me your prediction, try running in both.) (python2 -c "import sys; sys.stdout.write('\xDD')" && python3 -c "import sys; sys.stdout.buffer.write(b'\xDD'); sys.stdout.write('\xDD')") > Temp.bin && python3 -c "import binascii; print(binascii.hexlify(open('Temp.bin', 'rb').read()).decode('ascii'))" && rm Temp.bin

oefrha 7y ago
> Yes, so your code will break when the caller or user sets it to something else. No, it will break when the encoding is set to something where the output text simply can't be encoded. There's simply no such thing as “” in the ASCII locale, or the C locale, so when you try to print those you get an encoding error; if the encoding is set to cp1252 it would be fine. If you don't want to encode text according to user's preferred encoding, just encode as utf-8 regardless and write the bytes. Let me say this again: if you want to print text, print text; if you want to print a byte stream, print a byte stream. Python3 doesn't make this trivial but it's also not terribly hard. But by printing utf-8 byte stream under all scenarios, you would be conveniently ignoring the fact that default console encoding on Windows is cp437, not cp65001, so non-ASCII parts of the utf-8 byte stream would be garbled on Windows consoles by default. (And I see garbled Unicode filenames from programs written in other languages all too often when I'm on a Windows console.) It's a damned if you do, damned if you don't situation. Interestingly, the default now is utf-8 on Windows consoles, unless PYTHONLEGACYWINDOWSSTDIO is set. They probably decided that printing (the occasional) garbage is better than correctness. > Try predicting what this shell script should print on each platform. Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. If you want to write a byte 0xdd, don't use '\xDD' which is U+00DD which of course is encoded differently in utf-8 (0xc3 0x9d) and cp1252 (0xdd).
- mehrdadn 7y ago
  > Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. By "the encoding" you're referring to PYTHONIOENCODING, right? Somehow PYTHONIOENCODING is "not specific to Python at all"?
  - oefrha 7y ago
    PYTHONIOENCODING is an override env var you can set. cp1252 being the default on Windows is of course not Python-specific, it’s convention set by Microsoft. In fact, you do realize cp1252 aka Latin-1 was even the default encoding used by the HTML spec before the advent of HTML5 (postdates Python3)? Creators of Python3 certainly didn’t set this default to mess with you.
    - mehrdadn 7y ago
      No, this is just insane Python behavior. There's clearly nothing in the Windows I/O path that's doing this, and if it's a "convention", it's not even one that Python 2 has followed! PHP doesn't do it this way: php -r "echo json_decode(chr(34).'\u00DD'.chr(34));" Ruby doesn't do it this way either: ruby -e "puts \"\u00DD\"" Neither does MSYS2's Bash: printf '\u00DD' Neither does Node.js: node --eval="process.stdout.write('\u00DD');" Hell, even Python 2's behavior of raising an error is more sane than just encoding in cp1252 silently and expecting every developer to somehow know the resulting byte sequence is going to be different on each platform: python3 -c "print(u'\u00DD')" Obviously, I'm not the only one who thinks that whatever this (ancient?) so-called "convention" is, it's actively harmful enough to avoid in 2019. And as if that's not enough, that's not even the end of it. Even literally writing a string to a file (with no console or stdio in between!!) ends up producing a completely different file depending on which platform the code is run on: python3 -c "open('Temp.bin', 'w+').write(u'\u00DD')" && xxd Temp.bin && rm Temp.bin How is any programmer supposed to deal with this insanity? A program writes to a file, but produces files with completely different contents depending on the platform? Heaven help you if you try to interacting with a program in another language. At least with newlines, as painful as they are, most people know and recognize how to deal with them. But dealing with that mess is enough, and not a reason to make the situation even worse?! It's almost as if they took the CRLF issue and then went "Hmm, that's not fair to CR or LF, we need to do this to even more characters." Somehow we have all this nonsense in the name of "fixing" strings from Python 2. Instead of finding a better convention or at least leaving the existing behavior alone, Python 3—which is intended to let people write cross-platform code!—actively embraced it and botched Python 2's safe behavior, making programs produce garbled output completely silently... and blaming it on the stupid developer for trying to write a broken cross-platform program without first getting a PhD on the history of Code Pages on Windows.

oefrha 7y ago

> Yes, so your code will break when the caller or user sets it to something else. No, it will break when the encoding is set to something where the output text simply can't be encoded. There's simply no such thing as “” in the ASCII locale, or the C locale, so when you try to print those you get an encoding error; if the encoding is set to cp1252 it would be fine. If you don't want to encode text according to user's preferred encoding, just encode as utf-8 regardless and write the bytes. Let me say this again: if you want to print text, print text; if you want to print a byte stream, print a byte stream. Python3 doesn't make this trivial but it's also not terribly hard. But by printing utf-8 byte stream under all scenarios, you would be conveniently ignoring the fact that default console encoding on Windows is cp437, not cp65001, so non-ASCII parts of the utf-8 byte stream would be garbled on Windows consoles by default. (And I see garbled Unicode filenames from programs written in other languages all too often when I'm on a Windows console.) It's a damned if you do, damned if you don't situation. Interestingly, the default now is utf-8 on Windows consoles, unless PYTHONLEGACYWINDOWSSTDIO is set. They probably decided that printing (the occasional) garbage is better than correctness. > Try predicting what this shell script should print on each platform. Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. If you want to write a byte 0xdd, don't use '\xDD' which is U+00DD which of course is encoded differently in utf-8 (0xc3 0x9d) and cp1252 (0xdd).

mehrdadn 7y ago
> Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. By "the encoding" you're referring to PYTHONIOENCODING, right? Somehow PYTHONIOENCODING is "not specific to Python at all"?
- oefrha 7y ago
  PYTHONIOENCODING is an override env var you can set. cp1252 being the default on Windows is of course not Python-specific, it’s convention set by Microsoft. In fact, you do realize cp1252 aka Latin-1 was even the default encoding used by the HTML spec before the advent of HTML5 (postdates Python3)? Creators of Python3 certainly didn’t set this default to mess with you.
  - mehrdadn 7y ago
    No, this is just insane Python behavior. There's clearly nothing in the Windows I/O path that's doing this, and if it's a "convention", it's not even one that Python 2 has followed! PHP doesn't do it this way: php -r "echo json_decode(chr(34).'\u00DD'.chr(34));" Ruby doesn't do it this way either: ruby -e "puts \"\u00DD\"" Neither does MSYS2's Bash: printf '\u00DD' Neither does Node.js: node --eval="process.stdout.write('\u00DD');" Hell, even Python 2's behavior of raising an error is more sane than just encoding in cp1252 silently and expecting every developer to somehow know the resulting byte sequence is going to be different on each platform: python3 -c "print(u'\u00DD')" Obviously, I'm not the only one who thinks that whatever this (ancient?) so-called "convention" is, it's actively harmful enough to avoid in 2019. And as if that's not enough, that's not even the end of it. Even literally writing a string to a file (with no console or stdio in between!!) ends up producing a completely different file depending on which platform the code is run on: python3 -c "open('Temp.bin', 'w+').write(u'\u00DD')" && xxd Temp.bin && rm Temp.bin How is any programmer supposed to deal with this insanity? A program writes to a file, but produces files with completely different contents depending on the platform? Heaven help you if you try to interacting with a program in another language. At least with newlines, as painful as they are, most people know and recognize how to deal with them. But dealing with that mess is enough, and not a reason to make the situation even worse?! It's almost as if they took the CRLF issue and then went "Hmm, that's not fair to CR or LF, we need to do this to even more characters." Somehow we have all this nonsense in the name of "fixing" strings from Python 2. Instead of finding a better convention or at least leaving the existing behavior alone, Python 3—which is intended to let people write cross-platform code!—actively embraced it and botched Python 2's safe behavior, making programs produce garbled output completely silently... and blaming it on the stupid developer for trying to write a broken cross-platform program without first getting a PhD on the history of Code Pages on Windows.