3 ms·

> I've found it excruciatingly difficult to write Python code that handles non-ASCII stdio correctly, especially one that might display on a terminal, especiall

• n3storm 7y ago
  I think the issue comes from PHP magic with non-ASCII values. PHP tries to guess and never fails. Python instead doesn't do this for you. Anyway, some non-english speaker countries developers feel like programming language developers still have different encodings as second citizens like this was the sixties. I hope this changes some time in the future although 10 years ago I already hoped for today to be that future.
  • oefrha 7y ago
    Let me quote the sys.stdout/sys.stderr encoding rules: > The character encoding is platform-dependent. Non-Windows platforms use the locale encoding (see locale.getpreferredencoding()). > On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage). Non-console character devices such as NUL (i.e. where isatty() returns True) use the value of the console input and output codepages at startup, respectively for stdin and stdout/stderr. This defaults to the system locale encoding if the process is not initially attached to a console. > The special behaviour of the console can be overridden by setting the environment variable PYTHONLEGACYWINDOWSSTDIO before starting Python. In that case, the console codepages are used as for any other character device. > Under all platforms, you can override the character encoding by setting the PYTHONIOENCODING environment variable before starting Python or by using the new -X utf8 command line option and PYTHONUTF8 environment variable. However, for the Windows console, this only applies when PYTHONLEGACYWINDOWSSTDIO is also set. https://docs.python.org/3/library/sys.html#sys.stdout https://docs.python.org/3/library/sys.html#sys.stdout locale.getpreferredencoding doc is vague > Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess. https://docs.python.org/3/library/locale.html#locale.getpreferredencoding https://docs.python.org/3/library/locale.html#locale.getpref... But on *nix it's basically based on the second part of your locale. If it's anything.UTF-8 (and your terminal talks UTF-8) there won't be a problem. If you set your locale to C or something.ASCII or whatever you can't really blame Python for trying to encode the output according to what you claim to prefer. I fail to see how you can "guess" the terminal's expected encoding otherwise.
• mehrdadn 7y ago
  > Bad things only happen sys.stdout.encoding isn't utf-8 Yes, so your code will break when the caller or user sets it to something else. > which is rare on * nix systems "UNIX only" is not exactly what I meant by "portable"... > It doesn't come for free but it's definitely not "excruciatingly difficult". It's not when you're ignoring the difficult parts! > Encoding on windows is always a headache (not specific to Python at all) It's a headache in Windows, but in some cases encoding is worse in Python 3. Sadly I've never sat down to make a library of all the examples I come across, but here's one: Try predicting what this shell script should print on each platform. (It's Bash/Batch cross-compatible, so once you tell me your prediction, try running in both.) (python2 -c "import sys; sys.stdout.write('\xDD')" && python3 -c "import sys; sys.stdout.buffer.write(b'\xDD'); sys.stdout.write('\xDD')") > Temp.bin && python3 -c "import binascii; print(binascii.hexlify(open('Temp.bin', 'rb').read()).decode('ascii'))" && rm Temp.bin
  • oefrha 7y ago
    > Yes, so your code will break when the caller or user sets it to something else. No, it will break when the encoding is set to something where the output text simply can't be encoded. There's simply no such thing as “” in the ASCII locale, or the C locale, so when you try to print those you get an encoding error; if the encoding is set to cp1252 it would be fine. If you don't want to encode text according to user's preferred encoding, just encode as utf-8 regardless and write the bytes. Let me say this again: if you want to print text, print text; if you want to print a byte stream, print a byte stream. Python3 doesn't make this trivial but it's also not terribly hard. But by printing utf-8 byte stream under all scenarios, you would be conveniently ignoring the fact that default console encoding on Windows is cp437, not cp65001, so non-ASCII parts of the utf-8 byte stream would be garbled on Windows consoles by default. (And I see garbled Unicode filenames from programs written in other languages all too often when I'm on a Windows console.) It's a damned if you do, damned if you don't situation. Interestingly, the default now is utf-8 on Windows consoles, unless PYTHONLEGACYWINDOWSSTDIO is set. They probably decided that printing (the occasional) garbage is better than correctness. > Try predicting what this shell script should print on each platform. Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. If you want to write a byte 0xdd, don't use '\xDD' which is U+00DD which of course is encoded differently in utf-8 (0xc3 0x9d) and cp1252 (0xdd).