2 ms·
In my experience, there's much less trouble with Unicode in Python 3, as it forces you to explicitly convert to/from Unicode. Filesystem encoding is somewhat o
by eeZi 11y ago
In my experience, there's much less trouble with Unicode in Python 3, as it forces you to explicitly convert to/from Unicode.
Filesystem encoding is somewhat of an exception, but there's no good solution to this. In Python 2, the above would print binary characters, which is generally not what you want, potentially confusing the terminal emulator. In Python 3, you have to explicitly specify whether you want to output the original bytes or replace any bytes which aren't valid Unicode:
>>> os.listdir()[0].encode(errors='surrogateescape')
b'\xfe\xb1\xb7\x95[\xc9t\xc3\x18y'
>>> os.listdir()[0].encode(errors='replace')
b'????[?t?\x18y'
(the latter matches the behaviour of "ls")
The equivalent to print(os.listdir()[0]) would be the following:
sys.stdout.buffer.write(os.listdir()[0].encode(errors='surrogateescape'))
This is more verbose, but also much more explicit, which is a good thing. If you want this as the default behaviour, you can set the environment variable PYTHONIOENCODING=utf-8:surrogateescape (or alternatively "replace"), and the following now works, just as with Python 2:
>>> sys.stdout.write(os.listdir()[0])
(binary output)
>>> print(os.listdir()[0])
(binary output)
Alternatively, you can do the following:
>>> sys.stdout = io.TextIOWrapper(sys.stdout.buffer, errors="surrogateescape")
>>> print(os.listdir()[0])
(binary output)
Note that print(os.listdir()) would not blow up since it's a list and Python would just call repr() on it:
>>> print(os.listdir())
['\udcfe\udcb1\udcb7\udc95[\udcc9t\udcc3\x18y']
Judging from the responses in this thread, it's either not properly documented, or people aren't reading the docs.
There is some discussion about making "surrogateescape" the default sys.std{out,in,err} encoding error handler in Python 3.5 in order to avoid surprises.