3 ms·
> Are you implying that by using Python 3 incorrectly you can run into unicode problems? The problem is that `print(some_string)` is now incorrect, even so it
by grumbel 12y ago
> Are you implying that by using Python 3 incorrectly you can run into unicode problems?
The problem is that `print(some_string)` is now incorrect, even so it looks perfectly harmless. So far I haven't seen any "best practice" guide on how to fix that. `sys.stdout.write(os.fsdecode(some_string)` works, but is pretty ugly. Handling all filenames as bytes instead of strings works as well, but again, that gets ugly quick. Cleanest way seems to be:
PYTHONIOENCODING=utf-8:surrogateescape python3 somescript.py
But should I depend on users setting `PYTHONIOENCODING`? Should I do that in my scripts? It feels all very rough and causes a lot of issues that you really shouldn't have to deal with in the first place.
> The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated.
Not exactly "nicely separated", the unencodable bytes get squished as surrogates into the Unicode strings waiting to cause trouble later on.
- the_mitsuhiko 12y agoJFTR. In Python 3.4 and later I think, stdout has a surrogateescape error handler by default. Not that it helps much, because this will fall apart if you write to other things. The story with this apparently is now so confusing that I am not sure what the correct way to do it on Python 3 is. From my experience most people don't care anyways. In some of my libraries the Python 3 version just refuses to work unless the encoding is utf-8 to avoid accidental failures down the line which nobody can debug.
- eeZi 12y agoUnfortunately, that has not made it into Python 3.4.