6 ms·

Are you implying that by using Python 3 incorrectly you can run into unicode problems? I agree with that. However, there are ways to do this safely. At the bou

by eliben 12y ago

Are you implying that by using Python 3 incorrectly you can run into unicode problems? I agree with that. However, there are ways to do this safely. At the boundaries of your code you have to deal with the unicode/bytes question, in any language out there. The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated.

grumbel 12y ago
> Are you implying that by using Python 3 incorrectly you can run into unicode problems? The problem is that `print(some_string)` is now incorrect, even so it looks perfectly harmless. So far I haven't seen any "best practice" guide on how to fix that. `sys.stdout.write(os.fsdecode(some_string)` works, but is pretty ugly. Handling all filenames as bytes instead of strings works as well, but again, that gets ugly quick. Cleanest way seems to be: PYTHONIOENCODING=utf-8:surrogateescape python3 somescript.py But should I depend on users setting `PYTHONIOENCODING`? Should I do that in my scripts? It feels all very rough and causes a lot of issues that you really shouldn't have to deal with in the first place. > The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated. Not exactly "nicely separated", the unencodable bytes get squished as surrogates into the Unicode strings waiting to cause trouble later on.
- the_mitsuhiko 12y ago
  JFTR. In Python 3.4 and later I think, stdout has a surrogateescape error handler by default. Not that it helps much, because this will fall apart if you write to other things. The story with this apparently is now so confusing that I am not sure what the correct way to do it on Python 3 is. From my experience most people don't care anyways. In some of my libraries the Python 3 version just refuses to work unless the encoding is utf-8 to avoid accidental failures down the line which nobody can debug.
  - eeZi 12y ago
    Unfortunately, that has not made it into Python 3.4.
lmm 12y ago
Python 3 does not have a good way to do cross-platform filesystem access, because some OSes treat filenames as byte sequences and other OSes treat them as strings. Arguably this was always a problem, because there were values in your program that you weren't sure whether they were a byte sequence or a string. But in practice, under python 2 you could regard a filename as an opaque "handle" of type "string" (that would in fact be a string on some platforms and a byte sequence on others), and as long as you didn't need to manipulate it directly, everything would work. Python 3 can't do that, so it desperately needs a better cross-platform way of handling filenames.
- eeZi 12y ago
  It's still an opaque handler - as long as you're just passing the filename to other OS functions, everything works. If you're printing ("directly manipulating") it, however, you have to decide what to do about non-Unicode bytes.
  - saurik 12y ago
    Even this is new, though; I don't remember when they fixed it, but Python 3.0 seriously would just ignore filenames that were not in the expected encoding... as in listing the files on the directory would just skip them as if hey were not in the folder... Python 3 had this big promise of solving a bunch of Unicode issues, but it doesn't really seem to have enabled anything not already possible--and not even terribly hard--in Python 2, while making a lot of missteps along the way and only very recently (as in, not even yet in Python 3.4, though committed to some development branch, per statements by others elsewhere on this thread) providing anything actuallt new in the area most often considered the "carrot" for upgrading to Python 3.
    - thrownaway122 12y ago
      > Python 3.0 seriously would just ignore filenames Why are you worried about python 3.0?
      - 12y ago

by eliben 12y ago

grumbel 12y ago

> Are you implying that by using Python 3 incorrectly you can run into unicode problems? The problem is that `print(some_string)` is now incorrect, even so it looks perfectly harmless. So far I haven't seen any "best practice" guide on how to fix that. `sys.stdout.write(os.fsdecode(some_string)` works, but is pretty ugly. Handling all filenames as bytes instead of strings works as well, but again, that gets ugly quick. Cleanest way seems to be: PYTHONIOENCODING=utf-8:surrogateescape python3 somescript.py But should I depend on users setting `PYTHONIOENCODING`? Should I do that in my scripts? It feels all very rough and causes a lot of issues that you really shouldn't have to deal with in the first place. > The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated. Not exactly "nicely separated", the unencodable bytes get squished as surrogates into the Unicode strings waiting to cause trouble later on.

the_mitsuhiko 12y ago
JFTR. In Python 3.4 and later I think, stdout has a surrogateescape error handler by default. Not that it helps much, because this will fall apart if you write to other things. The story with this apparently is now so confusing that I am not sure what the correct way to do it on Python 3 is. From my experience most people don't care anyways. In some of my libraries the Python 3 version just refuses to work unless the encoding is utf-8 to avoid accidental failures down the line which nobody can debug.
- eeZi 12y ago
  Unfortunately, that has not made it into Python 3.4.

lmm 12y ago

Python 3 does not have a good way to do cross-platform filesystem access, because some OSes treat filenames as byte sequences and other OSes treat them as strings. Arguably this was always a problem, because there were values in your program that you weren't sure whether they were a byte sequence or a string. But in practice, under python 2 you could regard a filename as an opaque "handle" of type "string" (that would in fact be a string on some platforms and a byte sequence on others), and as long as you didn't need to manipulate it directly, everything would work. Python 3 can't do that, so it desperately needs a better cross-platform way of handling filenames.

eeZi 12y ago
It's still an opaque handler - as long as you're just passing the filename to other OS functions, everything works. If you're printing ("directly manipulating") it, however, you have to decide what to do about non-Unicode bytes.
- saurik 12y ago
  Even this is new, though; I don't remember when they fixed it, but Python 3.0 seriously would just ignore filenames that were not in the expected encoding... as in listing the files on the directory would just skip them as if hey were not in the folder... Python 3 had this big promise of solving a bunch of Unicode issues, but it doesn't really seem to have enabled anything not already possible--and not even terribly hard--in Python 2, while making a lot of missteps along the way and only very recently (as in, not even yet in Python 3.4, though committed to some development branch, per statements by others elsewhere on this thread) providing anything actuallt new in the area most often considered the "carrot" for upgrading to Python 3.
  - thrownaway122 12y ago
    > Python 3.0 seriously would just ignore filenames Why are you worried about python 3.0?
    - 12y ago