3 ms·

Python 3's Unicode handling is uniquely bad. I haven't heard of any other language where you can obtain magic strings that crash the program if you try to prin

• price 7y ago
  Oh, that's a fun example. ("Fun", anyway.) The key surprising thing that's going on here is this clever hack (clever, but a hack): > In Python, file names, command line arguments, and environment variables are represented using the string type. On some systems, decoding these strings to and from bytes is necessary before passing them to the operating system. Python uses the file system encoding to perform this conversion ... > > On some systems, conversion using the file system encoding may fail. In this case, Python uses the surrogateescape encoding error handler, which means that undecodable bytes are replaced by a Unicode character U+DCxx on decoding, and these are again translated to the original byte on encoding. https://docs.python.org/3/library/os.html#file-names-command-line-arguments-and-environment-variables https://docs.python.org/3/library/os.html#file-names-command... This is meant as a way of fudging the fact that (a) for UI purposes, you want to treat filenames as text strings; (b) your Linux filenames are probably all encoded as UTF-8 (or your locale encoding); (c) but they might not be -- they could be arbitrary bytes, except only NUL; (d) and if they are, you really want to not munge the name when you go back and try to operate on the file. The fudge is that filenames get decoded as (by default) UTF-8... but if invalid, the offending bytes get stuffed into the UTF-16 surrogate space. Then filesystem APIs encode as UTF-8, except they look for that surrogate hack and turn those to the original bytes, so it all round-trips. It goes pretty wrong if you try to hand such a hacked-up string to something that just expects to encode normal real Unicode with UTF-8, though. That's what's happening in your example. The magic words are `os.fsdecode` -- that's how you get back bytes round-trip clean from that hack.
  • baq 7y ago
    what an excellent writeup of why modern languages have something like 4 stringy types: bytes, unicode, os string, path string. you could go further if you want to talk to other computers.
• overgard 7y ago
  That seems more like a difference of opinion on how to handle a bad string than an example of python 3 being bad. I'd argue failing silently is worse, as in programs outside of your toy example you might end up storing badly encoded data etc. Broken data is way worse to deal with than a crashing app