5 ms·

Well, it's a misinterpretation of the semantics of traditional Unix filesystems. Other filesystems used by other operating systems (such as Windows' NTFS and OS

Well, it's a misinterpretation of the semantics of traditional Unix filesystems. Other filesystems used by other operating systems (such as Windows' NTFS and OS X's HFS+) genuinely do store filenames as Unicode strings, so on those platforms (or on Unix, if you've mounted one of those filesystems) Python 3.x's approach is exactly correct. That said, recent versions of Python 3 include a workaround for exactly the problem you describe: when the interpreter gets a byte-sequence value from the OS, filenames that decode to Unicode cleanly will be represented as proper Unicode strings, while filenames that can't be decoded will have the raw bytes represented as code-points in Unicode's Private Use Area. That way, even if Python can't decode the contents of a string, you can still, say, get a parameter from the command-line, pass it to open(), and be confident that you'll actually get the file the user intended.

saurik 14y ago
While I reference the issue of a mixed-codec directory in order to make clear the flaw in the operating assumption, the actual problem I am interesting in here, and which I conclude my statements with a reference to, is handling things like sys.path (hence this being a reply to the article). To respond to your comments, however: actually, the behavior on, let's say a Linux box, if you mount one of those aforementioned filesystems, will not be "exactly correct". Python 3 will attempt to encode the Unicode string using the current locale, pass it to the underlying Unix open() function, which will then have no clue what to do as it hits the filesystem. In fact, rather than just idly claiming this, I went ahead and set up exactly this test setup on one of my servers. I created a python3 script in a known specific source encoding (UTF-8) and asked it, in each of two different locales, to make a file that included an accented character, while mounted on an HFS+ disk image. hfs+# mount | grep hfs /.../hfs+.img on /.../hfs+ type hfsplus (rw,force) hfs+# cat test.py #!/usr/bin/python3 # -*- coding: utf-8 -*- open("helloä", "w") hfs+# LANG=en_US.UTF-8 ./test.py hfs+# LANG=fr_FR.ISO-8859-1 ./test.py hfs+# LANG=en_US.UTF-8 ls -la -rw-r--r-- 1 root root 0 2012-07-11 00:57 hello? -rw-r--r-- 1 root root 0 2012-07-11 00:57 helloä -rwxr-xr-x 1 root root 64 2012-07-11 00:56 test.py* hfs+# As you can see, the behavior here is really poor for something that claims to support Unicode. What we would expect to have happen is that, as I opened the file with a Unicode name on a Unicode filesystem that I would actually get the specific Unicode string that I had wanted. Instead, because Python 3 is only pretending to understand the semantics of filesystems with regards to character sets, and in fact has no way of taking advantage of the Unicode support in HFS+, we ended up with the encoding of the user's locale environment breaking our filenames. This would be akin to me doing JSON.encode() in a browser, and having a Unicode JavaScript string get converted into different JSON (which is also represented as a Unicode JavaScript string) depending on what language the user's browser is configured to use: that is a miserable Unicode failure, and not a success story. FWIW, the exact same code does work on Mac OS X (using the correct locale name of fr_FR.ISO8859-1): both versions of the script get the exact same filename. To be honest, I was somewhat surprised that they got that right ;P. I sadly do not have a Windows computer handy, as I'd love to see how it handles NTFS (which is technically UCS-2 and doesn't have the weird canonicalization behavior that HFS+ sometimes does: I can easily imagine broken corner cases with invalid UTF-16 surrogate pairs). Regardless, I really do not believe that you need to break the behavior on Linux in order to make Mac OS X work correctly. Even if such a tradeoff were required, I question whether it should be resolved with Linux on the losing end. I am not going to say that the correct solution doesn't even use Unicode strings: it just needs to more intelligently handle who is in charge of the encodings and what they semantically mean than Python 3 is prepared to do. As far as possible solutions go, I certainly do not believe that the private Unicode codepoint solution is either sufficient (as it doesn't solve the de novo filename creation problem) nor even remotely reasonable (as if I attempt to then communicate these filenames with other systems, or heaven-forbid wanted to store a filename with a private Unicode codepoint in it, I'm now screwed). (edit: Wow, BTW. They added this "solution" after I had already given up on Python 3k, so I hadn't followed up to see what they did. It seems like they ended up not using "private codepoints" as they were considering in 2008, but are instead using surrogate-halves: they are encoding "malformed UTF-8 sequences as malformed UTF-16 sequences"[1]. Given that you are allowed to store malformed UTF-16 on NTFS, I wonder how they expect that to work. Still doesn't solve either of my complaints, though.) 1: http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043934/kuhn-utf-8b.html http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043...
- lmm 14y ago
  If you set LANG=fr_FR.ISO-8859-1 you are declaring that you want your filenames encoded as ISO-8859-1; the behaviour sounds like exactly what I'd expect. It's certainly not "broken on linux"; at worst it's "broken on linux when not using a utf-8 locale", and frankly I'd expect most things to break under that circumstance.
  - saurik 14y ago
    You seem to be missing the point of this exercise: HFS+ is a Unicode-aware filesystem, so the idea that "you want your filenames encoded as ISO-8859-1" is fundamentally invalid and unimplementable. You can't specify an encoding for the files you are saving, as they are semantically "Unicode" and saved by the filesystem as UTF-16 on disk as an implementation detail. The reason that HFS+ came up is that, as a filesystem that stores filenames as actual "strings" as opposed to "arrays of bytes", you actually get the correct behavior now on OS X with Python 3 (which you also seem to have missed); thristian contended that this behavior would also work correctly if I mounted the HFS+ image on Linux, and it does not. To explicitly demonstrate this on Mac OS X (where the filename will be sent as Unicode through to the Unicode filesystem API and then saved correctly through HFS+ to disk with the original name, no matter what encoding you happen to have set as part of your locale, as your locale truly should be irrelevant in this specific situation, just as in my JSON analogy): mac$ cat test.py #!/Library/Frameworks/Python.framework/Versions/3.2/bin/python3 # -*- coding: utf-8 -*- import locale print(locale.getpreferredencoding()) open("testä", "w") mac$ LANG=en_US.UTF-8 ./test.py UTF-8 mac$ LANG=fr_FR.ISO8859-1 ./test.py ISO8859-1 mac$ ls -la total 8 drwxr-xr-x 4 saurik staff 136 Jul 11 00:36 ./ drwxr-xr-x+ 278 saurik staff 9452 Jul 11 00:28 ../ -rwxr-xr-x 1 saurik staff 159 Jul 10 18:07 test.py* -rw-r--r-- 1 saurik staff 0 Jul 11 00:36 testä mac$
    - lmm 14y ago
      >HFS+ is a Unicode-aware filesystem, so the idea that "you want your filenames encoded as ISO-8859-1" is fundamentally invalid and unimplementable Sure. But it's still the idea that linux would apply, and how linux APIs (that expect a filename to be a stream of bytes) work. I really think this is a linux/locale problem rather than a python problem.

saurik 14y ago

While I reference the issue of a mixed-codec directory in order to make clear the flaw in the operating assumption, the actual problem I am interesting in here, and which I conclude my statements with a reference to, is handling things like sys.path (hence this being a reply to the article). To respond to your comments, however: actually, the behavior on, let's say a Linux box, if you mount one of those aforementioned filesystems, will not be "exactly correct". Python 3 will attempt to encode the Unicode string using the current locale, pass it to the underlying Unix open() function, which will then have no clue what to do as it hits the filesystem. In fact, rather than just idly claiming this, I went ahead and set up exactly this test setup on one of my servers. I created a python3 script in a known specific source encoding (UTF-8) and asked it, in each of two different locales, to make a file that included an accented character, while mounted on an HFS+ disk image. hfs+# mount | grep hfs /.../hfs+.img on /.../hfs+ type hfsplus (rw,force) hfs+# cat test.py #!/usr/bin/python3 # -*- coding: utf-8 -*- open("helloä", "w") hfs+# LANG=en_US.UTF-8 ./test.py hfs+# LANG=fr_FR.ISO-8859-1 ./test.py hfs+# LANG=en_US.UTF-8 ls -la -rw-r--r-- 1 root root 0 2012-07-11 00:57 hello? -rw-r--r-- 1 root root 0 2012-07-11 00:57 helloä -rwxr-xr-x 1 root root 64 2012-07-11 00:56 test.py* hfs+# As you can see, the behavior here is really poor for something that claims to support Unicode. What we would expect to have happen is that, as I opened the file with a Unicode name on a Unicode filesystem that I would actually get the specific Unicode string that I had wanted. Instead, because Python 3 is only pretending to understand the semantics of filesystems with regards to character sets, and in fact has no way of taking advantage of the Unicode support in HFS+, we ended up with the encoding of the user's locale environment breaking our filenames. This would be akin to me doing JSON.encode() in a browser, and having a Unicode JavaScript string get converted into different JSON (which is also represented as a Unicode JavaScript string) depending on what language the user's browser is configured to use: that is a miserable Unicode failure, and not a success story. FWIW, the exact same code does work on Mac OS X (using the correct locale name of fr_FR.ISO8859-1): both versions of the script get the exact same filename. To be honest, I was somewhat surprised that they got that right ;P. I sadly do not have a Windows computer handy, as I'd love to see how it handles NTFS (which is technically UCS-2 and doesn't have the weird canonicalization behavior that HFS+ sometimes does: I can easily imagine broken corner cases with invalid UTF-16 surrogate pairs). Regardless, I really do not believe that you need to break the behavior on Linux in order to make Mac OS X work correctly. Even if such a tradeoff were required, I question whether it should be resolved with Linux on the losing end. I am not going to say that the correct solution doesn't even use Unicode strings: it just needs to more intelligently handle who is in charge of the encodings and what they semantically mean than Python 3 is prepared to do. As far as possible solutions go, I certainly do not believe that the private Unicode codepoint solution is either sufficient (as it doesn't solve the de novo filename creation problem) nor even remotely reasonable (as if I attempt to then communicate these filenames with other systems, or heaven-forbid wanted to store a filename with a private Unicode codepoint in it, I'm now screwed). (edit: Wow, BTW. They added this "solution" after I had already given up on Python 3k, so I hadn't followed up to see what they did. It seems like they ended up not using "private codepoints" as they were considering in 2008, but are instead using surrogate-halves: they are encoding "malformed UTF-8 sequences as malformed UTF-16 sequences"[1]. Given that you are allowed to store malformed UTF-16 on NTFS, I wonder how they expect that to work. Still doesn't solve either of my complaints, though.) 1: http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043934/kuhn-utf-8b.html http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043...

lmm 14y ago
If you set LANG=fr_FR.ISO-8859-1 you are declaring that you want your filenames encoded as ISO-8859-1; the behaviour sounds like exactly what I'd expect. It's certainly not "broken on linux"; at worst it's "broken on linux when not using a utf-8 locale", and frankly I'd expect most things to break under that circumstance.
- saurik 14y ago
  You seem to be missing the point of this exercise: HFS+ is a Unicode-aware filesystem, so the idea that "you want your filenames encoded as ISO-8859-1" is fundamentally invalid and unimplementable. You can't specify an encoding for the files you are saving, as they are semantically "Unicode" and saved by the filesystem as UTF-16 on disk as an implementation detail. The reason that HFS+ came up is that, as a filesystem that stores filenames as actual "strings" as opposed to "arrays of bytes", you actually get the correct behavior now on OS X with Python 3 (which you also seem to have missed); thristian contended that this behavior would also work correctly if I mounted the HFS+ image on Linux, and it does not. To explicitly demonstrate this on Mac OS X (where the filename will be sent as Unicode through to the Unicode filesystem API and then saved correctly through HFS+ to disk with the original name, no matter what encoding you happen to have set as part of your locale, as your locale truly should be irrelevant in this specific situation, just as in my JSON analogy): mac$ cat test.py #!/Library/Frameworks/Python.framework/Versions/3.2/bin/python3 # -*- coding: utf-8 -*- import locale print(locale.getpreferredencoding()) open("testä", "w") mac$ LANG=en_US.UTF-8 ./test.py UTF-8 mac$ LANG=fr_FR.ISO8859-1 ./test.py ISO8859-1 mac$ ls -la total 8 drwxr-xr-x 4 saurik staff 136 Jul 11 00:36 ./ drwxr-xr-x+ 278 saurik staff 9452 Jul 11 00:28 ../ -rwxr-xr-x 1 saurik staff 159 Jul 10 18:07 test.py* -rw-r--r-- 1 saurik staff 0 Jul 11 00:36 testä mac$
  - lmm 14y ago
    >HFS+ is a Unicode-aware filesystem, so the idea that "you want your filenames encoded as ISO-8859-1" is fundamentally invalid and unimplementable Sure. But it's still the idea that linux would apply, and how linux APIs (that expect a filename to be a stream of bytes) work. I really think this is a linux/locale problem rather than a python problem.