4 ms·

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a

> on Windows, paths are fundamentally text They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point. Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 called surrogates; surrogate pairs need to be collapsed into a single code point when decoding the file name. Surrogates are exactly those things that Python uses on Linux to hide bytes that are not valid UTF-8. Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). In summary, on Windows, you end up with effectively the same situation as Linux: file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding.

chungy 9y ago
> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.
- quietbritishjim 9y ago
  Indeed. I should have written something more like "in places Windows appears to accept UTF-16". But there is certainly a little truth to it: if you pass a surrogate pair to e.g. the text of a Windows label using an appropriate Unicode font, I believe it will show its UTF-16 interpretation.
- burfog 9y ago
  Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.
  - masklinn 9y ago
    > Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.
    - leeter 9y ago
      This is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-normalized paths (I assume).
      - freeone3000 9y ago
        Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.
        leeter 9y ago
        Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.
        TazeTSchnitzel 9y ago
        I think it's not a perfect round-trip due to differences in how the two standards encode certain characters. You will get a correct conversion either way, but the result of a round-trip might not be bit-identical.
        rspeer 9y ago
sametmax 9y ago
Funnily, because of the 2/3 transition, Python became a language obsessed with encoding correctness. Hence the team spent a great deal improving the situation, from version to version during the last 10 years. For a fantastic read (from Victor Stinner himself) on all the work done and get how twisted this get when you want to be cross plateform and abstract it: https://vstinner.github.io/python37-new-utf8-mode.html https://vstinner.github.io/python37-new-utf8-mode.html Windows and Linux FS encoding, of course, are at the center of the challenge. It's also the reason why we now have a __fspath__ protocol that allows any object to be converted to a file system path instead of having pathlib.Path inheriting from string.
liotier 9y ago
> file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding You just described my music collection, aggregated over two decades and haphazard successive migrations. I've given up salvaging the corrupted names in an automated manner...

chungy 9y ago

> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16). NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.

quietbritishjim 9y ago
Indeed. I should have written something more like "in places Windows appears to accept UTF-16". But there is certainly a little truth to it: if you pass a surrogate pair to e.g. the text of a Windows label using an appropriate Unicode font, I believe it will show its UTF-16 interpretation.
burfog 9y ago
Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. The kernel generally uses the 16-bit equivalent of the old Pascal string, that being a 16-bit count of 16-bit pieces of UTF-16 data. This allows a 16-bit NUL to get into various places that make the Win32 API choke.
- masklinn 9y ago
  > Old versions use UCS-2. New versions use UTF-16. (correctness not enforced) This is also how Java and OS X were updated. I usually call that ucs2-plus-surrogates, to make it clear that you may encounter unpaired surrogates, and thus invalid paths if you assume proper UTF-16.
  - leeter 9y ago
    This is actually kinda painful with NTFS as NTFS doesn't really care what's in a path other than the directory separators it's all binary. This means that different applications using different Unicode normalization will result in odd things happening. To me the right answer is NTFS should normalize all paths the same way internally but they have yet to implement it because it would break legacy systems that have un-normalized paths (I assume).
    - freeone3000 9y ago
      Precisely. Programs that internally use ShiftJIS, for instance, would stop functioning on UTF-16 enforced compatability or normalization. They're currently "broken" (as in, operating incorrectly) but in a way that works.
      - leeter 9y ago
        Can you explain why this would be the case? In theory this shouldn't be an issue because any ShiftJIS conversion to Unicode should be reversible.
        TazeTSchnitzel 9y ago
        I think it's not a perfect round-trip due to differences in how the two standards encode certain characters. You will get a correct conversion either way, but the result of a round-trip might not be bit-identical.
        rspeer 9y ago

sametmax 9y ago

Funnily, because of the 2/3 transition, Python became a language obsessed with encoding correctness. Hence the team spent a great deal improving the situation, from version to version during the last 10 years. For a fantastic read (from Victor Stinner himself) on all the work done and get how twisted this get when you want to be cross plateform and abstract it: https://vstinner.github.io/python37-new-utf8-mode.html https://vstinner.github.io/python37-new-utf8-mode.html Windows and Linux FS encoding, of course, are at the center of the challenge. It's also the reason why we now have a __fspath__ protocol that allows any object to be converted to a file system path instead of having pathlib.Path inheriting from string.

liotier 9y ago

> file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding You just described my music collection, aggregated over two decades and haphazard successive migrations. I've given up salvaging the corrupted names in an automated manner...