3 ms·
If you want a serious rabbit hole, think about Unicode characters in filenames. Windows filenames are supposedly UTF-16, but they do not enforce the requirement
by geofft 10y ago
If you want a serious rabbit hole, think about Unicode characters in filenames. Windows filenames are supposedly UTF-16, but they do not enforce the requirement that surrogate pairs (which represent characters over 0xFFFF, like emoji) must actually be paired, so you're not guaranteed that a UTF-16 decoder will actually read a filename successfully. UNIX filenames are just untyped byte strings that don't contain NUL or /, which by convention are ASCII or UTF-8 these days, but nothing enforces that; if you run ls, it will just print whatever bytes are in the filename to the terminal, and make it the terminal's problem. So round-tripping an arbitrarily weird UNIX filename to Windows, or vice versa, is challenging.
If you're in a position to enforce well-formed Unicode on all platforms, you're much better off. But many things (e.g. backup systems) don't get the option to just refuse files they don't like.
- jcrawfordor 10y agoThis points to a fascinating underlying difference between the operating systems. In general, UNIX attempts to be completely non-character-set aware and is built with the philosophy that how to render characters is strictly the terminal's problem. Windows, on the other hand, has a notion of a system character encoding and will try to keep everything compliant with it (with mixed success). There is a very important takeaway of this: case-sensitivity. UNIX cannot be case-insensitive for file names because the mapping of lowercase to uppercase characters is dependent on the character encoding used, which it doesn't know. Windows can (and does) coalesce case for file names because it knows the character set in use and can consult the relevant mapping. This difference in behavior produces all sorts of frustrating behavior when interacting between the two platforms, e.g. the classic case of Windows SMB mounting a share from a nix server that contains two files differentiated only by case. It'll show both entries but think they both point to the same thing. On the other hand, it's easy to create file names on a Windows device that are near impossible to name on nix. These are important things to be aware of if you ever implement a cross-platform network user environment.
- cesarb 10y ago> Windows can (and does) coalesce case for file names because it knows the character set in use and can consult the relevant mapping. Actually, it's not the character set in use. Windows uses a case mapping table which is part of the NTFS filesystem metadata. See for instance https://web.archive.org/web/20110308034840/http://blogs.msdn.com/b/michkap/archive/2005/01/16/353873.aspx https://web.archive.org/web/20110308034840/http://blogs.msdn... (Yes, this means that the mapping of lowercase to uppercase characters can change if the file is copied to another drive in the same machine!)
- legulere 10y agoThere's WTF-8 to convert broken UTF-16 into something UTF-8ish. https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ I kind of wonder if paths not being allowed to contain NUL or '/' was one reason why for codepoints that are represented through more than one byte in UTF-8 (-> all non ASCII codepoints) all bytes have the most significant bit set to 1 (https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description) This makes it impossible to have multi-byte to contain valid ascii chars like `/`. Note that macOS actually does decomposing unicode normalisation on file names, I guess because it makes handling case-insensitivity easier. (Just doing ascii case insensitivity also handles o+diaresis, but not the ö codepoint) https://developer.apple.com/library/mac/qa/qa1235/_index.html https://developer.apple.com/library/mac/qa/qa1235/_index.htm...