2 ms·

This problem is as Microsoft-specific as the one in the head article is IBM-specific, C as a language has very little to do with it (especially given that Micro

This problem is as Microsoft-specific as the one in the head article is IBM-specific, C as a language has very little to do with it (especially given that Microsoft very quickly pivoted from supporting portable C to exclusively doing a Windows-specific dialect and then a tacit deprecation in favour of C++). There are also limits to how far the UTF-8 illusion can go on Windows: while on Unix and friends a path is fundamentally a 0x00-terminated, 0x2F-separated sequence of 8-bit quantities[1], on NT a path is fundamentally a(n unterminated) 0x005C-separated sequence of 16-bit quantities, and Win32 puts a varying number of layers of makeup[2] on that. Thus on Unix you must be prepared to handle invalid UTF-8 in a filename, but can expect to roundtrip any byte sequence (sans 0x00 and 0x2F), and on UTF-8 Win32 you must be prepared to handle arbitrary WTF-8[3] and cannot expect to roundtrip any byte sequence (isolated surrogates can merge, though I don’t know if UTF-8 Win32 is willing to accept such invalid WTF-8). Note that Rust does not use the UTF-8 interfaces on Windows (neither does it use the fundamental UNICODE_STRING APIs, however). [1] https://yarchive.net/comp/linux/case_insensitive_filenames.html https://yarchive.net/comp/linux/case_insensitive_filenames.h... (of course, Linux filesystems have since developed case-insensitive mount options) [2] https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html https://googleprojectzero.blogspot.com/2016/02/the-definitiv... [3] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

nuc1e0n 4y ago
AFAIK There are no UTF-8 specific interfaces on windows, only the system codepage (A) and Wide character (W) interfaces. Configuring the system codepage to be utf-8 is possible, but doesn't solve all encoding problems in my experience. To get the commandline without mangling you have to use the wide character function. Plus, you don't need to be prepared to handle invalid utf-8 in filenames on unix, the fopen call can just be made to fail if needed.
- mananaysiempre 4y ago
  Right, by “UTF-8 Win32” I mean the *A Win32 functions (as used by the non-wide functions in the Microsoft C runtime) when a UTF-8 code page is active. Rust uses the *W ones instead. As being prepared for invalid UTF-8 on Unix, well, it depends. If you refuse to run with non-UTF-8 LC_CTYPE or to accept invalid UTF-8 in user-provided file names, I suppose that’s on you. (Though I sure hope you are not writing an implementation of rm or tar!) If you’re trying to erase or move everything in a directory, though, you’ll have to either deal with whatever’s there or at least recognize that the action may fail.

nuc1e0n 4y ago

AFAIK There are no UTF-8 specific interfaces on windows, only the system codepage (A) and Wide character (W) interfaces. Configuring the system codepage to be utf-8 is possible, but doesn't solve all encoding problems in my experience. To get the commandline without mangling you have to use the wide character function. Plus, you don't need to be prepared to handle invalid utf-8 in filenames on unix, the fopen call can just be made to fail if needed.

mananaysiempre 4y ago
Right, by “UTF-8 Win32” I mean the *A Win32 functions (as used by the non-wide functions in the Microsoft C runtime) when a UTF-8 code page is active. Rust uses the *W ones instead. As being prepared for invalid UTF-8 on Unix, well, it depends. If you refuse to run with non-UTF-8 LC_CTYPE or to accept invalid UTF-8 in user-provided file names, I suppose that’s on you. (Though I sure hope you are not writing an implementation of rm or tar!) If you’re trying to erase or move everything in a directory, though, you’ll have to either deal with whatever’s there or at least recognize that the action may fail.