6 ms·

By far the biggest problem with reading a file in C is Microsoft's ill-conceived wide-char functions, _wfopen, etc. that produced decades of "ensure the path ha

By far the biggest problem with reading a file in C is Microsoft's ill-conceived wide-char functions, _wfopen, etc. that produced decades of "ensure the path has no unicode characters" problems. Basically every C/C++ project has a wrapper to fix this. The good news is the bad days may be over soon, thanks to MS moving towards the Unix solution of using UTF8, as well as modern languages like Rust moving this stuff into the stdlib where you can't mess it up.

zokier 4y ago
> The good news is the bad days may be over soon, thanks to MS moving towards the Unix solution of using UTF8, as well as modern languages like Rust moving this stuff into the stdlib where you can't mess it up. It's not all roses on unix or rust side either. In unix filenames are not utf-8, which leads rust having fun things like OsString.
- jcranmer 4y ago
  On Unix, filenames are probably UTF-8. The kernel doesn't require them to be UTF-8, but if you're stuck trying to display a filename, you have to figure out what charset the filename is in, and UTF-8 is almost certainly the answer.
  - jefftk 4y ago
    APIs don't do well with "probably". For example, say you're working with a language (Python, Rust, etc) that distinguishes between utf-8 strings and byte strings. You're making an API for listing a directory: it gives you an array of strings but what kind should they be? (Python's approach is that os.listdir("/") gives you a List[str] (silently omitting undecodable entries), while os.listdir(b"/") gives you a List[bytes]. That is, if you give the path as a utf-8 string it returns utf-8 strings, otherwise it returns bytes.)
    - jcranmer 4y ago
      Honestly? Convince operating systems to have a switch that enforces UTF-8 path names, and then convince distros to flip that switch by default for new installs. That is to say, we need to move the world from a probably to a definitely state. The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.
      - Joker_vD 4y ago
        While I am all for an OS monoculture, it's still not there yet... and besides, my idea of what OS precisely must be the sole survivor is different from yours, and yours is different from somebody else's, etc. So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.
      - jefftk 4y ago
        On a system with that switch flipped, what should happen if you plug in a USB drive or untar an archive that has non-utf8 filenames?
        deathanatos 4y ago
        For a legacy/broken FS that permits non-utf8 filenames, and has them: have the FS driver map them into UTF-8 as best it can by making some sort of compromise. E.g., use the PUA to map malformed sequences in/out. For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.
mananaysiempre 4y ago
This problem is as Microsoft-specific as the one in the head article is IBM-specific, C as a language has very little to do with it (especially given that Microsoft very quickly pivoted from supporting portable C to exclusively doing a Windows-specific dialect and then a tacit deprecation in favour of C++). There are also limits to how far the UTF-8 illusion can go on Windows: while on Unix and friends a path is fundamentally a 0x00-terminated, 0x2F-separated sequence of 8-bit quantities[1], on NT a path is fundamentally a(n unterminated) 0x005C-separated sequence of 16-bit quantities, and Win32 puts a varying number of layers of makeup[2] on that. Thus on Unix you must be prepared to handle invalid UTF-8 in a filename, but can expect to roundtrip any byte sequence (sans 0x00 and 0x2F), and on UTF-8 Win32 you must be prepared to handle arbitrary WTF-8[3] and cannot expect to roundtrip any byte sequence (isolated surrogates can merge, though I don’t know if UTF-8 Win32 is willing to accept such invalid WTF-8). Note that Rust does not use the UTF-8 interfaces on Windows (neither does it use the fundamental UNICODE_STRING APIs, however). [1] https://yarchive.net/comp/linux/case_insensitive_filenames.html https://yarchive.net/comp/linux/case_insensitive_filenames.h... (of course, Linux filesystems have since developed case-insensitive mount options) [2] https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html https://googleprojectzero.blogspot.com/2016/02/the-definitiv... [3] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- nuc1e0n 4y ago
  AFAIK There are no UTF-8 specific interfaces on windows, only the system codepage (A) and Wide character (W) interfaces. Configuring the system codepage to be utf-8 is possible, but doesn't solve all encoding problems in my experience. To get the commandline without mangling you have to use the wide character function. Plus, you don't need to be prepared to handle invalid utf-8 in filenames on unix, the fopen call can just be made to fail if needed.
  - mananaysiempre 4y ago
    Right, by “UTF-8 Win32” I mean the *A Win32 functions (as used by the non-wide functions in the Microsoft C runtime) when a UTF-8 code page is active. Rust uses the *W ones instead. As being prepared for invalid UTF-8 on Unix, well, it depends. If you refuse to run with non-UTF-8 LC_CTYPE or to accept invalid UTF-8 in user-provided file names, I suppose that’s on you. (Though I sure hope you are not writing an implementation of rm or tar!) If you’re trying to erase or move everything in a directory, though, you’ll have to either deal with whatever’s there or at least recognize that the action may fail.

zokier 4y ago

> The good news is the bad days may be over soon, thanks to MS moving towards the Unix solution of using UTF8, as well as modern languages like Rust moving this stuff into the stdlib where you can't mess it up. It's not all roses on unix or rust side either. In unix filenames are not utf-8, which leads rust having fun things like OsString.

jcranmer 4y ago
On Unix, filenames are probably UTF-8. The kernel doesn't require them to be UTF-8, but if you're stuck trying to display a filename, you have to figure out what charset the filename is in, and UTF-8 is almost certainly the answer.
- jefftk 4y ago
  APIs don't do well with "probably". For example, say you're working with a language (Python, Rust, etc) that distinguishes between utf-8 strings and byte strings. You're making an API for listing a directory: it gives you an array of strings but what kind should they be? (Python's approach is that os.listdir("/") gives you a List[str] (silently omitting undecodable entries), while os.listdir(b"/") gives you a List[bytes]. That is, if you give the path as a utf-8 string it returns utf-8 strings, otherwise it returns bytes.)
  - jcranmer 4y ago
    Honestly? Convince operating systems to have a switch that enforces UTF-8 path names, and then convince distros to flip that switch by default for new installs. That is to say, we need to move the world from a probably to a definitely state. The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.
    - Joker_vD 4y ago
      While I am all for an OS monoculture, it's still not there yet... and besides, my idea of what OS precisely must be the sole survivor is different from yours, and yours is different from somebody else's, etc. So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.
    - jefftk 4y ago
      On a system with that switch flipped, what should happen if you plug in a USB drive or untar an archive that has non-utf8 filenames?
      - deathanatos 4y ago
        For a legacy/broken FS that permits non-utf8 filenames, and has them: have the FS driver map them into UTF-8 as best it can by making some sort of compromise. E.g., use the PUA to map malformed sequences in/out. For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.

mananaysiempre 4y ago

This problem is as Microsoft-specific as the one in the head article is IBM-specific, C as a language has very little to do with it (especially given that Microsoft very quickly pivoted from supporting portable C to exclusively doing a Windows-specific dialect and then a tacit deprecation in favour of C++). There are also limits to how far the UTF-8 illusion can go on Windows: while on Unix and friends a path is fundamentally a 0x00-terminated, 0x2F-separated sequence of 8-bit quantities[1], on NT a path is fundamentally a(n unterminated) 0x005C-separated sequence of 16-bit quantities, and Win32 puts a varying number of layers of makeup[2] on that. Thus on Unix you must be prepared to handle invalid UTF-8 in a filename, but can expect to roundtrip any byte sequence (sans 0x00 and 0x2F), and on UTF-8 Win32 you must be prepared to handle arbitrary WTF-8[3] and cannot expect to roundtrip any byte sequence (isolated surrogates can merge, though I don’t know if UTF-8 Win32 is willing to accept such invalid WTF-8). Note that Rust does not use the UTF-8 interfaces on Windows (neither does it use the fundamental UNICODE_STRING APIs, however). [1] https://yarchive.net/comp/linux/case_insensitive_filenames.html https://yarchive.net/comp/linux/case_insensitive_filenames.h... (of course, Linux filesystems have since developed case-insensitive mount options) [2] https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html https://googleprojectzero.blogspot.com/2016/02/the-definitiv... [3] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

nuc1e0n 4y ago
AFAIK There are no UTF-8 specific interfaces on windows, only the system codepage (A) and Wide character (W) interfaces. Configuring the system codepage to be utf-8 is possible, but doesn't solve all encoding problems in my experience. To get the commandline without mangling you have to use the wide character function. Plus, you don't need to be prepared to handle invalid utf-8 in filenames on unix, the fopen call can just be made to fail if needed.
- mananaysiempre 4y ago
  Right, by “UTF-8 Win32” I mean the *A Win32 functions (as used by the non-wide functions in the Microsoft C runtime) when a UTF-8 code page is active. Rust uses the *W ones instead. As being prepared for invalid UTF-8 on Unix, well, it depends. If you refuse to run with non-UTF-8 LC_CTYPE or to accept invalid UTF-8 in user-provided file names, I suppose that’s on you. (Though I sure hope you are not writing an implementation of rm or tar!) If you’re trying to erase or move everything in a directory, though, you’ll have to either deal with whatever’s there or at least recognize that the action may fail.