4 ms·
The real problem here is that * UNIX file systems allow any byte sequence that doesn't contain / or \0 as file and directory names * User interfaces have to r
by perlgeek 8y ago
The real problem here is that
* UNIX file systems allow any byte sequence that doesn't contain / or \0 as file and directory names
* User interfaces have to render that as strings, so they must decode
* There is no meta data about what the file encoding is
Many programs use the encoding from the current locale, which is mostly a good assumption, but the way that locales scope (basically per process) has nothing to do with how file names are scoped.
So, many programs make some assumptions. Some models are:
1) Assume file names are encoded in the current locale
2) Assume file names are encoded in UTF-8
3) Don't assume anything
The "correct" model would be 3), but it's not very useful. People want to be able to sort and display file names, which generally isn't very useful with binary data.
Which is why most programs, including python, use 1) or 2), and sometimes offer some kind of kludge for when the assumption doesn't hold -- and sometimes not.
IMHO a file system should store an encoding for the file names contained in it, and validate on writes that the names are correct. But of course that would be a huge POSIX incompatibility, and thus won't happen.
People just live with the current models, because they tend to be good enough. Mostly.
- makecheck 8y agoA file system doesn’t fix the problem either because there are regularly problems at file system boundaries. For example, moving from a case-sensitive file system to a case-insensitive file system, you will eventually find two distinct bags of bytes that something thinks are the same; if the original disk contains files with both names, you have to pick one or error out. Even if you know the first file system is UTF-8 and the 2nd is ISO-8859-1, the case difference can still be there. It seems to me that the only correct solution is to error out when there are at least two viable versions of a file. Even if you’re trying to “sort and display file names”, you would need to acknowledge at that point that the exact file name isn’t clear. And once your program is doing that, it doesn’t really matter why the two versions are different: maybe it’s a Unicode problem, maybe it’s not.
- goerz 8y agoWould it really be POSIX-incompatible? Does the standard mandate that a filesystem can place no such restriction on top of "filenames are unencoded bytes"? If not, then it's just that tools cannot blindly assume filenames are decodable. Isn't MacOS guaranteeing UTF8 these days, while still being POSIX-compliant?
- perlgeek 8y agoI don't know if there is an explicit mandate for that in the standard, but forbidding this that were previously allowed, both in code and documentation, is not backwards compatible. Just imagine a file system that wouldn't allow the character "e" in file names. Of course, the impact would be not as drastic, but still it's backwards incompatible.