5 ms·

> These days one of the systems level programming languages (Rust) is natively UTF-8. Except that the geniuses who maintain the extant Unix systems still could

by patrec 3y ago

> These days one of the systems level programming languages (Rust) is natively UTF-8. Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, so now we're forever stuck with the terrible OsString cruft and wasting brain and CPU cycles converting from and to OsString in some unsatisfactory manner. At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag.

gpderetta 3y ago
the geniuses that maintain the extant Unix systems invented UTF-8 precisely because they value backward compatibility above everything else and won't contemplate breaking stuff for the sake of programmer convenience.
- patrec 3y ago
  UTF-8 was invented by people who were sufficiently fed up with the shortcomings of Unix that they created a quite incompatible but much improved evolution of it. And it's not just some hack for the sake of backwards compatibility at the cost of other properties, it's a vastly superior design compared to the then existing unicode encodings. And there is no serious backwards compatibility problem. You just introduce a mount option to enforce utf-8, flip it to default after a say a decade, and people who then still have file systems with pathnames in latin-1 (or other craziness) can then flip it off and get a few extra decades to migrate their stuff. In the meantime the rest of the world just writes software under the assumption that it will be deployed to non-crazy installs, and software becomes magically more reliable and quicker to write. And it's not like even 1% of software now would handle non-utf-8 filenames robustly anyway (I mean even if you are aware of the problem and diligent about it, there is absolutely no good way to deal with non-utf8 filenames if you need to output them as text for consumption by humans or other programs, which is close to 100% of cases, because at the very least you will need to do so in error messages if there is an IO problem).
  - ok123456 3y ago
    Then what happens when you mount it with the utf-8 option on one machine, and without it on another. The one without the utf-8 options writes a filename that breaks utf-8 encoding. You then try to mount it again with the utf-8 option. What happens then?
    - gpderetta 3y ago
      Not only mount. What happens when you unpack an archive that uses non-fully utf-8 compliant names. Or checkout the history of a git repo with non-utf-8 filenames. Network filesystems are also a source of pain. And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.
      - patrec 3y ago
        > What happens when you unpack an archive that uses non-fully utf-8 compliant names. Exactly what you would want to happen (this is the actual error message you will get, BTW)? > (cd /mnt/funkyfilesystem && touch $'\370' && tar cf my-bad-file.tar $'\370') > tar xf /mnt/funkyfilesystem/my-bad-file.tar tar: \370: Cannot open: Invalid or incomplete multibyte or wide character tar: Exiting with failure status due to previous errors Or are you trying to tell me that your life isn't complete without tar or a git checkout silently dropping some nonsense byte sequences into your filesystem? In both cases you can map the names to something saner explicitly and continue with whatever you were doing, or checkout to e.g. a tmpfs where you flipped off utf8-enforcement. BTW, unless you work with fairly unusual git repos or tar archives on a regular basis, this is very unlikely to ever happen to you. > And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive. (Case-preserving you mean? I think truly case-insensitive died with 8.3 DOS filenames). These are also annoying, but a much more minor issue, e.g. you don't need to create a new weird string type just because of them.
        gpderetta 3y ago
cryptonector 3y ago
> Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, [...] As far as POSIX and Unix `open(2)` and friends go, pathnames are opaque binary with only two special byte values: NUL (0x00, because it's C strings, which are NUL-terminated) and '/' (because that's the path component separator). Any codeset and form that is compatible with that will "work" -- for some value of "work" where if the codeset isn't Unicode in UTF-8 then you'll be sad. Nothing keeps users / sites from declaring that thou shall use UTF-8 on the filesystem, or, even better, that thou shall use UTF-8 locales only, as the latter is the only simple way to [mostly] get the former. > At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag. Note that ZFS can't tell if the strings from user-land are in UTF-8. ZFS can only tell if they're not valid UTF-8.

by patrec 3y ago

gpderetta 3y ago

the geniuses that maintain the extant Unix systems invented UTF-8 precisely because they value backward compatibility above everything else and won't contemplate breaking stuff for the sake of programmer convenience.

patrec 3y ago
UTF-8 was invented by people who were sufficiently fed up with the shortcomings of Unix that they created a quite incompatible but much improved evolution of it. And it's not just some hack for the sake of backwards compatibility at the cost of other properties, it's a vastly superior design compared to the then existing unicode encodings. And there is no serious backwards compatibility problem. You just introduce a mount option to enforce utf-8, flip it to default after a say a decade, and people who then still have file systems with pathnames in latin-1 (or other craziness) can then flip it off and get a few extra decades to migrate their stuff. In the meantime the rest of the world just writes software under the assumption that it will be deployed to non-crazy installs, and software becomes magically more reliable and quicker to write. And it's not like even 1% of software now would handle non-utf-8 filenames robustly anyway (I mean even if you are aware of the problem and diligent about it, there is absolutely no good way to deal with non-utf8 filenames if you need to output them as text for consumption by humans or other programs, which is close to 100% of cases, because at the very least you will need to do so in error messages if there is an IO problem).
- ok123456 3y ago
  Then what happens when you mount it with the utf-8 option on one machine, and without it on another. The one without the utf-8 options writes a filename that breaks utf-8 encoding. You then try to mount it again with the utf-8 option. What happens then?
  - gpderetta 3y ago
    Not only mount. What happens when you unpack an archive that uses non-fully utf-8 compliant names. Or checkout the history of a git repo with non-utf-8 filenames. Network filesystems are also a source of pain. And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.
    - patrec 3y ago
      > What happens when you unpack an archive that uses non-fully utf-8 compliant names. Exactly what you would want to happen (this is the actual error message you will get, BTW)? > (cd /mnt/funkyfilesystem && touch $'\370' && tar cf my-bad-file.tar $'\370') > tar xf /mnt/funkyfilesystem/my-bad-file.tar tar: \370: Cannot open: Invalid or incomplete multibyte or wide character tar: Exiting with failure status due to previous errors Or are you trying to tell me that your life isn't complete without tar or a git checkout silently dropping some nonsense byte sequences into your filesystem? In both cases you can map the names to something saner explicitly and continue with whatever you were doing, or checkout to e.g. a tmpfs where you flipped off utf8-enforcement. BTW, unless you work with fairly unusual git repos or tar archives on a regular basis, this is very unlikely to ever happen to you. > And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive. (Case-preserving you mean? I think truly case-insensitive died with 8.3 DOS filenames). These are also annoying, but a much more minor issue, e.g. you don't need to create a new weird string type just because of them.
      - gpderetta 3y ago

cryptonector 3y ago

> Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, [...] As far as POSIX and Unix `open(2)` and friends go, pathnames are opaque binary with only two special byte values: NUL (0x00, because it's C strings, which are NUL-terminated) and '/' (because that's the path component separator). Any codeset and form that is compatible with that will "work" -- for some value of "work" where if the codeset isn't Unicode in UTF-8 then you'll be sad. Nothing keeps users / sites from declaring that thou shall use UTF-8 on the filesystem, or, even better, that thou shall use UTF-8 locales only, as the latter is the only simple way to [mostly] get the former. > At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag. Note that ZFS can't tell if the strings from user-land are in UTF-8. ZFS can only tell if they're not valid UTF-8.