4 ms·

> It's a problem with Unix filenames where the encoding is just a convention UNIX filenames are NOT necessarily printable text, they're byte strings. Don't tre

> It's a problem with Unix filenames where the encoding is just a convention UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. They're sequences of bytes not containing 0x00 or 0x2F, but with no encoding. Same for Windows registry keys. Don't mistake byte strings for text. Text is a byte string with an encoding which describes which byte values are valid and which characters each byte (or sequence of bytes) corresponds to. If the encoding information is discarded, it stops being text and becomes just a byte string.

Karellen 3y ago
> UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time. Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.
- SAI_Peregrinus 3y ago
  Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.
  - Karellen 3y ago
    OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.
    - SAI_Peregrinus 3y ago
      Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner. The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.
jcranmer 3y ago
To be honest, I disagree. Unix filenames are printable text, they're just text that the OS chooses not to enforce any validation on. Especially since the charset is implied by your choice of locale, which is a user decision, not a kernel decision. But we've transitioned into a world where pretty much all systems have settled on UTF-8 as the charset of choice, and the OS's refusal to even permit kernel options to forcibly validate filenames as UTF-8 is starting to look like a poor decision. (IIRC, at least one of the BSDs has actually moved to forcing filenames to be UTF-8 and refusing path names that aren't UTF-8. Would only that Linux moved down that path as well so that we could be done with this farce.)
- SAI_Peregrinus 3y ago
  A filename consisting of nothing but ASCII Bell characters (0x07) is valid. Those are non-printable characters that (used to) make a sound from the PC's speaker. POSIX filenames can be sound, not text. I'd agree it'd be nice if we could restrict filenames to valid UTF-8. But that's not the API that existing filesystems provide, nor what (most of) the existing OSes enforce.
  - glandium 3y ago
    Note that BEL(0x07) is valid UTF-8.
- snnn 3y ago
  Kernel developers need to consider backwards compatibility. You won't want to see some users lose their data because they upgraded the kernel. Therefore it is very hard to "force" something.
  - jcranmer 3y ago
    Filesystem mount option, have utilities default it to on when creating new partitions.
edflsafoiewq 3y ago
That's an important way of looking at it, and is correct as far as traditional UNIX operations like fopen go. But it isn't the whole story because many other operations require treating paths as text. For example, converting them to URIs, putting them in ZIP files, or looking up a file on a filesystem which internally stores filenames in UCS-2. Threading both of these needles at once basically requires viewing paths as potentially-invalid encoded text.

Karellen 3y ago

> UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time. Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.

SAI_Peregrinus 3y ago
Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.
- Karellen 3y ago
  OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.
  - SAI_Peregrinus 3y ago
    Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner. The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.

jcranmer 3y ago

To be honest, I disagree. Unix filenames are printable text, they're just text that the OS chooses not to enforce any validation on. Especially since the charset is implied by your choice of locale, which is a user decision, not a kernel decision. But we've transitioned into a world where pretty much all systems have settled on UTF-8 as the charset of choice, and the OS's refusal to even permit kernel options to forcibly validate filenames as UTF-8 is starting to look like a poor decision. (IIRC, at least one of the BSDs has actually moved to forcing filenames to be UTF-8 and refusing path names that aren't UTF-8. Would only that Linux moved down that path as well so that we could be done with this farce.)

SAI_Peregrinus 3y ago
A filename consisting of nothing but ASCII Bell characters (0x07) is valid. Those are non-printable characters that (used to) make a sound from the PC's speaker. POSIX filenames can be sound, not text. I'd agree it'd be nice if we could restrict filenames to valid UTF-8. But that's not the API that existing filesystems provide, nor what (most of) the existing OSes enforce.
- glandium 3y ago
  Note that BEL(0x07) is valid UTF-8.
snnn 3y ago
Kernel developers need to consider backwards compatibility. You won't want to see some users lose their data because they upgraded the kernel. Therefore it is very hard to "force" something.
- jcranmer 3y ago
  Filesystem mount option, have utilities default it to on when creating new partitions.

edflsafoiewq 3y ago

That's an important way of looking at it, and is correct as far as traditional UNIX operations like fopen go. But it isn't the whole story because many other operations require treating paths as text. For example, converting them to URIs, putting them in ZIP files, or looking up a file on a filesystem which internally stores filenames in UCS-2. Threading both of these needles at once basically requires viewing paths as potentially-invalid encoded text.