6 ms·
“Bags of bytes” has its own problems. If the idea of ‘character encoding’ is alien to the file system, the application layer cannot reliably show file names to
by Someone 4y ago
“Bags of bytes” has its own problems. If the idea of ‘character encoding’ is alien to the file system, the application layer cannot reliably show file names to users, other than as sequence of byte values. It cannot even assume 0x41 is an ‘A’, for example.
Historically, you configured your OS to match (what you thought was) the encoding used on your disk, but that starts to break when you have network disks or when users exchange floppies.
In practice, I think about every Unix/Linux file system at least assumes ASCII encoding for the printable ASCII range. How else does, for example, your shell find “ls” on your disk when you type those characters?
So, not even in Unix/Linux is a file name “a bag of bytes”
I think file systems should enforce an encoding, and provide a way for users of the file system to find out what it is.
Apple’s HFS, for example, stored explicit ‘code page’ ID so that the Finder knew, for example, to use the Apple Cyrillic encoding to interpret the bytes describing file names on disk for that disk.
The modern way is to just say file names are valid Unicode strings, with UTF-8 being the popular choice. Unfortunately that introduces the normalization problem.
- sacnoradhq 4y agoAre you running EBCDIC or a localization other than UTF-8 anywhere? The filesystem shouldn't arbitrate localization issues. It's the wrong place to do it in terms of performance, area-of-responsibility, and portability. NUL-terminated UTF-8 in memory and size+data on disk is the best encoding. Normalization is a multifaceted problem that needs to be handled at or before where the filename argument hits the standard library. Sometimes, normalization is NOT desirable and it is preferrable to normalize for comparison only to other strings and to faithfully store whatever was provided verbatim. Side-effects and doing 737 MAX MCAS underneath users is the road to ruin.
- gpderetta 4y ago> the application layer cannot reliably show file names to users, other than as sequence of byte values. As you say, it can just show the sequence of bytes (plus some visual hints that they are escaped). It is not like nonsensical filenames with seem composed of random numbers are uncommon (e.g. look into your .git dir) > How else does, for example, your shell find “ls” on your disk when you type those characters? This is purely a presentation layer issue. The filesystem assumes nothing.
- klausa 4y agoThe git files aren't supposed to be seen by a human being. What you're suggesting would mean that a Korean user saving a file on their desktop and naming it in Korean would in turn see the filename turn into garbage, which, I hope we can all agree, is not a reasonable suggestion.
- gpderetta 4y agoOf course if the file-name is correctly decodeable in the user preferred encoding (hopefully UTF-8) it should be shown as such. But giving up on supposedly malformed filenames shouldn't be an option for robust software.
- CorrectHorseBat 4y agoWhy though? What do you lose by forcing utf-8 at the filesystem level?
- gpderetta 4y agoCompatibility with a few decades of data and with other systems.
- eviks 4y agoBut the future is longer than a few decades, so why extend the mistake further?
- sacnoradhq 4y agoBecause change costs way more to than you think: maybe billions of dollars and impacting billions of users. Creating 2 different systems carries risks: it has a learning curve, a support cost, and breaks everything that came before. Perhaps, in a utopian green field, we redo a systems language, a portable systems API, and a common operating system with a compatibility layer to the old way on top.