6 ms·

The solution is simply what Linux does: treat filenames as bags of bytes instead of strings (but this would also include case-sensitivity). At least then the be

The solution is simply what Linux does: treat filenames as bags of bytes instead of strings (but this would also include case-sensitivity). At least then the behaviour is completely straightforward and transparent down in the file system layer. UNICODE details like normalization can be handled up in the application layer.

ClumsyPilot 4y ago
Do you speak other languages? Czech language has Á, Č, Ď, Ě, É, Í, Ň, Ó, Ř, Š, Ť, Ů, Ú, Ý Á can be represented in Unicode by its spesific codepoint, or can be made up of two - Letter A and the squiggly bit. So when a user tries to fOpen a file called ÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁ Should by application make 2^32 requests to the file systen to open a file for every possible combination of bytes this steing could be represented as?
- FeepingCreature 4y ago
  To be fair, what a sufficiently clever application could do would be to try and naively open the file, note a mismatch, then do a directory walk comparing normalized filenames. Still not great.
  - flohofwoe 4y ago
    Ideally, un-normalized paths wouldn't even make it down to the filesystem layer. I'm just saying that it shouldn't be the filesystem's job to care about text encoding details, but the layers above it (e.g. the filesystem would work in a simple "garbage in - garbage out" mode).
- deleted 4y ago
  [deleted]
- flohofwoe 4y ago
  I speak German which has its share of pointy letters :) I'm aware of the normalization problem, but that should be handled in the API layers above the filesystem. In which layer exactly is up for discussion though (but IMHO there should always be a fallback to access files via their "raw bytes path identifiers", if for nothing else than debugging).
  - sacnoradhq 4y ago
    Yes. Separate concerns and group logic where the knowledge and configuration to handle it for the use exist closer to the application, not scatter it throughout the system stack that shouldn't need to know anything above it. The main ways of finding files are by enumeration and filtering on that enumeration. Normalization should selectively occur during filtering rather than permanently destroying the original bytes of the name of the file. Without a binary-precise filename call to open() (fopen() is a library call, not a syscall), then case-folding and Unicode normalization shouldn't second-guess the request or help the program. Case-insensitive filesystems are industry nonstandard and a dumb choice Apple continues to use.
- Wicher 4y ago
  At this point the user will discover that easy to type unambiguously encoded filenames have value, and may opt to save the Á's for when they actually are writing a letter to grandma. "But that's subjecting the user to limitations of the system!" you might object, and while you'd be right we should also ponder whether a filesystem is a word processor or a system for locating data using filenames as identifiers, and consequently, what sound handling of identifiers is. And let's be realistic. Most users will choose the file to open from a file picker dialog. fopen() is not their interface, their interface is a file picker. The intersection of people for whom fopen() is their interface (no file pickers) but for whom learning lessons on the value of using unambiguous identifiers is somehow out of reach, would be exceedingly small.
  - hibbelig 4y ago
    > At this point the user will discover that easy to type unambiguously encoded filenames have value, and may opt to save the Á's for when they actually are writing a letter to grandma. The unambiguously encoded filenames wouldn't be in the natural language of the user, so you're asking the user to learn another language. Users shouldn't have to learn another language to use the computer.
    - Wicher 4y ago
      I'm not 100% sure on this, but in many/most/all scripts, wouldn't there be a subset of characters in unicode for which you don't need to worry about unicode ambiguities with respect to combining characters? For prose of course it wouldn't be acceptable if you can't use your full language, but this is a filesystem, and we're talking identifiers, not prose. We accept that for a wide range of everyday things — I don't think many people are getting worked up about web forms not accepting their phone numbers when entered in Latin numerals. Using bytestring labels sans normalization is just the rational choice, and actually quite accommodating. People _can_ use their natural language. If you can encode your prose in bytes, then there you go, that'll be your filename ;-) and if you can't remember what diacritic-combining strategy you used when you created the file, and can't manage to fopen() it anymore by typing out the filename, use a file picker or search. And then, perhaps, go "that was a hassle, I'll avoid those fu͝n͜n͏y͝ sq̧u͜igg̶l̢įȩş for my next filename, computer are stupid" - not the end of the world, not the worst outcome.
sacnoradhq 4y ago
This is the answer. Normalization cannot and should not be guessed at by the filesystem. It's the wrong place to do it. If normalization is generally (but not universally) desirable, then the standard library (libc, etc.) should provide options to make it more convenient with sane defaults. Second-guessing users and developers with assumptions outside of its domain is a recipe for failure and endless workarounds.
eviks 4y ago
It's straightforward, transparent, and wrong, because filenames are not a bag of bytes, they are labels by and for humans, so no good API should ignore this simple fact
- Wicher 4y ago
  Nothing stops you from encoding your human-labels as bytestrings. Most fellow humans do exactly that. "Gosh I wish the relation between this file's human-label and its filesystem-identity were something other than bijective", said no Linux user ever. A filename is an identity and if identities are ambiguous, you run into trouble, examples in this thread abound. Putting it another way, do you work with relational databases? Primary keys? Those are identities. I don't even want to think about a world in which I can't roundtrip data through a database and have things NOT come out the way I put them in. "Human labels" or not ;-) Of course it's fine to search for things case-insensitively or otherwise normalized. And so we have "find -iname" and "locate -i" etc. etc. That's totally reasonable.
  - kalleboo 4y ago
    Personally I loved how Classic MacOS stored all it's file references as an "alias" data type which used the equivalent of inode, going as far as encoding file server connection details for auto-reconnect and a fallback path of inodes and names to traverse in case the file was removed and replaced. But that worked in an OS that had no command line - UNIX users are allergic to binary configuration formats and demand to be able to type path names.
  - eviks 4y ago
    > Nothing stops you from encoding your human-labels as bytestrings The fact that inhuman labels can sneak in stops me > said no Linux user ever You just weren't listening > you run into trouble, examples in this thread abound Bag of bytes approach is not trouble free > Primary keys? Do those accept any bytes in a bag? > can't roundtrip data through a database and have things NOT come out the way I put them in So you just want normalisation-preserving filesystem? But I see that you care so little about human users that you think they should learn to avoid Á outside of letters to grandma...
Someone 4y ago
“Bags of bytes” has its own problems. If the idea of ‘character encoding’ is alien to the file system, the application layer cannot reliably show file names to users, other than as sequence of byte values. It cannot even assume 0x41 is an ‘A’, for example. Historically, you configured your OS to match (what you thought was) the encoding used on your disk, but that starts to break when you have network disks or when users exchange floppies. In practice, I think about every Unix/Linux file system at least assumes ASCII encoding for the printable ASCII range. How else does, for example, your shell find “ls” on your disk when you type those characters? So, not even in Unix/Linux is a file name “a bag of bytes” I think file systems should enforce an encoding, and provide a way for users of the file system to find out what it is. Apple’s HFS, for example, stored explicit ‘code page’ ID so that the Finder knew, for example, to use the Apple Cyrillic encoding to interpret the bytes describing file names on disk for that disk. The modern way is to just say file names are valid Unicode strings, with UTF-8 being the popular choice. Unfortunately that introduces the normalization problem.
- sacnoradhq 4y ago
  Are you running EBCDIC or a localization other than UTF-8 anywhere? The filesystem shouldn't arbitrate localization issues. It's the wrong place to do it in terms of performance, area-of-responsibility, and portability. NUL-terminated UTF-8 in memory and size+data on disk is the best encoding. Normalization is a multifaceted problem that needs to be handled at or before where the filename argument hits the standard library. Sometimes, normalization is NOT desirable and it is preferrable to normalize for comparison only to other strings and to faithfully store whatever was provided verbatim. Side-effects and doing 737 MAX MCAS underneath users is the road to ruin.
- gpderetta 4y ago
  > the application layer cannot reliably show file names to users, other than as sequence of byte values. As you say, it can just show the sequence of bytes (plus some visual hints that they are escaped). It is not like nonsensical filenames with seem composed of random numbers are uncommon (e.g. look into your .git dir) > How else does, for example, your shell find “ls” on your disk when you type those characters? This is purely a presentation layer issue. The filesystem assumes nothing.
j16sdiz 4y ago
... and this cause lots of problem in python2 to python3 migration. Command line argument is treated as (unicode) string, and file name are (sometimes) treated as bag of byte .....

ClumsyPilot 4y ago

Do you speak other languages? Czech language has Á, Č, Ď, Ě, É, Í, Ň, Ó, Ř, Š, Ť, Ů, Ú, Ý Á can be represented in Unicode by its spesific codepoint, or can be made up of two - Letter A and the squiggly bit. So when a user tries to fOpen a file called ÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁÁ Should by application make 2^32 requests to the file systen to open a file for every possible combination of bytes this steing could be represented as?

FeepingCreature 4y ago
To be fair, what a sufficiently clever application could do would be to try and naively open the file, note a mismatch, then do a directory walk comparing normalized filenames. Still not great.
- flohofwoe 4y ago
  Ideally, un-normalized paths wouldn't even make it down to the filesystem layer. I'm just saying that it shouldn't be the filesystem's job to care about text encoding details, but the layers above it (e.g. the filesystem would work in a simple "garbage in - garbage out" mode).
deleted 4y ago
[deleted]
flohofwoe 4y ago
I speak German which has its share of pointy letters :) I'm aware of the normalization problem, but that should be handled in the API layers above the filesystem. In which layer exactly is up for discussion though (but IMHO there should always be a fallback to access files via their "raw bytes path identifiers", if for nothing else than debugging).
- sacnoradhq 4y ago
  Yes. Separate concerns and group logic where the knowledge and configuration to handle it for the use exist closer to the application, not scatter it throughout the system stack that shouldn't need to know anything above it. The main ways of finding files are by enumeration and filtering on that enumeration. Normalization should selectively occur during filtering rather than permanently destroying the original bytes of the name of the file. Without a binary-precise filename call to open() (fopen() is a library call, not a syscall), then case-folding and Unicode normalization shouldn't second-guess the request or help the program. Case-insensitive filesystems are industry nonstandard and a dumb choice Apple continues to use.
Wicher 4y ago
At this point the user will discover that easy to type unambiguously encoded filenames have value, and may opt to save the Á's for when they actually are writing a letter to grandma. "But that's subjecting the user to limitations of the system!" you might object, and while you'd be right we should also ponder whether a filesystem is a word processor or a system for locating data using filenames as identifiers, and consequently, what sound handling of identifiers is. And let's be realistic. Most users will choose the file to open from a file picker dialog. fopen() is not their interface, their interface is a file picker. The intersection of people for whom fopen() is their interface (no file pickers) but for whom learning lessons on the value of using unambiguous identifiers is somehow out of reach, would be exceedingly small.
- hibbelig 4y ago
  > At this point the user will discover that easy to type unambiguously encoded filenames have value, and may opt to save the Á's for when they actually are writing a letter to grandma. The unambiguously encoded filenames wouldn't be in the natural language of the user, so you're asking the user to learn another language. Users shouldn't have to learn another language to use the computer.
  - Wicher 4y ago
    I'm not 100% sure on this, but in many/most/all scripts, wouldn't there be a subset of characters in unicode for which you don't need to worry about unicode ambiguities with respect to combining characters? For prose of course it wouldn't be acceptable if you can't use your full language, but this is a filesystem, and we're talking identifiers, not prose. We accept that for a wide range of everyday things — I don't think many people are getting worked up about web forms not accepting their phone numbers when entered in Latin numerals. Using bytestring labels sans normalization is just the rational choice, and actually quite accommodating. People _can_ use their natural language. If you can encode your prose in bytes, then there you go, that'll be your filename ;-) and if you can't remember what diacritic-combining strategy you used when you created the file, and can't manage to fopen() it anymore by typing out the filename, use a file picker or search. And then, perhaps, go "that was a hassle, I'll avoid those fu͝n͜n͏y͝ sq̧u͜igg̶l̢įȩş for my next filename, computer are stupid" - not the end of the world, not the worst outcome.

sacnoradhq 4y ago

This is the answer. Normalization cannot and should not be guessed at by the filesystem. It's the wrong place to do it. If normalization is generally (but not universally) desirable, then the standard library (libc, etc.) should provide options to make it more convenient with sane defaults. Second-guessing users and developers with assumptions outside of its domain is a recipe for failure and endless workarounds.

eviks 4y ago

It's straightforward, transparent, and wrong, because filenames are not a bag of bytes, they are labels by and for humans, so no good API should ignore this simple fact

Wicher 4y ago
Nothing stops you from encoding your human-labels as bytestrings. Most fellow humans do exactly that. "Gosh I wish the relation between this file's human-label and its filesystem-identity were something other than bijective", said no Linux user ever. A filename is an identity and if identities are ambiguous, you run into trouble, examples in this thread abound. Putting it another way, do you work with relational databases? Primary keys? Those are identities. I don't even want to think about a world in which I can't roundtrip data through a database and have things NOT come out the way I put them in. "Human labels" or not ;-) Of course it's fine to search for things case-insensitively or otherwise normalized. And so we have "find -iname" and "locate -i" etc. etc. That's totally reasonable.
- kalleboo 4y ago
  Personally I loved how Classic MacOS stored all it's file references as an "alias" data type which used the equivalent of inode, going as far as encoding file server connection details for auto-reconnect and a fallback path of inodes and names to traverse in case the file was removed and replaced. But that worked in an OS that had no command line - UNIX users are allergic to binary configuration formats and demand to be able to type path names.
- eviks 4y ago
  > Nothing stops you from encoding your human-labels as bytestrings The fact that inhuman labels can sneak in stops me > said no Linux user ever You just weren't listening > you run into trouble, examples in this thread abound Bag of bytes approach is not trouble free > Primary keys? Do those accept any bytes in a bag? > can't roundtrip data through a database and have things NOT come out the way I put them in So you just want normalisation-preserving filesystem? But I see that you care so little about human users that you think they should learn to avoid Á outside of letters to grandma...

Someone 4y ago

“Bags of bytes” has its own problems. If the idea of ‘character encoding’ is alien to the file system, the application layer cannot reliably show file names to users, other than as sequence of byte values. It cannot even assume 0x41 is an ‘A’, for example. Historically, you configured your OS to match (what you thought was) the encoding used on your disk, but that starts to break when you have network disks or when users exchange floppies. In practice, I think about every Unix/Linux file system at least assumes ASCII encoding for the printable ASCII range. How else does, for example, your shell find “ls” on your disk when you type those characters? So, not even in Unix/Linux is a file name “a bag of bytes” I think file systems should enforce an encoding, and provide a way for users of the file system to find out what it is. Apple’s HFS, for example, stored explicit ‘code page’ ID so that the Finder knew, for example, to use the Apple Cyrillic encoding to interpret the bytes describing file names on disk for that disk. The modern way is to just say file names are valid Unicode strings, with UTF-8 being the popular choice. Unfortunately that introduces the normalization problem.

sacnoradhq 4y ago
Are you running EBCDIC or a localization other than UTF-8 anywhere? The filesystem shouldn't arbitrate localization issues. It's the wrong place to do it in terms of performance, area-of-responsibility, and portability. NUL-terminated UTF-8 in memory and size+data on disk is the best encoding. Normalization is a multifaceted problem that needs to be handled at or before where the filename argument hits the standard library. Sometimes, normalization is NOT desirable and it is preferrable to normalize for comparison only to other strings and to faithfully store whatever was provided verbatim. Side-effects and doing 737 MAX MCAS underneath users is the road to ruin.
gpderetta 4y ago
> the application layer cannot reliably show file names to users, other than as sequence of byte values. As you say, it can just show the sequence of bytes (plus some visual hints that they are escaped). It is not like nonsensical filenames with seem composed of random numbers are uncommon (e.g. look into your .git dir) > How else does, for example, your shell find “ls” on your disk when you type those characters? This is purely a presentation layer issue. The filesystem assumes nothing.

j16sdiz 4y ago

... and this cause lots of problem in python2 to python3 migration. Command line argument is treated as (unicode) string, and file name are (sometimes) treated as bag of byte .....