4 ms·

The problem is that the filenames are interpreted by computers, not by people. Your comment that filenames are "human-readable strings" is subtly changing the

by adekok 14y ago

The problem is that the filenames are interpreted by computers, not by people. Your comment that filenames are "human-readable strings" is subtly changing the problem statement. You're conflating bits (interpreted by computers) with display (interpreted by humans). They really are two fundamentally different things. I've been working though similar i18n issues in the IETF. Various protocols started off as ASCII, and now need to support more. There have been 5+ years of heated discussion around the subject. The consensus appears to be reaching the view that issues related to people are solely managed by the "human IO" portions of the computer. i.e. the display turns bits into glyphs, and the input turns keystrokes into normalized buckets of bits. Once the bits reach the rest of the system, everything else treats them as buckets of bits. This approach means that the non-IO parts of the system are simple. Things like name comparisons are done via "memcmp". No i18n issues apply. I'll go a little more extreme here, and say that IMHO this is the only approach which makes sense. Anything else is madness. Anything else means that your file system handles i18n.

eridius 14y ago
You cannot simply put a "human" layer on top of the filesystem and expect it to work correctly or efficiently. The most obvious case is, if I create two filenames that are identical except for normalization, how can the "human" layer possibly handle this? Heck, even with just one filename it's trivial to produce braindead results. Assuming keyboard input is decomposed (I honestly forget), if I have a filename using composed characters, and I want to type this name into my terminal, I can't. And that's completely crazy to me as someone who works on user-facing software.
- adekok 14y ago
  > if I create two filenames that are identical except for normalization, how can the "human" layer possibly handle this? The "human IO" layer is responsible for normalization of strings sent to the rest of the system. Therefore, your scenario cannot occur in normal operations. If it does occur, too bad. Some part of the system allowed an illegal operation, and you now have two filenames which may display as identical, but which are stored as different "buckets of bits". Your other scenario is the same. The issue of composed versus non-composed characters doesn't come up, as all input is normalized. If your human input layer can't normalize it, then tough. The system is broken. What happens when you create a filename on a "new" system, and then mount it on an "old" system? What happens when you add new code points to unicode? What happens when you create new composition rules? In your design, the answer is "oh, crap". The older file systems will make different decisions about name comparisons than a new file system. This leads to exactly the inconsistencies you're worried about. In contrast, the IETF design means all file systems always make the same decisions. "memcmp" always returns the same results. Old systems cannot create file names using new code points, because their "display IO" layer forbids it. Again, if such a string are created, then the system created a string it can't understand. The system is broken. It seems you have strong opinions here. These issues are being worked through in all internet protocols. The answer is not "fix every protocol to do string normalization and/or i18n". Wait 5 years. You'll see this design take over every internet protocol, and (IMHO) every new file system. Nothing else is workable.
  - eridius 14y ago
    You're not making much sense. If all filenames that make it to the filesystem have passed through this layer, then this layer is effectively part of the filesystem. And since you say it shouldn't be part of the filesystem, then this must not be what you're trying to say. If only filenames that are shown to/input by the human go through this layer, then you're fucked. You open the door to having all sorts of nasty filenames that the human cannot distinguish between, or that the human cannot even type. This is not good. And I really don't get your point about new codepoints. Unicode is not going to introduce new composed codepoints. The set of composed codepoints that exist now is the same as the set of composed codepoints that will exist in 5 years. It really seems like you're arguing for massive extra complexity that introduces all sorts of really bad edge cases simply so you can say "my filesystem preserves filenames", without actually getting any benefit of that because normal interaction is going to be normalizing those filenames anyway.

by adekok 14y ago

eridius 14y ago

You cannot simply put a "human" layer on top of the filesystem and expect it to work correctly or efficiently. The most obvious case is, if I create two filenames that are identical except for normalization, how can the "human" layer possibly handle this? Heck, even with just one filename it's trivial to produce braindead results. Assuming keyboard input is decomposed (I honestly forget), if I have a filename using composed characters, and I want to type this name into my terminal, I can't. And that's completely crazy to me as someone who works on user-facing software.

adekok 14y ago
> if I create two filenames that are identical except for normalization, how can the "human" layer possibly handle this? The "human IO" layer is responsible for normalization of strings sent to the rest of the system. Therefore, your scenario cannot occur in normal operations. If it does occur, too bad. Some part of the system allowed an illegal operation, and you now have two filenames which may display as identical, but which are stored as different "buckets of bits". Your other scenario is the same. The issue of composed versus non-composed characters doesn't come up, as all input is normalized. If your human input layer can't normalize it, then tough. The system is broken. What happens when you create a filename on a "new" system, and then mount it on an "old" system? What happens when you add new code points to unicode? What happens when you create new composition rules? In your design, the answer is "oh, crap". The older file systems will make different decisions about name comparisons than a new file system. This leads to exactly the inconsistencies you're worried about. In contrast, the IETF design means all file systems always make the same decisions. "memcmp" always returns the same results. Old systems cannot create file names using new code points, because their "display IO" layer forbids it. Again, if such a string are created, then the system created a string it can't understand. The system is broken. It seems you have strong opinions here. These issues are being worked through in all internet protocols. The answer is not "fix every protocol to do string normalization and/or i18n". Wait 5 years. You'll see this design take over every internet protocol, and (IMHO) every new file system. Nothing else is workable.
- eridius 14y ago
  You're not making much sense. If all filenames that make it to the filesystem have passed through this layer, then this layer is effectively part of the filesystem. And since you say it shouldn't be part of the filesystem, then this must not be what you're trying to say. If only filenames that are shown to/input by the human go through this layer, then you're fucked. You open the door to having all sorts of nasty filenames that the human cannot distinguish between, or that the human cannot even type. This is not good. And I really don't get your point about new codepoints. Unicode is not going to introduce new composed codepoints. The set of composed codepoints that exist now is the same as the set of composed codepoints that will exist in 5 years. It really seems like you're arguing for massive extra complexity that introduces all sorts of really bad edge cases simply so you can say "my filesystem preserves filenames", without actually getting any benefit of that because normal interaction is going to be normalizing those filenames anyway.