3 ms·

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interop

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

wtetzner 7y ago
It sounds like they're saying the opposite. All programs dealing with filenames need to be able to support an arbitrary stream of bytes, they can't just assume UTF-8.
masklinn 7y ago
> 2) Bye bye backward compatibility and interoperability It's already not really a thing. Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f, and I think HFS+ requires valid UTF-16 and allows everything (including NUL). The OS then adds its own limitations e.g. win32 forbids \, :, *, ", ?, <, >, | (as well as a few special names I think) and OSX forbids 0000 and 003a (":"), the latter of which gets converted to and from "/" (and similarly forbidden) by the POSIX compatibility layer. The latter is really weird to see in action, if you have access to an OSX machine: open a terminal, try to create a file called "/" and it'll fail. Now create one called ":". Switch over to the Finder, and you'll see that that file is now called "/" (and creating a file called ":" fails). Oh yeah and ZFS doesn't really care but can require that all paths be valid UTF8 (by setting the utf8only flag).
- account42 7y ago
  > Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f. For just Windows -> Linux you can represent everything by mapping WTF-16 to WTF-8.
ghettoimp 7y ago
Backward compatibility is a laudable goal and is not to be broken lightly. But sometimes, things are so fundamentally broken that we would be far better off with a clean break. Interoperability is quite possibly a good argument for coming up with some reasonable restrictions on filenames. Today you could easily (case sensitive names, special characters, etc.) create a ZIP file or similar that cannot be successfully extracted on this platform or that. In an excellent article, David A. Wheeler [1] lays out a compelling case against the status quo. TL;DR: bad filenames are too hard to handle correctly. Programs, standards, and operating systems already assume there are no bad filenames. Your programs will fail in numerous ways when they encounter bad filenames. Some of these failures are security problems. He concludes: "In sum: It’d be far better if filenames were more limited so that they would be safer and easier to use. This would eliminate a whole class of errors and vulnerabilities in programs that “look correct” but subtly fail when unusual filenames are created (possibly by attackers)." He goes on to consider many ideas towards getting to this goal. [1] https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html
- deleted 7y ago
  [deleted]
malkia 7y ago
1) Nope. 2) Yes, we need to keep backward compatibility. What I'm saying is that promoting UTF8 everywhere, without specifically stressing the fact that filesystems (in general) do no observe UFT8, leads to API/LIB designs that lack good support there. Path/filename/dirname/whatever should be a different kind of "string".

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

wtetzner 7y ago

It sounds like they're saying the opposite. All programs dealing with filenames need to be able to support an arbitrary stream of bytes, they can't just assume UTF-8.

masklinn 7y ago

> 2) Bye bye backward compatibility and interoperability It's already not really a thing. Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f, and I think HFS+ requires valid UTF-16 and allows everything (including NUL). The OS then adds its own limitations e.g. win32 forbids \, :, *, ", ?, <, >, | (as well as a few special names I think) and OSX forbids 0000 and 003a (":"), the latter of which gets converted to and from "/" (and similarly forbidden) by the POSIX compatibility layer. The latter is really weird to see in action, if you have access to an OSX machine: open a terminal, try to create a file called "/" and it'll fail. Now create one called ":". Switch over to the Finder, and you'll see that that file is now called "/" (and creating a file called ":" fails). Oh yeah and ZFS doesn't really care but can require that all paths be valid UTF8 (by setting the utf8only flag).

account42 7y ago
> Traditional unices allow arbitrary bytes with the exception of 00 and 2f, NTFS allows arbitrary utf-16 code units (including unpaired surrogates) with the exception of 0000 and 002f. For just Windows -> Linux you can represent everything by mapping WTF-16 to WTF-8.

ghettoimp 7y ago

Backward compatibility is a laudable goal and is not to be broken lightly. But sometimes, things are so fundamentally broken that we would be far better off with a clean break. Interoperability is quite possibly a good argument for coming up with some reasonable restrictions on filenames. Today you could easily (case sensitive names, special characters, etc.) create a ZIP file or similar that cannot be successfully extracted on this platform or that. In an excellent article, David A. Wheeler [1] lays out a compelling case against the status quo. TL;DR: bad filenames are too hard to handle correctly. Programs, standards, and operating systems already assume there are no bad filenames. Your programs will fail in numerous ways when they encounter bad filenames. Some of these failures are security problems. He concludes: "In sum: It’d be far better if filenames were more limited so that they would be safer and easier to use. This would eliminate a whole class of errors and vulnerabilities in programs that “look correct” but subtly fail when unusual filenames are created (possibly by attackers)." He goes on to consider many ideas towards getting to this goal. [1] https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html

deleted 7y ago
[deleted]

malkia 7y ago

1) Nope. 2) Yes, we need to keep backward compatibility. What I'm saying is that promoting UTF8 everywhere, without specifically stressing the fact that filesystems (in general) do no observe UFT8, leads to API/LIB designs that lack good support there. Path/filename/dirname/whatever should be a different kind of "string".