6 ms·

> I'm sure most reasonable people would agree that (simple) unicode characters are fine, What's simple? Chinese? How about vertical scripts like Mongolian? htt

> I'm sure most reasonable people would agree that (simple) unicode characters are fine, What's simple? Chinese? How about vertical scripts like Mongolian? http://en.wikipedia.org/wiki/Mongolian_script http://en.wikipedia.org/wiki/Mongolian_script What about composition characters, where unicode is basically acting like a set of instructions just like a terminal control character? > what possible use is there for having control characters in a filename, like newline or carriage return? Terminal control characters are only a problem because the terminal is interpreting them due to another ancient broken paradigm, but the fact is that these filenames exist. They're present right now. You're breaking old code by changing the rules on them. Keep 'rm' from operating on them, and then I can't delete them when some stupid program goes and creates them. Keep 'vi' from operating on them and I can't look inside to see wtf caused it. That's what I meant by re-quoting Linus there. You don't get to pick the reality that you already live in, and the reality that we live in has silly messy filenames with "beep" in them. Besides, unless you go hyper-restrictive like [A-Z0-9]{1,8}, there are going to be edge-cases. Just spaces are enough to trip up bad shell scripts. Spaces are effectively control characters as far as the shell is concerned. If you allow spaces, you already have to deal with the "bad" filenames. I don't pretend to have a good answer, so I guess my point here is that these kinds of issues are more deep-seated than they appear at first. The fact that "ls" allows what's basically an injection attack on your terminal isn't ls's fault, or necessarily the terminal's fault, or the shell's fault, it's because of a whole generation of bad assumptions and "good enough" design that adds up across these tools. Just "limiting filenames" seems like a quick fix, but isn't going to fix it because that's not the root of the problem.

overgard 13y ago
By simple I mean this: * Printable (space excluded) * Doesn't change the flow/layout of characters (no weird reverse characters or superscript/subscripts or characters that flow up/down). Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Which policy seems more realistic: A) sensible restrictions on file names that will please the majority of people and keep most programs working, while slightly annoying a lunatic fringe that wants EVERYTHING. B) Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? (Keeping in mind that even browsers, some of the most complicated pieces of software in existence, still tend to screw up the corner cases).
- toast0 13y ago
  If you're going to whitelist characters from unicode, we're going to have several problems with managing the whitelist: a) overhead for the whitelist is going to be an issue on some systems b) I'm going to need a kernel update every time there's a new useful code point for example, the Indian Rupee Sign http://www.fileformat.info/info/unicode/char/20b9/index.htm http://www.fileformat.info/info/unicode/char/20b9/index.htm
  - dwheeler 13y ago
    You can just check "is it UTF-8 encoded?". Then it doesn't matter when a new code point is added, it would still be correctly encoded. You could make this a configurable kernel option; when enabled, filenames can only be created when they encode with UTF-8. The next big problem is control characters. But really, the only people who need control characters embedded in filenames are people breaking into computers using them. It doesn't matter if you use a GUI or a CUI, people generally don't embed newlines, or escape, or tabs in filenames. The leading "-" is probably more controversial, but the POSIX standard specifically notes such filenames as nonstandard.
- ketralnis 13y ago
  > Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Simple enough for a Western language speaker, sure. Pretty inhibiting for a speaker of Kannada that they can't name their résumé file with their own name. > Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? I assert again that once you allow spaces, you're already in the situation that you need to treat filenames as sequences of bytes that need quoting. Also, I want to point out again that almost all of these problems are only when writing shell scripts. C programs or e.g. GUI applications with file pickers tend (with exceptions of course) to not have these problems since you get the filenames into them.
  - overgard 13y ago
    Ok so fine, we allow people to write filenames top-to-bottom. So programs that show the filename in the title bar: how do they display it? Really big title bar? Align the title bar on the left or right? What if I mix english and Kannada? What then? Or what if I start with an english word in the terminal, and then switch to kannada, what's the terminal supposed to do? I haven't used a lot of asian language systems, but I'm going to guess two things that I'm pretty sure are true: A) they're already implicitly dealing with these restrictions, whether the file system enforces it or not B) they probably already have workarounds to deal with these cases anyway. (alternate scripts or whatever). Regardless of culture, I'm going to make what I think is a pretty obvious statement: good filenames are easy to type and unambiguous. If your filename is neither of those, it's not a good filename. The point of a filename isn't to be as expressive as possible, it's to be universal, descriptive, and easy to work with. None of that requires special characters. With regard to the space thing, I think it's overblown. Spaces are common enough that good software engineers think to handle it. Newlines in files? I've /never/ seen that, even though it's possible. Also, space doesn't require an escape character. Once you start getting to newlines, it does. Once you start introducing escape characters into it, it's an entirely new layer of complexity.
rat87 13y ago
I think tools should ideally not create strange filenames without forcing(and for some tools as gui tools should be missing the force override). Ideally they should still be able to view and delete them. If you a file with that type of name and edit and attempt to save it, the tool should offer to rename it(deleting the version with the "bad name") by default but allow an override to save with the old bad name.
Flimm 13y ago
The fact is, we already support filesystems on UNIX with restricted filenames (eg: FAT and NTFS). If you ban terrible filenames from the filesystem, those filenames would not exist because they could never be created in the first place.
- jlgreco 13y ago
  Yeah, but we don't have a time machine, so we don't get to pick the reality where all UNIX-native filesystems restricted filenames like FAT. This isn't something that you can retroactively change, so we are stuck on this reality and need to find ways to cope.

overgard 13y ago

By simple I mean this: * Printable (space excluded) * Doesn't change the flow/layout of characters (no weird reverse characters or superscript/subscripts or characters that flow up/down). Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Which policy seems more realistic: A) sensible restrictions on file names that will please the majority of people and keep most programs working, while slightly annoying a lunatic fringe that wants EVERYTHING. B) Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? (Keeping in mind that even browsers, some of the most complicated pieces of software in existence, still tend to screw up the corner cases).

toast0 13y ago
If you're going to whitelist characters from unicode, we're going to have several problems with managing the whitelist: a) overhead for the whitelist is going to be an issue on some systems b) I'm going to need a kernel update every time there's a new useful code point for example, the Indian Rupee Sign http://www.fileformat.info/info/unicode/char/20b9/index.htm http://www.fileformat.info/info/unicode/char/20b9/index.htm
- dwheeler 13y ago
  You can just check "is it UTF-8 encoded?". Then it doesn't matter when a new code point is added, it would still be correctly encoded. You could make this a configurable kernel option; when enabled, filenames can only be created when they encode with UTF-8. The next big problem is control characters. But really, the only people who need control characters embedded in filenames are people breaking into computers using them. It doesn't matter if you use a GUI or a CUI, people generally don't embed newlines, or escape, or tabs in filenames. The leading "-" is probably more controversial, but the POSIX standard specifically notes such filenames as nonstandard.
ketralnis 13y ago
> Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Simple enough for a Western language speaker, sure. Pretty inhibiting for a speaker of Kannada that they can't name their résumé file with their own name. > Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? I assert again that once you allow spaces, you're already in the situation that you need to treat filenames as sequences of bytes that need quoting. Also, I want to point out again that almost all of these problems are only when writing shell scripts. C programs or e.g. GUI applications with file pickers tend (with exceptions of course) to not have these problems since you get the filenames into them.
- overgard 13y ago
  Ok so fine, we allow people to write filenames top-to-bottom. So programs that show the filename in the title bar: how do they display it? Really big title bar? Align the title bar on the left or right? What if I mix english and Kannada? What then? Or what if I start with an english word in the terminal, and then switch to kannada, what's the terminal supposed to do? I haven't used a lot of asian language systems, but I'm going to guess two things that I'm pretty sure are true: A) they're already implicitly dealing with these restrictions, whether the file system enforces it or not B) they probably already have workarounds to deal with these cases anyway. (alternate scripts or whatever). Regardless of culture, I'm going to make what I think is a pretty obvious statement: good filenames are easy to type and unambiguous. If your filename is neither of those, it's not a good filename. The point of a filename isn't to be as expressive as possible, it's to be universal, descriptive, and easy to work with. None of that requires special characters. With regard to the space thing, I think it's overblown. Spaces are common enough that good software engineers think to handle it. Newlines in files? I've /never/ seen that, even though it's possible. Also, space doesn't require an escape character. Once you start getting to newlines, it does. Once you start introducing escape characters into it, it's an entirely new layer of complexity.

rat87 13y ago

I think tools should ideally not create strange filenames without forcing(and for some tools as gui tools should be missing the force override). Ideally they should still be able to view and delete them. If you a file with that type of name and edit and attempt to save it, the tool should offer to rename it(deleting the version with the "bad name") by default but allow an override to save with the old bad name.

Flimm 13y ago

The fact is, we already support filesystems on UNIX with restricted filenames (eg: FAT and NTFS). If you ban terrible filenames from the filesystem, those filenames would not exist because they could never be created in the first place.

jlgreco 13y ago
Yeah, but we don't have a time machine, so we don't get to pick the reality where all UNIX-native filesystems restricted filenames like FAT. This isn't something that you can retroactively change, so we are stuck on this reality and need to find ways to cope.