8 ms·

Half of these problems are problems with the shell and have nothing to do with the filesystem. The shell is really, really, really bad about doing things like e

Half of these problems are problems with the shell and have nothing to do with the filesystem. The shell is really, really, really bad about doing things like expanding spaces in filenames or environment variables into separate arguments. Notice that every example here about how it's "wrong" is a small shell script that has unexpected behaviour. That's mostly because the shell is wrong! > Oh, and don’t display filenames. Filenames could contain control characters that control the terminal (and X-windows), causing nasty side-effects on display That's the fault of the terminal, mostly. If it's an otherwise "reasonable" name containing, say, encoded unicode, that's everyone's fault from the filesystem to the terminal to (sometimes) the shell. But the fault is spread out over more than just the fact that the filesystem lets you name files however you like. But now, Linus' quote there: > "...filesystem people should aim to make "badly written" code "just work" unless people are really really unlucky. Because like it or not, that’s what 99% of all code is... Crying that it’s an application bug is like crying over the speed of light: you should deal with reality, not what you wish reality was." — Linus Torvalds, on a slightly different topic (but I like the sentiment) (http://lwn.net/Articles/326505/ http://lwn.net/Articles/326505/) speaks to both "sides" here: there are applications in the real world that create filenames with special characters in them. The simplest most obvious of these is the space, but yes, in the real world we have applications that do this. So neither alternate reality ("restrict filesystems' names" or "keep doing what we're doing but have bug-free applications") exists, so which "reality" are you proposing that we deal with here?

overgard 13y ago
I'm sure most reasonable people would agree that (simple) unicode characters are fine, but what possible use is there for having control characters in a filename, like newline or carriage return? (Or if you're really evil, beep). If you remove "special" characters from the filesystem, you also remove a lot of complexity from other parts of the system for dealing with pathological corner cases. It's a win for everyone.
- stass 13y ago
  It's not. Just look at what limitation of such sort lead to in Windows, where each application tries to serialize an otherwise perfectly valid filename into a set of characters allowed in Windows. Of corse, majority of them never get that right. Unix is tools, not policy. It's what you make of it, and that's its beauty.
  - nmcfarl 13y ago
    I’m agree on the last point - but I’ll point out that at this point we already have policy limitations on what makes a file name. As I understand it most unixes require that they are composed of valid characters (not random bytes (or more evilly bits)) and can be no longer than a file system imposed length (commonly 255 bytes).
    - stass 13y ago
      What do you mean by valid characters? The file name usually can include any characters except the forward slash. There are, of course, filename limitiations, which are implementation specific.
      - mpyne 13y ago
        Well, any character but 0x2f and 0x00. :P
        nmcfarl 13y ago
        I was thinking that - but more than that I was thinking of FSs that allow UTF8 or UTF16 characters - which do not allow invalid code points. But from this morning's research that seems to only be NTFS and UTF16, which is not exactly a "unix" FS.
        ketralnis 13y ago
        I don't think that's strictly true. Most unix filesystems "allow" UTF-8 characters, specifically because they treat the filename only as an array of bytes and don't interpret that array at all. Perhaps NTFS does do some work to present it as UTF-16 codepoints, I don't know, but it's far from the only file system that allows this to happen.
- ketralnis 13y ago
  > I'm sure most reasonable people would agree that (simple) unicode characters are fine, What's simple? Chinese? How about vertical scripts like Mongolian? http://en.wikipedia.org/wiki/Mongolian_script http://en.wikipedia.org/wiki/Mongolian_script What about composition characters, where unicode is basically acting like a set of instructions just like a terminal control character? > what possible use is there for having control characters in a filename, like newline or carriage return? Terminal control characters are only a problem because the terminal is interpreting them due to another ancient broken paradigm, but the fact is that these filenames exist. They're present right now. You're breaking old code by changing the rules on them. Keep 'rm' from operating on them, and then I can't delete them when some stupid program goes and creates them. Keep 'vi' from operating on them and I can't look inside to see wtf caused it. That's what I meant by re-quoting Linus there. You don't get to pick the reality that you already live in, and the reality that we live in has silly messy filenames with "beep" in them. Besides, unless you go hyper-restrictive like [A-Z0-9]{1,8}, there are going to be edge-cases. Just spaces are enough to trip up bad shell scripts. Spaces are effectively control characters as far as the shell is concerned. If you allow spaces, you already have to deal with the "bad" filenames. I don't pretend to have a good answer, so I guess my point here is that these kinds of issues are more deep-seated than they appear at first. The fact that "ls" allows what's basically an injection attack on your terminal isn't ls's fault, or necessarily the terminal's fault, or the shell's fault, it's because of a whole generation of bad assumptions and "good enough" design that adds up across these tools. Just "limiting filenames" seems like a quick fix, but isn't going to fix it because that's not the root of the problem.
  - overgard 13y ago
    By simple I mean this: * Printable (space excluded) * Doesn't change the flow/layout of characters (no weird reverse characters or superscript/subscripts or characters that flow up/down). Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Which policy seems more realistic: A) sensible restrictions on file names that will please the majority of people and keep most programs working, while slightly annoying a lunatic fringe that wants EVERYTHING. B) Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? (Keeping in mind that even browsers, some of the most complicated pieces of software in existence, still tend to screw up the corner cases).
- gilgoomesh 13y ago
  Ignoring the fact that non-terminal users regularly want spaces, hyphens and newlines in their filenames since they have visual purpose... The argument in favor of newlines is the same argument in favor of spaces in filenames or filenames starting with hyphen: the filesystem should not be restricted by bash and other shell scripts continual confusion between data and instructions. We should fix/replace lazy programs, don't try to fix lazy programming by neutering the system. I think the correct action should be to design shell scripting languages that are robust about data versus instructions. The lack of robustness in shell scripting languages has been a burden since their inception. Restricting the filesystem won't stop problems with shell scripts making the same mistakes on other kinds of data.
  - overgard 13y ago
    So one solution is to replace all the popular shells from the ground up, throwing away about 30 years of history because they're "not elegant". (Ignoring the fact that almost every attempt to do this in the past has been a resounding failure). The other solution is to agree not put overly weird shit in filenames. Which seems more likely to happen in our lifetime? Which would you rather code against? I'm baffled that people are so hell bent on what "should" be allowed that they're ignoring the fact that allowing any possible character introduces exceptional complexity while offering extremely marginal benefits. I mean, imagine trying to make a gui for a filesystem browser that allows for characters that flow both up and down along with left and right. And newlines all over the place. It'd be a mess. I mean.. why? Is filename expressivity really such a major deal?
    - gilgoomesh 13y ago
      If you're looking at "30 years of history" then shells have endured "weird shit in filenames" for their entire existence and civilisation hasn't collapsed. Clearly, we don't need to change anything for them so I don't think filesystem are going to recant on their longstanding policies by making any changes to further accommodate scripting languages in our lifetimes. Meanwhile, new shells (or new versions of shells) come and go all the time. Could one of these implement a foolproof (or at least, foolresistant) way of escaping arguments? Maybe.
- hannibal5 13y ago
  This is solving the problem in the wrong place. The proper way for OS is to treat filenames as binary sequences like Linux does and have no problems. The problems that arise elsewhere should be solved elsewhere.
  - overgard 13y ago
    So instead of solving it in one place, we should solve it in hundreds of places?
    - ketralnis 13y ago
      If the problem is the shell like I assert, there are probably fewer shells than filesystems
- derleth 13y ago
  Your character set's control character is another character set's perfectly normal printable character. As for making the kernel enforce a single character set for the filesystem: No. No. No. Put another way: No. http://yarchive.net/comp/linux/utf8.html http://yarchive.net/comp/linux/utf8.html From Al Viro: Bullshit. It has _nothing_ to characters, wide or not. For system filenames are opaque. The only things that have special meanings are: octet 0x2f ('/') splits the pathname into components "." as a component has a special meaning ".." as a component has a special meaning. That's it. The rest is never interpreted by the kernel.
  - judk 13y ago
    What happens if I try to create a filename using a character set in which octect 0x2f appears as part of a multibyte character? Oops?
    - lifthrasiir 13y ago
      I think neither ISO 2022 nor UTF-16/32 have never seen use in filesystem, exactly for this reason. The closest analogue of this problem in non-POSIX system is Shift_JIS, which can contain a backslash in a multibyte character (which I regard as a stupidity, given the possibility of using FD/FE bytes instead).
      - derleth 13y ago
        > can contain a backslash in a multibyte character This is potentially confusing but, ultimately, fine. The problem is slash (byte 0x2f, in ASCII '/') in multibyte characters.
        lifthrasiir 13y ago
        In Windows backslashes are analogous to slashes in the path.
    - ninkendo 13y ago
      You can't do that. That's one of the upshots of the way Linux handles filenames. Linus has talked about this in the past... because of the fact that Linux only cares about 1 "special" character (the ASCII slash), the only sensible way to do unicode for filesystem names is to use UTF-8, which is backwards compatible with ASCII and thus obeys the kernel's expectations. It's one of those "you can have any color car so long as it's black" ultimatums... you can use whatever encoding you want, so long as it uses 0x2f for slashes, and slashes only.
ambrop7 13y ago
My thoughts exactly. The design of the unix shell is broken, and not anything else. In particular, the lack of a well supported list/array. If there was a list type, you could store filenames in a list, and there would be no reason for the IFS stuff. And by well-supported, I mean you could easily encode a list unambiguously to stdout, and decode it on the other end. I know about bash arrays, but those are neither well-supported nor standard. For example you can't easily return an array from a bash function, which is a major problem.
Flimm 13y ago
Leading dashes in filenames are not a problem caused by the shell but a problem caused by the way most binaries interpret argv.
einhverfr 13y ago
I don't think it is just the shell though. As soon as you start trying to do any automatic interchange of filesystem metadata, you are going to run into the same problems. Namely you have to have semantically important sets of bytes, and filesystems which allow arbitrary binary strings can insert semantic data into your exchange system, which necessitates extra escaping on all sides. So the question is what assumptions we have, and how those should be enforced. Here are some humble recommendations. 1. Disable non-printing characters in filenames (this is encoding dependent and assumes a kernel aware of encoding). 2. Allow system administrators to configure additional rules, like "no leading/training whitespace, no starting with a hyphen, no internal tabs, no SGML special characters, UTF-8 only" and the like. This would empower application developers to state what assumptions they rely on and eventual standards to emerge.

overgard 13y ago

I'm sure most reasonable people would agree that (simple) unicode characters are fine, but what possible use is there for having control characters in a filename, like newline or carriage return? (Or if you're really evil, beep). If you remove "special" characters from the filesystem, you also remove a lot of complexity from other parts of the system for dealing with pathological corner cases. It's a win for everyone.

stass 13y ago
It's not. Just look at what limitation of such sort lead to in Windows, where each application tries to serialize an otherwise perfectly valid filename into a set of characters allowed in Windows. Of corse, majority of them never get that right. Unix is tools, not policy. It's what you make of it, and that's its beauty.
- nmcfarl 13y ago
  I’m agree on the last point - but I’ll point out that at this point we already have policy limitations on what makes a file name. As I understand it most unixes require that they are composed of valid characters (not random bytes (or more evilly bits)) and can be no longer than a file system imposed length (commonly 255 bytes).
  - stass 13y ago
    What do you mean by valid characters? The file name usually can include any characters except the forward slash. There are, of course, filename limitiations, which are implementation specific.
    - mpyne 13y ago
      Well, any character but 0x2f and 0x00. :P
      - nmcfarl 13y ago
        I was thinking that - but more than that I was thinking of FSs that allow UTF8 or UTF16 characters - which do not allow invalid code points. But from this morning's research that seems to only be NTFS and UTF16, which is not exactly a "unix" FS.
        ketralnis 13y ago
        I don't think that's strictly true. Most unix filesystems "allow" UTF-8 characters, specifically because they treat the filename only as an array of bytes and don't interpret that array at all. Perhaps NTFS does do some work to present it as UTF-16 codepoints, I don't know, but it's far from the only file system that allows this to happen.
ketralnis 13y ago
> I'm sure most reasonable people would agree that (simple) unicode characters are fine, What's simple? Chinese? How about vertical scripts like Mongolian? http://en.wikipedia.org/wiki/Mongolian_script http://en.wikipedia.org/wiki/Mongolian_script What about composition characters, where unicode is basically acting like a set of instructions just like a terminal control character? > what possible use is there for having control characters in a filename, like newline or carriage return? Terminal control characters are only a problem because the terminal is interpreting them due to another ancient broken paradigm, but the fact is that these filenames exist. They're present right now. You're breaking old code by changing the rules on them. Keep 'rm' from operating on them, and then I can't delete them when some stupid program goes and creates them. Keep 'vi' from operating on them and I can't look inside to see wtf caused it. That's what I meant by re-quoting Linus there. You don't get to pick the reality that you already live in, and the reality that we live in has silly messy filenames with "beep" in them. Besides, unless you go hyper-restrictive like [A-Z0-9]{1,8}, there are going to be edge-cases. Just spaces are enough to trip up bad shell scripts. Spaces are effectively control characters as far as the shell is concerned. If you allow spaces, you already have to deal with the "bad" filenames. I don't pretend to have a good answer, so I guess my point here is that these kinds of issues are more deep-seated than they appear at first. The fact that "ls" allows what's basically an injection attack on your terminal isn't ls's fault, or necessarily the terminal's fault, or the shell's fault, it's because of a whole generation of bad assumptions and "good enough" design that adds up across these tools. Just "limiting filenames" seems like a quick fix, but isn't going to fix it because that's not the root of the problem.
- overgard 13y ago
  By simple I mean this: * Printable (space excluded) * Doesn't change the flow/layout of characters (no weird reverse characters or superscript/subscripts or characters that flow up/down). Will that make everyone happy? Of course not. But it's pretty much "expressive enough" and it makes dealing with filenames 10x simpler. Which policy seems more realistic: A) sensible restrictions on file names that will please the majority of people and keep most programs working, while slightly annoying a lunatic fringe that wants EVERYTHING. B) Require that every program that handles files in some capacity can handle every single possible weird unicode and control code subtlety in existence? (Keeping in mind that even browsers, some of the most complicated pieces of software in existence, still tend to screw up the corner cases).
gilgoomesh 13y ago
Ignoring the fact that non-terminal users regularly want spaces, hyphens and newlines in their filenames since they have visual purpose... The argument in favor of newlines is the same argument in favor of spaces in filenames or filenames starting with hyphen: the filesystem should not be restricted by bash and other shell scripts continual confusion between data and instructions. We should fix/replace lazy programs, don't try to fix lazy programming by neutering the system. I think the correct action should be to design shell scripting languages that are robust about data versus instructions. The lack of robustness in shell scripting languages has been a burden since their inception. Restricting the filesystem won't stop problems with shell scripts making the same mistakes on other kinds of data.
- overgard 13y ago
  So one solution is to replace all the popular shells from the ground up, throwing away about 30 years of history because they're "not elegant". (Ignoring the fact that almost every attempt to do this in the past has been a resounding failure). The other solution is to agree not put overly weird shit in filenames. Which seems more likely to happen in our lifetime? Which would you rather code against? I'm baffled that people are so hell bent on what "should" be allowed that they're ignoring the fact that allowing any possible character introduces exceptional complexity while offering extremely marginal benefits. I mean, imagine trying to make a gui for a filesystem browser that allows for characters that flow both up and down along with left and right. And newlines all over the place. It'd be a mess. I mean.. why? Is filename expressivity really such a major deal?
  - gilgoomesh 13y ago
    If you're looking at "30 years of history" then shells have endured "weird shit in filenames" for their entire existence and civilisation hasn't collapsed. Clearly, we don't need to change anything for them so I don't think filesystem are going to recant on their longstanding policies by making any changes to further accommodate scripting languages in our lifetimes. Meanwhile, new shells (or new versions of shells) come and go all the time. Could one of these implement a foolproof (or at least, foolresistant) way of escaping arguments? Maybe.
hannibal5 13y ago
This is solving the problem in the wrong place. The proper way for OS is to treat filenames as binary sequences like Linux does and have no problems. The problems that arise elsewhere should be solved elsewhere.
- overgard 13y ago
  So instead of solving it in one place, we should solve it in hundreds of places?
  - ketralnis 13y ago
    If the problem is the shell like I assert, there are probably fewer shells than filesystems
derleth 13y ago
Your character set's control character is another character set's perfectly normal printable character. As for making the kernel enforce a single character set for the filesystem: No. No. No. Put another way: No. http://yarchive.net/comp/linux/utf8.html http://yarchive.net/comp/linux/utf8.html From Al Viro: Bullshit. It has _nothing_ to characters, wide or not. For system filenames are opaque. The only things that have special meanings are: octet 0x2f ('/') splits the pathname into components "." as a component has a special meaning ".." as a component has a special meaning. That's it. The rest is never interpreted by the kernel.
- judk 13y ago
  What happens if I try to create a filename using a character set in which octect 0x2f appears as part of a multibyte character? Oops?
  - lifthrasiir 13y ago
    I think neither ISO 2022 nor UTF-16/32 have never seen use in filesystem, exactly for this reason. The closest analogue of this problem in non-POSIX system is Shift_JIS, which can contain a backslash in a multibyte character (which I regard as a stupidity, given the possibility of using FD/FE bytes instead).
    - derleth 13y ago
      > can contain a backslash in a multibyte character This is potentially confusing but, ultimately, fine. The problem is slash (byte 0x2f, in ASCII '/') in multibyte characters.
      - lifthrasiir 13y ago
        In Windows backslashes are analogous to slashes in the path.
  - ninkendo 13y ago
    You can't do that. That's one of the upshots of the way Linux handles filenames. Linus has talked about this in the past... because of the fact that Linux only cares about 1 "special" character (the ASCII slash), the only sensible way to do unicode for filesystem names is to use UTF-8, which is backwards compatible with ASCII and thus obeys the kernel's expectations. It's one of those "you can have any color car so long as it's black" ultimatums... you can use whatever encoding you want, so long as it uses 0x2f for slashes, and slashes only.

ambrop7 13y ago

My thoughts exactly. The design of the unix shell is broken, and not anything else. In particular, the lack of a well supported list/array. If there was a list type, you could store filenames in a list, and there would be no reason for the IFS stuff. And by well-supported, I mean you could easily encode a list unambiguously to stdout, and decode it on the other end. I know about bash arrays, but those are neither well-supported nor standard. For example you can't easily return an array from a bash function, which is a major problem.

Flimm 13y ago

Leading dashes in filenames are not a problem caused by the shell but a problem caused by the way most binaries interpret argv.

einhverfr 13y ago

I don't think it is just the shell though. As soon as you start trying to do any automatic interchange of filesystem metadata, you are going to run into the same problems. Namely you have to have semantically important sets of bytes, and filesystems which allow arbitrary binary strings can insert semantic data into your exchange system, which necessitates extra escaping on all sides. So the question is what assumptions we have, and how those should be enforced. Here are some humble recommendations. 1. Disable non-printing characters in filenames (this is encoding dependent and assumes a kernel aware of encoding). 2. Allow system administrators to configure additional rules, like "no leading/training whitespace, no starting with a hyphen, no internal tabs, no SGML special characters, UTF-8 only" and the like. This would empower application developers to state what assumptions they rely on and eventual standards to emerge.