7 ms·

> There are still many cases where you need to execute external programs. But you don't need to call the shell to execute external programs.

dwheeler 8y ago
Exactly. Even if you didn't have a shell installed, these file names would be a problem. If a file name begins with the dash, and you naively pass that file name as argument one, you are still in trouble, even if you do not have a shell installed on your system. What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That is because leading dashes are allowable as file names. It is possible to workaround this, for example by pre pending "./". But our goal should be to make things easy, not just make it possible to do things correctly.
- pdkl95 8y ago
  > It is possible to workaround this, for example by pre pending "./". Or you could use the solutio9n provided by most argument parsers: the "--" option that disables option parsing for the remaining options. Trying to fix it yourself by "fixing" filenames (by prefixing with "./" or whatever) requires you to enumerate all possible problem[1] or you're just creating a new weird machine[2]. You're worrying a lot about specific problems like if a filename might be confused with an option, but there are many other ways filenames can be a problem *even if their names are limited to "simple"[3] alphanumeric characters. If you're accepting input from the user (or worse, over a network), you will have far worse problem than crashing on a bad filename if you aren't strictly validating and parsing that input before sending it to any other program or using it as a filename. [1] ("#2) Enumerating Badness") http://www.ranum.com/security/computer_security/editorials/dumb/ http://www.ranum.com/security/computer_security/editorials/d... [2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit... [3] Defining concepts like "alphanumeric" or even "codepoints that can be safely [printed to the screen" are NOT simple co9ncepts if you support Unicode (which most programs should support).
  - pmoriarty 8y ago
    The article addresses your suggestion: "The "obvious" way to do this is to litter command invocations with "--" before the filename(s). But it turns out this doesn't really work, because not all commands support "--" (ugh!). For example, the widely-used "echo" command is not required to support "--". What's worse, echo does support at least one dash option, so we need to escape leading-dash values somehow. POSIX recommends that you use printf(1) instead of echo(1), but some old systems do not include printf(1)."
    - pdkl95 8y ago
      > But it turns out this doesn't really work, because not all commands support "--" So what? If you don't read the documentation for your the programs you run, mistaking filenames for an option is only only one of the many types of problem you might have. The problem of using programs incorrectly cannot be solved by changing filenames. You solve that type of problem by carefully understanding how to run each program you want to use (which means using -- for many (but not all) programs) and strictly validating & sanitizing your input.
      - faho 8y ago
        >The problem of using programs incorrectly cannot be solved by changing filenames If changing filenames makes an incorrect solution work, then it solves it, in that case. ---- In this thread, you're missing the point somewhat - the problem isn't that it's _impossible_ to work with all sorts of filenames using shell, the problem is that it's a PITA to do so, even for experienced users but especially for newbies. I.e. sure I can use `"$@"` everywhere and quote every single variable I ever use, and prefix every path with `./` or use `--` for every command that supports it. What if I didn't have to? Wouldn't that be better?
        pdkl95 8y ago
        > In this thread, you're missing the point somewhat I assure you I'm not; I'm trying to explain why worrying about these problems at the filename level doesn't actually solve the problem, which should be handled at a different level of abstraction. > sure I can use `"$@"` everywhere and quote every single variable If you are writing Bourne shell without doing that, you are always going to have problems, because arguments are not always filenames. Yes, the syntax can be slightly annoying for historical/compatibility reasons. > What if I didn't have to? You don't. Use a different shell, or write your scripts in one of the many languages that are widely available like ruby, python, etc. If a scrip0t has #!/bin/sh at the top, you can probably rewrite it into any language you prefer.
- alerighi 8y ago
  A more simple and clean solution, that I always use in my python scripts: always work with absolute paths, that starts with /, and you solve all these problems. No reason to have relative paths outside the shell for me, relative paths are only a convenience for the shell user that doesn't have to specify the full path as an argument, but if you invoke a program from another program, it doesn't cost you nothing to pass to it as an argument the full path.
  - deleted 8y ago
    [deleted]
  - cesarb 8y ago
    There are situations (most of them involving symbolic links) in which using absolute paths can change the meaning of the path. The most obvious is when creating a symbolic link, for instance a relative symbolic link from "README" to "README.md" will not break when the whole directory is moved. A more worrying one is that using absolute paths everywhere is vulnerable to TOCTOU issues, while relative paths can be less vulnerable (and can be combined with newer APIs like openat to reduce these issues further).
- zbentley 8y ago
  > What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That's true, and a very real problem in many cases. A lot of those cases, though, are because people use positional arguments when they should use switched options ('--foo=bar' or '--foo bar') instead. This shields them from all kinds of pretty severe input-validation errors, and not just with filenames. It's a few more characters to type, but a great many option-parsing libs in all sorts of languages can automatically generate terse ('--foo=bar' can be specified as '-f bar') I know that puts some of the onus on developers, but as you and others have pointed out, relying on the shell to handle these weird cases likely puts responsibility where it doesn't belong (calling subprocesses from another language shouldn't use the shell unless it's overwhelmingly necessary) and on something that handles it poorly (all your points about quoting hell). That's a narrow quibble, and is exclusively concerned with getting data in to invoked programs intact, from a non-shell invoker. All of your other points about encoding, validation, shell-altering STDOUT, and broken behavior when you are working in the shell (like the whitespace-related issues you point out, which are still a hassle to work around even when using switched options) are well put and very well taken.
swiley 8y ago
If the user is the one entering the program name not calling the shell can cause unexpected behavior.
- zbentley 8y ago
  How so? The only cases in which I've seen this cause problems are programs which rely on the presence of a tty when they should not, or assume the existence of environment variables that they don't really need.
  - swiley 8y ago
    Neither of those really have anything to do with the shell? And it's not like it's all impossible to do yourself without the shell there's just a lot and the shell does it in a nice and well known way. Here are some examples: Searching the path Setting custom environment variables builtins One weird one that I feel like a lot of people don't know about is that you can run shell scripts from the shell without the shebang. But if you exec one you'll get a format error. It's not good to rely on this but it's another thing that surprises people with messy scripts.

dwheeler 8y ago

Exactly. Even if you didn't have a shell installed, these file names would be a problem. If a file name begins with the dash, and you naively pass that file name as argument one, you are still in trouble, even if you do not have a shell installed on your system. What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That is because leading dashes are allowable as file names. It is possible to workaround this, for example by pre pending "./". But our goal should be to make things easy, not just make it possible to do things correctly.

pdkl95 8y ago
> It is possible to workaround this, for example by pre pending "./". Or you could use the solutio9n provided by most argument parsers: the "--" option that disables option parsing for the remaining options. Trying to fix it yourself by "fixing" filenames (by prefixing with "./" or whatever) requires you to enumerate all possible problem[1] or you're just creating a new weird machine[2]. You're worrying a lot about specific problems like if a filename might be confused with an option, but there are many other ways filenames can be a problem *even if their names are limited to "simple"[3] alphanumeric characters. If you're accepting input from the user (or worse, over a network), you will have far worse problem than crashing on a bad filename if you aren't strictly validating and parsing that input before sending it to any other program or using it as a filename. [1] ("#2) Enumerating Badness") http://www.ranum.com/security/computer_security/editorials/dumb/ http://www.ranum.com/security/computer_security/editorials/d... [2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit... [3] Defining concepts like "alphanumeric" or even "codepoints that can be safely [printed to the screen" are NOT simple co9ncepts if you support Unicode (which most programs should support).
- pmoriarty 8y ago
  The article addresses your suggestion: "The "obvious" way to do this is to litter command invocations with "--" before the filename(s). But it turns out this doesn't really work, because not all commands support "--" (ugh!). For example, the widely-used "echo" command is not required to support "--". What's worse, echo does support at least one dash option, so we need to escape leading-dash values somehow. POSIX recommends that you use printf(1) instead of echo(1), but some old systems do not include printf(1)."
  - pdkl95 8y ago
    > But it turns out this doesn't really work, because not all commands support "--" So what? If you don't read the documentation for your the programs you run, mistaking filenames for an option is only only one of the many types of problem you might have. The problem of using programs incorrectly cannot be solved by changing filenames. You solve that type of problem by carefully understanding how to run each program you want to use (which means using -- for many (but not all) programs) and strictly validating & sanitizing your input.
    - faho 8y ago
      >The problem of using programs incorrectly cannot be solved by changing filenames If changing filenames makes an incorrect solution work, then it solves it, in that case. ---- In this thread, you're missing the point somewhat - the problem isn't that it's _impossible_ to work with all sorts of filenames using shell, the problem is that it's a PITA to do so, even for experienced users but especially for newbies. I.e. sure I can use `"$@"` everywhere and quote every single variable I ever use, and prefix every path with `./` or use `--` for every command that supports it. What if I didn't have to? Wouldn't that be better?
      - pdkl95 8y ago
        > In this thread, you're missing the point somewhat I assure you I'm not; I'm trying to explain why worrying about these problems at the filename level doesn't actually solve the problem, which should be handled at a different level of abstraction. > sure I can use `"$@"` everywhere and quote every single variable If you are writing Bourne shell without doing that, you are always going to have problems, because arguments are not always filenames. Yes, the syntax can be slightly annoying for historical/compatibility reasons. > What if I didn't have to? You don't. Use a different shell, or write your scripts in one of the many languages that are widely available like ruby, python, etc. If a scrip0t has #!/bin/sh at the top, you can probably rewrite it into any language you prefer.
alerighi 8y ago
A more simple and clean solution, that I always use in my python scripts: always work with absolute paths, that starts with /, and you solve all these problems. No reason to have relative paths outside the shell for me, relative paths are only a convenience for the shell user that doesn't have to specify the full path as an argument, but if you invoke a program from another program, it doesn't cost you nothing to pass to it as an argument the full path.
- deleted 8y ago
  [deleted]
- cesarb 8y ago
  There are situations (most of them involving symbolic links) in which using absolute paths can change the meaning of the path. The most obvious is when creating a symbolic link, for instance a relative symbolic link from "README" to "README.md" will not break when the whole directory is moved. A more worrying one is that using absolute paths everywhere is vulnerable to TOCTOU issues, while relative paths can be less vulnerable (and can be combined with newer APIs like openat to reduce these issues further).
zbentley 8y ago
> What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That's true, and a very real problem in many cases. A lot of those cases, though, are because people use positional arguments when they should use switched options ('--foo=bar' or '--foo bar') instead. This shields them from all kinds of pretty severe input-validation errors, and not just with filenames. It's a few more characters to type, but a great many option-parsing libs in all sorts of languages can automatically generate terse ('--foo=bar' can be specified as '-f bar') I know that puts some of the onus on developers, but as you and others have pointed out, relying on the shell to handle these weird cases likely puts responsibility where it doesn't belong (calling subprocesses from another language shouldn't use the shell unless it's overwhelmingly necessary) and on something that handles it poorly (all your points about quoting hell). That's a narrow quibble, and is exclusively concerned with getting data in to invoked programs intact, from a non-shell invoker. All of your other points about encoding, validation, shell-altering STDOUT, and broken behavior when you are working in the shell (like the whitespace-related issues you point out, which are still a hassle to work around even when using switched options) are well put and very well taken.

swiley 8y ago

If the user is the one entering the program name not calling the shell can cause unexpected behavior.

zbentley 8y ago
How so? The only cases in which I've seen this cause problems are programs which rely on the presence of a tty when they should not, or assume the existence of environment variables that they don't really need.
- swiley 8y ago
  Neither of those really have anything to do with the shell? And it's not like it's all impossible to do yourself without the shell there's just a lot and the shell does it in a nice and well known way. Here are some examples: Searching the path Setting custom environment variables builtins One weird one that I feel like a lot of people don't know about is that you can run shell scripts from the shell without the shebang. But if you exec one you'll get a format error. It's not good to rely on this but it's another thing that surprises people with messy scripts.