7 ms·

False. Many of these problems affect all languages, including Python3. There are still many cases where you need to execute external programs. It's true that so

False. Many of these problems affect all languages, including Python3. There are still many cases where you need to execute external programs. It's true that some problems are worse in shell, but even then, most languages (including Python) have constructs that call shell. Since the problems hit all languages, we should work on trying to make things better. Where possible, the simple and obvious thing should be the safe thing.

kazinator 8y ago
I completely agree with your idea; I've long believed that the OS kernel should reject dubious filenames.
- kazinator 8y ago
  However, from past thoughts and discussions, there are issues. (1) We have to be able to mount filesystems which have bad filenames and allow users to work with them. In that situation, that means the kernel cannot reject all requests for bad file names, only creation requests. And perhaps not even that. It can be a mount option with three values: off; reject all dubious names in all filesystem syscalls; or just in creating syscalls: open with O_CREAT, mknod, link, ... It could also be a capability. Superuser could work with the dubious filenames and set up capabilities for non-privileged child processes to do same, so people needing to work with a filesystem full of these names can get into an environment where they can do that. (2) Enforcement at the kernel level is too late in some situations. It doesn't entirely compensate for broken programs. Here is a trivial example of what this means. Suppose that some program receives an argument "foo bar" and decides to split that into "foo" and "bar", treating this as two file names to create or process. Those filenames look fine to the kernel, so it is allowed. In shell scripts, not all name-related issues arise from files that are already in the system. In fact, shell scripts are often robust in the face of existing files with poor names. For instance names generated with globbing expansion are safe, even if they contain spaces and shell meta-characters. The one danger is that the prefix of the expanded list looks like command line options to the program. The issues that arise in shell programming from failing to quote variable expansions cannot be fixed outside of the shell. The OS cannot fix the lack of quoting in command $VAR; it can't prevent a shell internal variable VAR from containing spaces.
rujuladanh 8y ago
The fact that languages have a way to run a command on the shell does not mean you should. Actually, you never should, unless you have a very, very good reason. Even without taking into account encoding issues, you still shouldn't.
- dwheeler 8y ago
  Please re-read the article. Even if you never use the shell, running a command can still cause a lot of problems because of file name issues. They would be a problem even if you uninstalled every shell.
tyingq 8y ago
It's also not hard to find Python/Perl/etc scripts that do things like "filename per line" or "split filenames on whitespace".
netheril96 8y ago
> There are still many cases where you need to execute external programs. But you don't need to call the shell to execute external programs.
- dwheeler 8y ago
  Exactly. Even if you didn't have a shell installed, these file names would be a problem. If a file name begins with the dash, and you naively pass that file name as argument one, you are still in trouble, even if you do not have a shell installed on your system. What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That is because leading dashes are allowable as file names. It is possible to workaround this, for example by pre pending "./". But our goal should be to make things easy, not just make it possible to do things correctly.
  - pdkl95 8y ago
    > It is possible to workaround this, for example by pre pending "./". Or you could use the solutio9n provided by most argument parsers: the "--" option that disables option parsing for the remaining options. Trying to fix it yourself by "fixing" filenames (by prefixing with "./" or whatever) requires you to enumerate all possible problem[1] or you're just creating a new weird machine[2]. You're worrying a lot about specific problems like if a filename might be confused with an option, but there are many other ways filenames can be a problem *even if their names are limited to "simple"[3] alphanumeric characters. If you're accepting input from the user (or worse, over a network), you will have far worse problem than crashing on a bad filename if you aren't strictly validating and parsing that input before sending it to any other program or using it as a filename. [1] ("#2) Enumerating Badness") http://www.ranum.com/security/computer_security/editorials/dumb/ http://www.ranum.com/security/computer_security/editorials/d... [2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit... [3] Defining concepts like "alphanumeric" or even "codepoints that can be safely [printed to the screen" are NOT simple co9ncepts if you support Unicode (which most programs should support).
the8472 8y ago
> most languages (including Python) have constructs that call shell. Rust's stdlib doesn't. You must pass arguments as array. https://doc.rust-lang.org/std/process/struct.Command.html https://doc.rust-lang.org/std/process/struct.Command.html
- kazinator 8y ago
  An exec-like mechanism be used to construct a [ "/bin/sh" "/bin/sh" "-c" "yourcommandhere", NULL ] argument vector.
  - the8472 8y ago
    sure, but then you're intentionally making things harder on yourself
    - kazinator 8y ago
      Making what harder on yourself? What if yourcommandhere is shell syntax, like "for x in *.foo; do echo $foo; done"? That can't be executed directly as a process argument list. If all you have is a wrapper for execv, then you have to construct a call to "sh", where that command is all in one argument following a "-c" argument. You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation. A language and its library won't prevent applications from calling the shell, unless it's uselessly crippled.
      - the8472 8y ago
        > What if yourcommandhere is shell syntax, like "for x in *.foo; do echo $foo; done" Then you readdir, loop, println. no need to use the shell > You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation. You're stating the obvious. My point was that it has no footgun which automatically defers to the shell or does shell-like argument extraction from a single string when spawning child processes. Other languages have those things in their standard library which is a source of errors that rust avoids. It does provide a regular gun which you can repurposed as a make-shift footgun by taking the conscious action of aiming down first.
catern 8y ago
Come on. The problems you list (in the introduction) are: - control characters in filenames: irrelevant to non-shell languages - leading dashes in filenames: doesn't affect normal Python/non-shell languages; for example, unlike in shell, directory walking in Python is implemented by calling os.walk, not execing a separate process "find" and parsing its output - the lack of a standard character encoding scheme: doesn't affect Python 3, now that the filesystem encoding system is complete and cleanly handles encoding errors - spaces in filenames can cause problems: irrelevant to non-shell languages - special metacharacters in filenames cause some problems: irrelevant to non-shell languages As you admit, these problems are worse in shell (I contend, only meaningfully exist in shell). So why is your article written using shell scripting for its examples? - If you really believe that shell is worse for this, then you should be warding people away from shell - If you really believe these problems are pervasive, take on the hard target, instead of going for the easy target of shell - If these problems really still affect Python programmers who believe they are safe from these footguns that are so prevalent in shell, then you should be warning them, not the shell programmers who already know that shell is an incredibly hard language to write robust programs in But as your article is using shell for its examples, I can't help but conclude that this is just another piece of shell-zealotry. If you seriously want to fix this problem for people, you should be advising them to stop writing programs in shell. That is undeniably the fastest and easiest mitigation to the majority of these problems. On that, surely you must agree. ----------------------------------------------------------------- An aside on the topic of leading dashes in filenames: it's true that any program parsing filenames from the command line will run into this issue where data and options can be confused. You diagnose this as an issue with filenames, but I disagree: This is an issue with traditional argument parsing, which is a very poor serialization format. Any number of alternative serialization formats would avoid this issue; for example, https://github.com/NuxiNL/argdata https://github.com/NuxiNL/argdata does not have this problem.
- swiley 8y ago
  >control characters in filenames I don't understand how this doesn't affect non-shell languages, it's not the shell that interprets these (unless were talking about different things) > spaces in filenames can cause problems: irrelevant to non-shell languages Again, this is a format/protocol problem not a language problem. A lot of formats use spaces as delimiters and if you put delimiters in the filename you're going to have to escape it.

kazinator 8y ago

I completely agree with your idea; I've long believed that the OS kernel should reject dubious filenames.

kazinator 8y ago
However, from past thoughts and discussions, there are issues. (1) We have to be able to mount filesystems which have bad filenames and allow users to work with them. In that situation, that means the kernel cannot reject all requests for bad file names, only creation requests. And perhaps not even that. It can be a mount option with three values: off; reject all dubious names in all filesystem syscalls; or just in creating syscalls: open with O_CREAT, mknod, link, ... It could also be a capability. Superuser could work with the dubious filenames and set up capabilities for non-privileged child processes to do same, so people needing to work with a filesystem full of these names can get into an environment where they can do that. (2) Enforcement at the kernel level is too late in some situations. It doesn't entirely compensate for broken programs. Here is a trivial example of what this means. Suppose that some program receives an argument "foo bar" and decides to split that into "foo" and "bar", treating this as two file names to create or process. Those filenames look fine to the kernel, so it is allowed. In shell scripts, not all name-related issues arise from files that are already in the system. In fact, shell scripts are often robust in the face of existing files with poor names. For instance names generated with globbing expansion are safe, even if they contain spaces and shell meta-characters. The one danger is that the prefix of the expanded list looks like command line options to the program. The issues that arise in shell programming from failing to quote variable expansions cannot be fixed outside of the shell. The OS cannot fix the lack of quoting in command $VAR; it can't prevent a shell internal variable VAR from containing spaces.

rujuladanh 8y ago

The fact that languages have a way to run a command on the shell does not mean you should. Actually, you never should, unless you have a very, very good reason. Even without taking into account encoding issues, you still shouldn't.

dwheeler 8y ago
Please re-read the article. Even if you never use the shell, running a command can still cause a lot of problems because of file name issues. They would be a problem even if you uninstalled every shell.

tyingq 8y ago

It's also not hard to find Python/Perl/etc scripts that do things like "filename per line" or "split filenames on whitespace".

netheril96 8y ago

> There are still many cases where you need to execute external programs. But you don't need to call the shell to execute external programs.

dwheeler 8y ago
Exactly. Even if you didn't have a shell installed, these file names would be a problem. If a file name begins with the dash, and you naively pass that file name as argument one, you are still in trouble, even if you do not have a shell installed on your system. What will happen is the receiving program will see a dash as the first character of its first argument, and in most cases assume it is an option, not a filename. That is because leading dashes are allowable as file names. It is possible to workaround this, for example by pre pending "./". But our goal should be to make things easy, not just make it possible to do things correctly.
- pdkl95 8y ago
  > It is possible to workaround this, for example by pre pending "./". Or you could use the solutio9n provided by most argument parsers: the "--" option that disables option parsing for the remaining options. Trying to fix it yourself by "fixing" filenames (by prefixing with "./" or whatever) requires you to enumerate all possible problem[1] or you're just creating a new weird machine[2]. You're worrying a lot about specific problems like if a filename might be confused with an option, but there are many other ways filenames can be a problem *even if their names are limited to "simple"[3] alphanumeric characters. If you're accepting input from the user (or worse, over a network), you will have far worse problem than crashing on a bad filename if you aren't strictly validating and parsing that input before sending it to any other program or using it as a filename. [1] ("#2) Enumerating Badness") http://www.ranum.com/security/computer_security/editorials/dumb/ http://www.ranum.com/security/computer_security/editorials/d... [2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurity https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit... [3] Defining concepts like "alphanumeric" or even "codepoints that can be safely [printed to the screen" are NOT simple co9ncepts if you support Unicode (which most programs should support).

the8472 8y ago

> most languages (including Python) have constructs that call shell. Rust's stdlib doesn't. You must pass arguments as array. https://doc.rust-lang.org/std/process/struct.Command.html https://doc.rust-lang.org/std/process/struct.Command.html

kazinator 8y ago
An exec-like mechanism be used to construct a [ "/bin/sh" "/bin/sh" "-c" "yourcommandhere", NULL ] argument vector.
- the8472 8y ago
  sure, but then you're intentionally making things harder on yourself
  - kazinator 8y ago
    Making what harder on yourself? What if yourcommandhere is shell syntax, like "for x in *.foo; do echo $foo; done"? That can't be executed directly as a process argument list. If all you have is a wrapper for execv, then you have to construct a call to "sh", where that command is all in one argument following a "-c" argument. You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation. A language and its library won't prevent applications from calling the shell, unless it's uselessly crippled.
    - the8472 8y ago
      > What if yourcommandhere is shell syntax, like "for x in *.foo; do echo $foo; done" Then you readdir, loop, println. no need to use the shell > You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation. You're stating the obvious. My point was that it has no footgun which automatically defers to the shell or does shell-like argument extraction from a single string when spawning child processes. Other languages have those things in their standard library which is a source of errors that rust avoids. It does provide a regular gun which you can repurposed as a make-shift footgun by taking the conscious action of aiming down first.

catern 8y ago

Come on. The problems you list (in the introduction) are: - control characters in filenames: irrelevant to non-shell languages - leading dashes in filenames: doesn't affect normal Python/non-shell languages; for example, unlike in shell, directory walking in Python is implemented by calling os.walk, not execing a separate process "find" and parsing its output - the lack of a standard character encoding scheme: doesn't affect Python 3, now that the filesystem encoding system is complete and cleanly handles encoding errors - spaces in filenames can cause problems: irrelevant to non-shell languages - special metacharacters in filenames cause some problems: irrelevant to non-shell languages As you admit, these problems are worse in shell (I contend, only meaningfully exist in shell). So why is your article written using shell scripting for its examples? - If you really believe that shell is worse for this, then you should be warding people away from shell - If you really believe these problems are pervasive, take on the hard target, instead of going for the easy target of shell - If these problems really still affect Python programmers who believe they are safe from these footguns that are so prevalent in shell, then you should be warning them, not the shell programmers who already know that shell is an incredibly hard language to write robust programs in But as your article is using shell for its examples, I can't help but conclude that this is just another piece of shell-zealotry. If you seriously want to fix this problem for people, you should be advising them to stop writing programs in shell. That is undeniably the fastest and easiest mitigation to the majority of these problems. On that, surely you must agree. ----------------------------------------------------------------- An aside on the topic of leading dashes in filenames: it's true that any program parsing filenames from the command line will run into this issue where data and options can be confused. You diagnose this as an issue with filenames, but I disagree: This is an issue with traditional argument parsing, which is a very poor serialization format. Any number of alternative serialization formats would avoid this issue; for example, https://github.com/NuxiNL/argdata https://github.com/NuxiNL/argdata does not have this problem.

swiley 8y ago
>control characters in filenames I don't understand how this doesn't affect non-shell languages, it's not the shell that interprets these (unless were talking about different things) > spaces in filenames can cause problems: irrelevant to non-shell languages Again, this is a format/protocol problem not a language problem. A lot of formats use spaces as delimiters and if you put delimiters in the filename you're going to have to escape it.