3 ms·

Safe Shell String Interpolation

21 points by Wingy 2y ago

amelius 2y ago
This is one reason why, really, nobody should use a shell that was optimized for commandline use, for scripting. On the commandline, all your inputs are usually known and trusted, so the shell language will take a few shortcuts to make everything more convenient for the user. But of course, when scripting, these assumptions don't hold and you need a language with more notational rigor. Bash et al. are dangerous scripting languages, and they become even more dangerous when users (who typically don't want to understand the subtleties) start using LLMs to generate code for them.
- BoingBoomTschak 2y ago
  That example is pretty sad, the solution has always been the same (even in sh), use an argv based wrapper instead of something like system(3): exec(`figlet "Welcome, ${username}" | lolcat -f`) // Should be spawn(["sh", "-c", "figlet \"Welcome, $1\" | lolcat -f", "argv0", username]) > This is one reason why, really, nobody should use a shell that was optimized for commandline use, for scripting. Tcl showed me you can have your cake and eat it, you just need sane quoting rules (like don't make IFS splitting the default and use matched braces for your quoting syntax to solve nesting) and arrays to escape the infamous quoting hell. Though I agree that sh is dangerous for newbies, I always advise "1. ALWAYS set -eu, 2. shellcheck" to them.
  - amelius 2y ago
    > Though I agree that sh is dangerous for newbies The problem is that expert users of sh will write scripts that newbies may at some point modify. This would be much safer if the scripts were originally in a better language. Bash is a footgun. Even expert users have shot themselves in the foot with it at some point in their career.
telotortium 2y ago
No love for Python's shlex.quote[1] or their equivalents in other languages to quote arguments for shell? In his case, which is interpolating a variable inside a shell double-quoted string, I would probably use environment variables, or else use shell `printf` with `shlex.quote`. But generally I interpolate directly into the command line using `shlex.quote`, when I can't avoid executing using the shell. I was happy to learn about Python's template string proposal (i.e., t-strings) from https://peps.python.org/pep-0750/ https://peps.python.org/pep-0750/, although it probably won't become common for a while. [1] https://docs.python.org/3/library/shlex.html#shlex.quote https://docs.python.org/3/library/shlex.html#shlex.quote
- Wingy 2y ago
  Yes, you can quote the values and embed them directly, but what if you want to do something like this? shell('echo "Hello, {user}"') You would have to do something like: arg_for_echo = f"Hello, {user}" shell(f"echo {shlex.quote(arg_for_echo)}") The main drawback of quoting values like that is that you can't embed them inside an argument in the shell. The environment variable "trick" I used avoids this problem. Using quotes to sanitize arguments like that is how google/zx (library for writing shell/JS hybrid scripts) handles arguments: https://google.github.io/zx/quotes https://google.github.io/zx/quotes
  - oasisaimlessly 2y ago
    My proposal: shell('echo '+shlex.quote(f'Hello, {user}')) i.e. never quote any (non-constant?) shell args by hand.
deleted 2y ago
[deleted]
js2 2y ago
Pipeline in Python w/o invoking the shell: username = input("Hello, what's your name? ") p1 = Popen(["figlet", f"Welcome, {username}"], stdout=PIPE) p2 = Popen(["lolcat", "-f"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. banner = p2.communicate()[0] print(banner) https://docs.python.org/3/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3/library/subprocess.html#replacing-... Don't use the shell unless you absolutely have to, and when you do have to, use shlex.quote for quoting arguments: https://docs.python.org/3/library/shlex.html#shlex.quote https://docs.python.org/3/library/shlex.html#shlex.quote username = input("Hello, what's your name? ") banner = check_output(f"figlet "Welcome, {quote(username)}" | lolcat -f", shell=True) print(banner) For something this simple, you could also just use `subprocess.check_output` twice: username = input("Hello, what's your name? ") banner = check_output(["figlet", f"Welcome, {username}"]) banner = check_output(["lolcat", "-f"], input=banner) print(banner)

21 points by Wingy 2y ago

This is one reason why, really, nobody should use a shell that was optimized for commandline use, for scripting. On the commandline, all your inputs are usually known and trusted, so the shell language will take a few shortcuts to make everything more convenient for the user. But of course, when scripting, these assumptions don't hold and you need a language with more notational rigor. Bash et al. are dangerous scripting languages, and they become even more dangerous when users (who typically don't want to understand the subtleties) start using LLMs to generate code for them.

BoingBoomTschak 2y ago
That example is pretty sad, the solution has always been the same (even in sh), use an argv based wrapper instead of something like system(3): exec(`figlet "Welcome, ${username}" | lolcat -f`) // Should be spawn(["sh", "-c", "figlet \"Welcome, $1\" | lolcat -f", "argv0", username]) > This is one reason why, really, nobody should use a shell that was optimized for commandline use, for scripting. Tcl showed me you can have your cake and eat it, you just need sane quoting rules (like don't make IFS splitting the default and use matched braces for your quoting syntax to solve nesting) and arrays to escape the infamous quoting hell. Though I agree that sh is dangerous for newbies, I always advise "1. ALWAYS set -eu, 2. shellcheck" to them.
- amelius 2y ago
  > Though I agree that sh is dangerous for newbies The problem is that expert users of sh will write scripts that newbies may at some point modify. This would be much safer if the scripts were originally in a better language. Bash is a footgun. Even expert users have shot themselves in the foot with it at some point in their career.

telotortium 2y ago

No love for Python's shlex.quote[1] or their equivalents in other languages to quote arguments for shell? In his case, which is interpolating a variable inside a shell double-quoted string, I would probably use environment variables, or else use shell `printf` with `shlex.quote`. But generally I interpolate directly into the command line using `shlex.quote`, when I can't avoid executing using the shell. I was happy to learn about Python's template string proposal (i.e., t-strings) from https://peps.python.org/pep-0750/ https://peps.python.org/pep-0750/, although it probably won't become common for a while. [1] https://docs.python.org/3/library/shlex.html#shlex.quote https://docs.python.org/3/library/shlex.html#shlex.quote

Wingy 2y ago
Yes, you can quote the values and embed them directly, but what if you want to do something like this? shell('echo "Hello, {user}"') You would have to do something like: arg_for_echo = f"Hello, {user}" shell(f"echo {shlex.quote(arg_for_echo)}") The main drawback of quoting values like that is that you can't embed them inside an argument in the shell. The environment variable "trick" I used avoids this problem. Using quotes to sanitize arguments like that is how google/zx (library for writing shell/JS hybrid scripts) handles arguments: https://google.github.io/zx/quotes https://google.github.io/zx/quotes
- oasisaimlessly 2y ago
  My proposal: shell('echo '+shlex.quote(f'Hello, {user}')) i.e. never quote any (non-constant?) shell args by hand.

deleted 2y ago

[deleted]

js2 2y ago

Pipeline in Python w/o invoking the shell: username = input("Hello, what's your name? ") p1 = Popen(["figlet", f"Welcome, {username}"], stdout=PIPE) p2 = Popen(["lolcat", "-f"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. banner = p2.communicate()[0] print(banner) https://docs.python.org/3/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3/library/subprocess.html#replacing-... Don't use the shell unless you absolutely have to, and when you do have to, use shlex.quote for quoting arguments: https://docs.python.org/3/library/shlex.html#shlex.quote https://docs.python.org/3/library/shlex.html#shlex.quote username = input("Hello, what's your name? ") banner = check_output(f"figlet "Welcome, {quote(username)}" | lolcat -f", shell=True) print(banner) For something this simple, you could also just use `subprocess.check_output` twice: username = input("Hello, what's your name? ") banner = check_output(["figlet", f"Welcome, {username}"]) banner = check_output(["lolcat", "-f"], input=banner) print(banner)