7 ms·

Just thinking of one of OP's examples, his (very cool) autoquoter allows him to type: > ssh user@host awk '{print $1}' file.txt And have the "$1" quoted or es

Just thinking of one of OP's examples, his (very cool) autoquoter allows him to type: > ssh user@host awk '{print $1}' file.txt And have the "$1" quoted or escaped so that it's interpreted as an awk keyword on the remote system. But what if I actually want it interpreted as a shell variable that should be expanded on the local or remote system? How does it know? Why is shell quoting still so hard?

ianthehenry 4y ago
This is a good question without a good answer. zsh-autoquoter doesn't know -- it doesn't know anything, really. The only thing it does is pass the literal string you type as an argument to the command you run (in this case, ssh). So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead: ssh user@host awk "{print $1}" file.txt But there's no way, using zsh-autoquoter, to say "yeah but this particular part of the string should not be escaped; let my local shell handle it."
Beltalowda 4y ago
> Why is shell quoting still so hard? The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script. And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk. So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.
- haroldp 4y ago
  > The problem with the awk example here specifically is that you're mixing two different languages (shell and awk) Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.
a1369209993 4y ago
> > ssh user@host awk '{print $1}' file.txt > it's interpreted as an awk keyword on the remote system. That's the behaviour in bash (and I think pretty much every other vaguely Bourne-like shell). That's what single quotes do. > what if I actually want it interpreted as a shell variable that should be expanded on the local [...] system? $ ssh user@host awk "{print $1}" file.txt But note that `$1` will be used as awk code, not string data. You need somthing like `\"$(echo $1 | s!(\W)!\\$1!g)\"`, the details of which will be shell-specific, if you want it to be awk code for a string constant. > what if I actually want it interpreted as a shell variable that should be expanded on the [...] remote system? $ ssh user@host sh -c 'awk "{print $1}" file.txt' Where `sh` is whatever shell you're using and `$1` specifically is obviously not very useful here.
- haroldp 4y ago
  I mean, yes I know how to quote and escape. My point was that only I know what my intent was. Do I want local shell interpretation, remote shell interpretation, awk interpretation or a literal string? I'm not sure how great an autoquoter is going to be if it doesn't know my intent. In the end, I'll need to do it myself, since I know what I want. And the shame of that is that there aren't a lot of tools to help. Utility functions like shell_escape() would be nice, but you still need to handle escaping for the local shell. That's essentially true of any programming language.
  - a1369209993 4y ago
    > My point was that only I know what my intent was. [...] In the end, I'll need to do it myself, since I know what I want. I think we're in violent agreement here; my point was that the autoquoter is superfluous, since the command in your previous comment already does what > his (very cool) autoquoter [supposedly] allows even without a autoquoter. > Utility functions like shell_escape() would be nice Note that in this case you specifically do not want shell_escape(). You're trying to produce awk code that evaluates to a given string, and that requires knowing awk syntax, not shell syntax; if you escape according to shell syntax, a attacker may be able to find a string where the shell expression for that string, when interpreted as awk code, does something other than evalute to a string. You could have a generic_escape() function that (say) replaced any non-alphanumeric byte with '\xHH' or '\B' (for 'B'==0xHH), but there will always some language where whatever generic strategy you picked doesn't work. $ echo "\x3E" x3E # shell doesn't accept \xHH $ echo '12+3<45' | grep -oE '\<.' | paste -sd' ' 1 3 4 # posix ERE doesn't accept \B

ianthehenry 4y ago

This is a good question without a good answer. zsh-autoquoter doesn't know -- it doesn't know anything, really. The only thing it does is pass the literal string you type as an argument to the command you run (in this case, ssh). So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead: ssh user@host awk "{print $1}" file.txt But there's no way, using zsh-autoquoter, to say "yeah but this particular part of the string should not be escaped; let my local shell handle it."

Beltalowda 4y ago

> Why is shell quoting still so hard? The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script. And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk. So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.

haroldp 4y ago
> The problem with the awk example here specifically is that you're mixing two different languages (shell and awk) Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.

a1369209993 4y ago

> > ssh user@host awk '{print $1}' file.txt > it's interpreted as an awk keyword on the remote system. That's the behaviour in bash (and I think pretty much every other vaguely Bourne-like shell). That's what single quotes do. > what if I actually want it interpreted as a shell variable that should be expanded on the local [...] system? $ ssh user@host awk "{print $1}" file.txt But note that `$1` will be used as awk code, not string data. You need somthing like `\"$(echo $1 | s!(\W)!\\$1!g)\"`, the details of which will be shell-specific, if you want it to be awk code for a string constant. > what if I actually want it interpreted as a shell variable that should be expanded on the [...] remote system? $ ssh user@host sh -c 'awk "{print $1}" file.txt' Where `sh` is whatever shell you're using and `$1` specifically is obviously not very useful here.

haroldp 4y ago
I mean, yes I know how to quote and escape. My point was that only I know what my intent was. Do I want local shell interpretation, remote shell interpretation, awk interpretation or a literal string? I'm not sure how great an autoquoter is going to be if it doesn't know my intent. In the end, I'll need to do it myself, since I know what I want. And the shame of that is that there aren't a lot of tools to help. Utility functions like shell_escape() would be nice, but you still need to handle escaping for the local shell. That's essentially true of any programming language.
- a1369209993 4y ago
  > My point was that only I know what my intent was. [...] In the end, I'll need to do it myself, since I know what I want. I think we're in violent agreement here; my point was that the autoquoter is superfluous, since the command in your previous comment already does what > his (very cool) autoquoter [supposedly] allows even without a autoquoter. > Utility functions like shell_escape() would be nice Note that in this case you specifically do not want shell_escape(). You're trying to produce awk code that evaluates to a given string, and that requires knowing awk syntax, not shell syntax; if you escape according to shell syntax, a attacker may be able to find a string where the shell expression for that string, when interpreted as awk code, does something other than evalute to a string. You could have a generic_escape() function that (say) replaced any non-alphanumeric byte with '\xHH' or '\B' (for 'B'==0xHH), but there will always some language where whatever generic strategy you picked doesn't work. $ echo "\x3E" x3E # shell doesn't accept \xHH $ echo '12+3<45' | grep -oE '\<.' | paste -sd' ' 1 3 4 # posix ERE doesn't accept \B