6 ms·

Zsh-autoquoter makes shell quoting slightly less annoying

90 points by ianthehenry 4y ago

dncornholio 4y ago
That's truly an impressive Moth. Good find. The zsh plugin looks nice too!
mmastrac 4y ago
The shell quoting rabbithole goes pretty deep, especially when you want to preserve the whitespace within arguments. Has this been solved generally yet? I suspect it requires full knowledge of whatever shell is running on the other side of SSH.
- weinzierl 4y ago
  An autoquoter is a bit too much magic for me, but I wish there was a quote explainer. Much like 'cdecl' for C declarations, a program that explains expressions with quotes in plain English. Problem, I guess, is that shell quoting depends on the context where it is used.
- bool3max 4y ago
  It is part of the reason why I use fish whenever possible.
  - bool3max 4y ago
    String escaping in fish is phenomenal. Painless. You don't have to think about it at all. A single variable expansion results in exactly a single command line argument. As it should. As is logical.
- haroldp 4y ago
  Just thinking of one of OP's examples, his (very cool) autoquoter allows him to type: > ssh user@host awk '{print $1}' file.txt And have the "$1" quoted or escaped so that it's interpreted as an awk keyword on the remote system. But what if I actually want it interpreted as a shell variable that should be expanded on the local or remote system? How does it know? Why is shell quoting still so hard?
  - ianthehenry 4y ago
    This is a good question without a good answer. zsh-autoquoter doesn't know -- it doesn't know anything, really. The only thing it does is pass the literal string you type as an argument to the command you run (in this case, ssh). So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead: ssh user@host awk "{print $1}" file.txt But there's no way, using zsh-autoquoter, to say "yeah but this particular part of the string should not be escaped; let my local shell handle it."
  - Beltalowda 4y ago
    > Why is shell quoting still so hard? The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script. And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk. So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.
    - haroldp 4y ago
      > The problem with the awk example here specifically is that you're mixing two different languages (shell and awk) Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.
  - a1369209993 4y ago
    > > ssh user@host awk '{print $1}' file.txt > it's interpreted as an awk keyword on the remote system. That's the behaviour in bash (and I think pretty much every other vaguely Bourne-like shell). That's what single quotes do. > what if I actually want it interpreted as a shell variable that should be expanded on the local [...] system? $ ssh user@host awk "{print $1}" file.txt But note that `$1` will be used as awk code, not string data. You need somthing like `\"$(echo $1 | s!(\W)!\\$1!g)\"`, the details of which will be shell-specific, if you want it to be awk code for a string constant. > what if I actually want it interpreted as a shell variable that should be expanded on the [...] remote system? $ ssh user@host sh -c 'awk "{print $1}" file.txt' Where `sh` is whatever shell you're using and `$1` specifically is obviously not very useful here.
orsenthil 4y ago
I am still afraid to switch from Bash to Zsh, because these are things I don't fully understand and will find it hard to debug if _something_ breaks. Is something like this available for bash?
- dmd 4y ago
  It's puzzling to me that people still think of zsh as "that newfangled shell I'm not sure about yet". Bash came out in 1989. Zsh came out in 1990.
  - __del__ 4y ago
    i use zsh, but i often invoke bash just to make sure some behavior is "normal" rather than zsh-specific, or specific to how i wrote my .zshrc last decade [and haven't changed]. i don't think it's about the newfangledness; i think it's about the potential for subtle differences in syntax which scare people away from zsh.
    - Beltalowda 4y ago
      I think the initial setup can be quite daunting; unlike bash, which never seemed to have moved beyond 1990, zsh added a lot of features, many of which are very useful (some of which less so, like csh-compatibility things) but there's a lot of it, and the defaults are somewhat bare-bones. Hence things like oh-my-zsh, which is quite intimidating in its own way. The fish shell "fixed" a lot of that by adding more or less the same feature-set as zsh but without the extensive configurability, which is a really great trade-off if the fish defaults work well for you (they don't for me personally though).
- monkpit 4y ago
  Write your scripts using bash or sh in the hashbang and use zsh for your daily work. Also, use shellcheck.
- chasil 4y ago
  It is useful to try your scripts in a POSIX shell that avoids non-standard extensions. The best known is the Debian dash reroll of the Almquist shell, but there are other implementations (I've heard of mrsh and gsh, and the one in OCaml). POSIX shells do not support arrays, and many other common bash extensions. However, dash is very small, and very fast (4x faster than bash according to some sources). Scripts written for the POSIX shell are maximally-portable.
NegativeLatency 4y ago
One of the nice things about fish is that it does this when pasting: https://github.com/fish-shell/fish-shell/issues/967 https://github.com/fish-shell/fish-shell/issues/967
- jchw 4y ago
  PowerShell does this too, at least for URLs. That said, it’s a true pain in the ass when it kicks in incorrectly, which unfortunately does happen.
- re 4y ago
  So can Zsh -- this functionality has shipped with Zsh since for several years, though not enabled by default. See https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-magic https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b... or https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-url-magic https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b...
- a1369209993 4y ago
  Honestly, the "when pasting" part seems very much a case of solving the wrong problem there. For my shell, I just had any unquoted ':/' or ':?' mark the word as a url, causing glob characters (and some others like '&', for obvious[0] reasons) to be implicitly quoted until the end of the word. 0: assuming you've seen a url like http://example.com/foo?bar=1&baz=2 http://example.com/foo?bar=1&baz=2 before
drw 4y ago
More flexible would be a double-quote-region widget which can be applied arbitrarily to any region of a command line: emulate -L zsh if [[ "$REGION_ACTIVE" -eq 0 ]]; then return fi if [[ "$MARK" -le "$CURSOR" ]]; then left_region_bound="$MARK" right_region_bound="$CURSOR" else left_region_bound="$CURSOR" right_region_bound="$MARK" fi length_region="$((right_region_bound - left_region_bound))" before_region="${BUFFER:0:$left_region_bound}" region="${BUFFER:$left_region_bound:$length_region}" after_region="${BUFFER:$right_region_bound}" quoted_region="${(qqq)region}" cursor_offset="$(($#quoted_region - $#region))" BUFFER="$before_region$quoted_region$after_region" CURSOR="$((CURSOR + cursor_offset))" REGION_ACTIVE=0 bind with eg zle -N double-quote-region bindkey '\e"' double-quote-region
andmis 4y ago
Try fish. The quoting rules are simple. I never have to read the docs or try repeatedly to get quoting right. The shell syntax in general is small and simple. The only downside to fish is that it's not POSIX compliant, but if you're familiar with POSIX shells this is rarely a problem for interactive use, and you can use Bash for scripting if you like.
noname120 4y ago
I'd recommend skipping directly to how the matching configuration works: https://github.com/ianthehenry/zsh-autoquoter#behavior https://github.com/ianthehenry/zsh-autoquoter#behavior

90 points by ianthehenry 4y ago

That's truly an impressive Moth. Good find. The zsh plugin looks nice too!

The shell quoting rabbithole goes pretty deep, especially when you want to preserve the whitespace within arguments. Has this been solved generally yet? I suspect it requires full knowledge of whatever shell is running on the other side of SSH.

weinzierl 4y ago
An autoquoter is a bit too much magic for me, but I wish there was a quote explainer. Much like 'cdecl' for C declarations, a program that explains expressions with quotes in plain English. Problem, I guess, is that shell quoting depends on the context where it is used.
bool3max 4y ago
It is part of the reason why I use fish whenever possible.
- bool3max 4y ago
  String escaping in fish is phenomenal. Painless. You don't have to think about it at all. A single variable expansion results in exactly a single command line argument. As it should. As is logical.
haroldp 4y ago
Just thinking of one of OP's examples, his (very cool) autoquoter allows him to type: > ssh user@host awk '{print $1}' file.txt And have the "$1" quoted or escaped so that it's interpreted as an awk keyword on the remote system. But what if I actually want it interpreted as a shell variable that should be expanded on the local or remote system? How does it know? Why is shell quoting still so hard?
- ianthehenry 4y ago
  This is a good question without a good answer. zsh-autoquoter doesn't know -- it doesn't know anything, really. The only thing it does is pass the literal string you type as an argument to the command you run (in this case, ssh). So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead: ssh user@host awk "{print $1}" file.txt But there's no way, using zsh-autoquoter, to say "yeah but this particular part of the string should not be escaped; let my local shell handle it."
- Beltalowda 4y ago
  > Why is shell quoting still so hard? The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script. And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk. So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.
  - haroldp 4y ago
    > The problem with the awk example here specifically is that you're mixing two different languages (shell and awk) Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.
- a1369209993 4y ago
  > > ssh user@host awk '{print $1}' file.txt > it's interpreted as an awk keyword on the remote system. That's the behaviour in bash (and I think pretty much every other vaguely Bourne-like shell). That's what single quotes do. > what if I actually want it interpreted as a shell variable that should be expanded on the local [...] system? $ ssh user@host awk "{print $1}" file.txt But note that `$1` will be used as awk code, not string data. You need somthing like `\"$(echo $1 | s!(\W)!\\$1!g)\"`, the details of which will be shell-specific, if you want it to be awk code for a string constant. > what if I actually want it interpreted as a shell variable that should be expanded on the [...] remote system? $ ssh user@host sh -c 'awk "{print $1}" file.txt' Where `sh` is whatever shell you're using and `$1` specifically is obviously not very useful here.

orsenthil 4y ago

I am still afraid to switch from Bash to Zsh, because these are things I don't fully understand and will find it hard to debug if _something_ breaks. Is something like this available for bash?

dmd 4y ago
It's puzzling to me that people still think of zsh as "that newfangled shell I'm not sure about yet". Bash came out in 1989. Zsh came out in 1990.
- __del__ 4y ago
  i use zsh, but i often invoke bash just to make sure some behavior is "normal" rather than zsh-specific, or specific to how i wrote my .zshrc last decade [and haven't changed]. i don't think it's about the newfangledness; i think it's about the potential for subtle differences in syntax which scare people away from zsh.
  - Beltalowda 4y ago
    I think the initial setup can be quite daunting; unlike bash, which never seemed to have moved beyond 1990, zsh added a lot of features, many of which are very useful (some of which less so, like csh-compatibility things) but there's a lot of it, and the defaults are somewhat bare-bones. Hence things like oh-my-zsh, which is quite intimidating in its own way. The fish shell "fixed" a lot of that by adding more or less the same feature-set as zsh but without the extensive configurability, which is a really great trade-off if the fish defaults work well for you (they don't for me personally though).
monkpit 4y ago
Write your scripts using bash or sh in the hashbang and use zsh for your daily work. Also, use shellcheck.
chasil 4y ago
It is useful to try your scripts in a POSIX shell that avoids non-standard extensions. The best known is the Debian dash reroll of the Almquist shell, but there are other implementations (I've heard of mrsh and gsh, and the one in OCaml). POSIX shells do not support arrays, and many other common bash extensions. However, dash is very small, and very fast (4x faster than bash according to some sources). Scripts written for the POSIX shell are maximally-portable.

NegativeLatency 4y ago

One of the nice things about fish is that it does this when pasting: https://github.com/fish-shell/fish-shell/issues/967 https://github.com/fish-shell/fish-shell/issues/967

jchw 4y ago
PowerShell does this too, at least for URLs. That said, it’s a true pain in the ass when it kicks in incorrectly, which unfortunately does happen.
re 4y ago
So can Zsh -- this functionality has shipped with Zsh since for several years, though not enabled by default. See https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-magic https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b... or https://github.com/zsh-users/zsh/blob/master/Functions/Zle/bracketed-paste-url-magic https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b...
a1369209993 4y ago
Honestly, the "when pasting" part seems very much a case of solving the wrong problem there. For my shell, I just had any unquoted ':/' or ':?' mark the word as a url, causing glob characters (and some others like '&', for obvious[0] reasons) to be implicitly quoted until the end of the word. 0: assuming you've seen a url like http://example.com/foo?bar=1&baz=2 http://example.com/foo?bar=1&baz=2 before

drw 4y ago

More flexible would be a double-quote-region widget which can be applied arbitrarily to any region of a command line: emulate -L zsh if [[ "$REGION_ACTIVE" -eq 0 ]]; then return fi if [[ "$MARK" -le "$CURSOR" ]]; then left_region_bound="$MARK" right_region_bound="$CURSOR" else left_region_bound="$CURSOR" right_region_bound="$MARK" fi length_region="$((right_region_bound - left_region_bound))" before_region="${BUFFER:0:$left_region_bound}" region="${BUFFER:$left_region_bound:$length_region}" after_region="${BUFFER:$right_region_bound}" quoted_region="${(qqq)region}" cursor_offset="$(($#quoted_region - $#region))" BUFFER="$before_region$quoted_region$after_region" CURSOR="$((CURSOR + cursor_offset))" REGION_ACTIVE=0 bind with eg zle -N double-quote-region bindkey '\e"' double-quote-region

andmis 4y ago

Try fish. The quoting rules are simple. I never have to read the docs or try repeatedly to get quoting right. The shell syntax in general is small and simple. The only downside to fish is that it's not POSIX compliant, but if you're familiar with POSIX shells this is rarely a problem for interactive use, and you can use Bash for scripting if you like.

noname120 4y ago

I'd recommend skipping directly to how the matching configuration works: https://github.com/ianthehenry/zsh-autoquoter#behavior https://github.com/ianthehenry/zsh-autoquoter#behavior