8 ms·

Shell Completions in Rust

150 points by JoshMcguigan 7y ago

JoshMcguigan 7y ago
Author here. This blog post is the result of my puzzling over how auto-complete works in the shell, and subsequently whether shell completion scripts could be written in Rust. It also introduces a new Rust crate, shell_completion[0], which is a library that exposes some low level primitives for writing shell completion scripts in Rust. shell_completion is certainly in its infancy, so I'd be happy to have contributors of any kind (both code and opinions are welcome). [0]: https://github.com/joshmcguigan/shell_completion https://github.com/joshmcguigan/shell_completion
- sam0x17 7y ago
  This is very cool -- thanks for your work! I wonder if the API for windows auto-completion is sophisticated enough that you could add cross-platform support for this.
  - jbreiding 7y ago
    No need to worry if you just use the same shell through WSL. The integration is with the shell not the os.
- sbr464 7y ago
  I’m curious about your approach, did you have any resources or projects that you found yourself going back to? (Regarding shell completions in general)
  - JoshMcguigan 7y ago
    These [0] two [1] sections of the GNU Bash man pages were very useful. I also found that this tutorial [2] provided a useful perspective. I spent a small amount of time looking through the bash-completion [3] project as well as the git bash completions [4]. Generally, my process for writing is google around until I have some base level of knowledge, then building one or more demo applications to ensure my understanding is correct. Then as I'm writing the blog post I'll be refining the demo applications into examples for the blog post and/or library code that I want to open source. [0]: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion https://www.gnu.org/software/bash/manual/html_node/Programma... [1]: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins https://www.gnu.org/software/bash/manual/html_node/Programma... [2]: https://hackaday.com/2018/01/19/linux-fu-custom-bash-command-completion/ https://hackaday.com/2018/01/19/linux-fu-custom-bash-command... [3]: https://github.com/scop/bash-completion https://github.com/scop/bash-completion [4]: https://github.com/git/git/blob/master/contrib/completion/git-completion.bash https://github.com/git/git/blob/master/contrib/completion/gi...
    - sbr464 7y ago
      Thanks for sharing (and @snowe2010)
  - snowe2010 7y ago
    I've actually written two separate articles on shell completion, explicitly for subcommands, because the resources regarding subcommands is so lacking. https://tylerthrailkill.com/2019-01-13/writing-zsh-completion-scripts/ https://tylerthrailkill.com/2019-01-13/writing-zsh-completio... https://tylerthrailkill.com/2019-01-19/writing-bash-completion-script-with-subcommands/ https://tylerthrailkill.com/2019-01-19/writing-bash-completi... I need to improve them and write a bit more information, I was writing a library to generate completion scripts for both bash and zsh, so I had to know the differences quite well.
tormeh 7y ago
The ion shell is written in what I believe is pure Rust, and has autocompletion by default.
- JoshMcguigan 7y ago
  It is not possible for a shell to know how to perform completions for every binary on a users machine. Shell completion scripts define the auto-completion for each specific binary, so for autocomplete to work (well) it needs to be supported both by the shell and by the application. This post is about writing shell completion scripts (the application side, not the shell side).
chubot 7y ago
If I'm understanding correctly, this is a shim that: 1) serializes the fields the bash completion API gives you (when you hit TAB) 2) exports them to Rust, then 3) passes the Rust program's results back to bash? I've been working on similar problems, and I understand why that's appealing. Bash is a pretty bad language to write completion scripts in! It's really bad at string and array processing [1]. ---- Though on a quick glance, if you're exporting $COMP_LINE and $COMP_POINT, then it's making the job of the completion script very hard, since you're passing off the responsibility of parsing shell to it: In other words, if you have something like: mycommand --flag arg\ with\ spaces --flag2=<TAB> Then it's not desirable to pass the entire string. It would be better to parse it and pass: ["mycommand", "--flag", "arg with spaces", "--flag2="] # and indicate index 3 being completed Another common case is: echo $(dirname <TAB> # I don't need echo $( to complete I noticed this deficiency of bash while essentially cloning the bash API for Oil: https://www.oilshell.org/ https://www.oilshell.org/ One consequence of this is that the bash-completion project (maintained separately from bash) actually has an ad hoc bash parser in bash! This is really bad -- unsurprisingly it has obvious bugs, like not being able to complete inside command subs. An ad hoc bash parser in Rust would also be bad. So instead Oil parses shell for you -- since it has a shell parser! -- and then it gives you the partial argv array. So then the plugin is only responsible for what it can reasonably do. ----- Additional, I proposed that this could be made into a protocol that other shells consume. So instead of writing bash completions in Rust, you could write zsh/fish/bash/Oil completions in Rust. https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos... A lot of people are already "scraping" bash completions this way, i.e. to use in zsh or their own shells, so it should not be hard to come up with something "standard", given those working examples. https://github.com/oilshell/oil/wiki/Projects-Already-Doing-Something-Like-Shellac https://github.com/oilshell/oil/wiki/Projects-Already-Doing-... If you're interested in chatting about it please join https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ . I wouldn't say it's the #1 priority now because Oil already emulates bash quite well, but I think a bunch of people are motivated because they 1) don't want to write completions N times, for every shell and 2) Don't want to write completions in the bash language. home page: main page: https://github.com/oilshell/oil/wiki/Shell-Autocompletion https://github.com/oilshell/oil/wiki/Shell-Autocompletion IMO it makes sense to write completions in whatever language the command line tool / flag parser is written in. [1] https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html
codetrotter 7y ago
Seeing this got me curious to know if any work has been done on generating completions for applications that make use of the clap command-line argument parser. https://github.com/clap-rs/clap https://github.com/clap-rs/clap https://crates.io/crates/clap https://crates.io/crates/clap Among other things I found this currently open issue that has a lot of discussion and information in it: https://github.com/clap-rs/clap/issues/568 https://github.com/clap-rs/clap/issues/568 It might be of interest to you as well, OP?
- JoshMcguigan 7y ago
  I did come across this work by clap while doing research for this post. The major difference in strategy is that clap is generating completion scripts in Bash, while shell_completion is a library which helps writing completion scripts in Rust. I think one exciting bit of future potential for shell_completion is it could potentially allow building something like what clap has done using the standard Rust macro system. In any event, thanks for bringing this up.
- snowe2010 7y ago
  I actually wrote a library that generates completions for both zsh and bash for the Ruby Thor CLI library. https://github.com/snowe2010/fylla/ https://github.com/snowe2010/fylla/ This could be useful in research to make it work for clap. I wrote two articles that kind of highlight the differences in completions between the two, but only in regards to subcommands, since that's what I was trying to cover. https://tylerthrailkill.com/2019-01-19/writing-bash-completion-script-with-subcommands/ https://tylerthrailkill.com/2019-01-19/writing-bash-completi... https://tylerthrailkill.com/2019-01-13/writing-zsh-completion-scripts/ https://tylerthrailkill.com/2019-01-13/writing-zsh-completio...
dmix 7y ago
That sounds useful. I used to write tons of ZSH scripts when I used to obsessively make my Linux desktop experience nice and automated. Writing custom completions always felt awkward for me and unintuitive (like most ZSH programming). Does your shell_completion library work with ZSH? I'm not familiar with how sh/bash differs here from ZSH but I'd image it'd be easy to make compatible. I'm waiting patiently for Evlish to mature so I can use their Go-like (and Go-powered) scripting language and burn my old ZSH scripts in a fire. https://github.com/elves/elvish https://github.com/elves/elvish
- chubot 7y ago
  If you're using multiple shells like zsh and Elvish, then you might be interested in my proposal below for shell-agnostic autocompletion: https://news.ycombinator.com/item?id=19950892 https://news.ycombinator.com/item?id=19950892 I talked with the author of Elvish about it, and we both want it to happen, but are both working on other things now AFAIK. If you managed to write a working bash/zsh completion script, then kudos :) It's not easy. The basic idea is that shell completions could be both (1) shared among shells and (2) moved into the same language that the binary is written in (optionally).
sara7262 7y ago
Cute dog doing exercise like humans http://bit.ly/2Jq3rz1 http://bit.ly/2Jq3rz1 Funny gorilla acting like humans see his video on link below http://bit.ly/2JmTNxr http://bit.ly/2JmTNxr Best way to get rid from dark spots on face just follow this https://zoomtips.blogspot.com/2019/05/Dark-spots.html https://zoomtips.blogspot.com/2019/05/Dark-spots.html Funny panda is playing with friends they are sho cute http://bit.ly/2JvDQF4 http://bit.ly/2JvDQF4
emily2749 7y ago
Cute dog doing exercise like humans http://bit.ly/2Jq3rz1 http://bit.ly/2Jq3rz1 Funny gorilla acting like humans see his video on link below http://bit.ly/2JmTNxr http://bit.ly/2JmTNxr Best way to get rid from dark spots on face just follow this https://zoomtips.blogspot.com/2019/05/Dark-spots.html https://zoomtips.blogspot.com/2019/05/Dark-spots.html Funny panda is playing with friends they are sho cute http://bit.ly/2JvDQF4 http://bit.ly/2JvDQF4
xliiv 7y ago
I did this https://github.com/xliiv/fui https://github.com/xliiv/fui This is something different, but maybe someone could say it's sort of "completions"? Fui also has a feature which allow to copy/dump a whole form (ctrl-k) string which can be pasted to bash.

150 points by JoshMcguigan 7y ago

Author here. This blog post is the result of my puzzling over how auto-complete works in the shell, and subsequently whether shell completion scripts could be written in Rust. It also introduces a new Rust crate, shell_completion[0], which is a library that exposes some low level primitives for writing shell completion scripts in Rust. shell_completion is certainly in its infancy, so I'd be happy to have contributors of any kind (both code and opinions are welcome). [0]: https://github.com/joshmcguigan/shell_completion https://github.com/joshmcguigan/shell_completion

sam0x17 7y ago
This is very cool -- thanks for your work! I wonder if the API for windows auto-completion is sophisticated enough that you could add cross-platform support for this.
- jbreiding 7y ago
  No need to worry if you just use the same shell through WSL. The integration is with the shell not the os.
sbr464 7y ago
I’m curious about your approach, did you have any resources or projects that you found yourself going back to? (Regarding shell completions in general)
- JoshMcguigan 7y ago
  These [0] two [1] sections of the GNU Bash man pages were very useful. I also found that this tutorial [2] provided a useful perspective. I spent a small amount of time looking through the bash-completion [3] project as well as the git bash completions [4]. Generally, my process for writing is google around until I have some base level of knowledge, then building one or more demo applications to ensure my understanding is correct. Then as I'm writing the blog post I'll be refining the demo applications into examples for the blog post and/or library code that I want to open source. [0]: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion https://www.gnu.org/software/bash/manual/html_node/Programma... [1]: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins https://www.gnu.org/software/bash/manual/html_node/Programma... [2]: https://hackaday.com/2018/01/19/linux-fu-custom-bash-command-completion/ https://hackaday.com/2018/01/19/linux-fu-custom-bash-command... [3]: https://github.com/scop/bash-completion https://github.com/scop/bash-completion [4]: https://github.com/git/git/blob/master/contrib/completion/git-completion.bash https://github.com/git/git/blob/master/contrib/completion/gi...
  - sbr464 7y ago
    Thanks for sharing (and @snowe2010)
- snowe2010 7y ago
  I've actually written two separate articles on shell completion, explicitly for subcommands, because the resources regarding subcommands is so lacking. https://tylerthrailkill.com/2019-01-13/writing-zsh-completion-scripts/ https://tylerthrailkill.com/2019-01-13/writing-zsh-completio... https://tylerthrailkill.com/2019-01-19/writing-bash-completion-script-with-subcommands/ https://tylerthrailkill.com/2019-01-19/writing-bash-completi... I need to improve them and write a bit more information, I was writing a library to generate completion scripts for both bash and zsh, so I had to know the differences quite well.

tormeh 7y ago

The ion shell is written in what I believe is pure Rust, and has autocompletion by default.

JoshMcguigan 7y ago
It is not possible for a shell to know how to perform completions for every binary on a users machine. Shell completion scripts define the auto-completion for each specific binary, so for autocomplete to work (well) it needs to be supported both by the shell and by the application. This post is about writing shell completion scripts (the application side, not the shell side).

chubot 7y ago

If I'm understanding correctly, this is a shim that: 1) serializes the fields the bash completion API gives you (when you hit TAB) 2) exports them to Rust, then 3) passes the Rust program's results back to bash? I've been working on similar problems, and I understand why that's appealing. Bash is a pretty bad language to write completion scripts in! It's really bad at string and array processing [1]. ---- Though on a quick glance, if you're exporting $COMP_LINE and $COMP_POINT, then it's making the job of the completion script very hard, since you're passing off the responsibility of parsing shell to it: In other words, if you have something like: mycommand --flag arg\ with\ spaces --flag2=<TAB> Then it's not desirable to pass the entire string. It would be better to parse it and pass: ["mycommand", "--flag", "arg with spaces", "--flag2="] # and indicate index 3 being completed Another common case is: echo $(dirname <TAB> # I don't need echo $( to complete I noticed this deficiency of bash while essentially cloning the bash API for Oil: https://www.oilshell.org/ https://www.oilshell.org/ One consequence of this is that the bash-completion project (maintained separately from bash) actually has an ad hoc bash parser in bash! This is really bad -- unsurprisingly it has obvious bugs, like not being able to complete inside command subs. An ad hoc bash parser in Rust would also be bad. So instead Oil parses shell for you -- since it has a shell parser! -- and then it gives you the partial argv array. So then the plugin is only responsible for what it can reasonably do. ----- Additional, I proposed that this could be made into a protocol that other shells consume. So instead of writing bash completions in Rust, you could write zsh/fish/bash/Oil completions in Rust. https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos... A lot of people are already "scraping" bash completions this way, i.e. to use in zsh or their own shells, so it should not be hard to come up with something "standard", given those working examples. https://github.com/oilshell/oil/wiki/Projects-Already-Doing-Something-Like-Shellac https://github.com/oilshell/oil/wiki/Projects-Already-Doing-... If you're interested in chatting about it please join https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ . I wouldn't say it's the #1 priority now because Oil already emulates bash quite well, but I think a bunch of people are motivated because they 1) don't want to write completions N times, for every shell and 2) Don't want to write completions in the bash language. home page: main page: https://github.com/oilshell/oil/wiki/Shell-Autocompletion https://github.com/oilshell/oil/wiki/Shell-Autocompletion IMO it makes sense to write completions in whatever language the command line tool / flag parser is written in. [1] https://www.oilshell.org/blog/2016/11/06.html https://www.oilshell.org/blog/2016/11/06.html

codetrotter 7y ago

Seeing this got me curious to know if any work has been done on generating completions for applications that make use of the clap command-line argument parser. https://github.com/clap-rs/clap https://github.com/clap-rs/clap https://crates.io/crates/clap https://crates.io/crates/clap Among other things I found this currently open issue that has a lot of discussion and information in it: https://github.com/clap-rs/clap/issues/568 https://github.com/clap-rs/clap/issues/568 It might be of interest to you as well, OP?

JoshMcguigan 7y ago
I did come across this work by clap while doing research for this post. The major difference in strategy is that clap is generating completion scripts in Bash, while shell_completion is a library which helps writing completion scripts in Rust. I think one exciting bit of future potential for shell_completion is it could potentially allow building something like what clap has done using the standard Rust macro system. In any event, thanks for bringing this up.
snowe2010 7y ago
I actually wrote a library that generates completions for both zsh and bash for the Ruby Thor CLI library. https://github.com/snowe2010/fylla/ https://github.com/snowe2010/fylla/ This could be useful in research to make it work for clap. I wrote two articles that kind of highlight the differences in completions between the two, but only in regards to subcommands, since that's what I was trying to cover. https://tylerthrailkill.com/2019-01-19/writing-bash-completion-script-with-subcommands/ https://tylerthrailkill.com/2019-01-19/writing-bash-completi... https://tylerthrailkill.com/2019-01-13/writing-zsh-completion-scripts/ https://tylerthrailkill.com/2019-01-13/writing-zsh-completio...

dmix 7y ago

That sounds useful. I used to write tons of ZSH scripts when I used to obsessively make my Linux desktop experience nice and automated. Writing custom completions always felt awkward for me and unintuitive (like most ZSH programming). Does your shell_completion library work with ZSH? I'm not familiar with how sh/bash differs here from ZSH but I'd image it'd be easy to make compatible. I'm waiting patiently for Evlish to mature so I can use their Go-like (and Go-powered) scripting language and burn my old ZSH scripts in a fire. https://github.com/elves/elvish https://github.com/elves/elvish

chubot 7y ago
If you're using multiple shells like zsh and Elvish, then you might be interested in my proposal below for shell-agnostic autocompletion: https://news.ycombinator.com/item?id=19950892 https://news.ycombinator.com/item?id=19950892 I talked with the author of Elvish about it, and we both want it to happen, but are both working on other things now AFAIK. If you managed to write a working bash/zsh completion script, then kudos :) It's not easy. The basic idea is that shell completions could be both (1) shared among shells and (2) moved into the same language that the binary is written in (optionally).

sara7262 7y ago

Cute dog doing exercise like humans http://bit.ly/2Jq3rz1 http://bit.ly/2Jq3rz1 Funny gorilla acting like humans see his video on link below http://bit.ly/2JmTNxr http://bit.ly/2JmTNxr Best way to get rid from dark spots on face just follow this https://zoomtips.blogspot.com/2019/05/Dark-spots.html https://zoomtips.blogspot.com/2019/05/Dark-spots.html Funny panda is playing with friends they are sho cute http://bit.ly/2JvDQF4 http://bit.ly/2JvDQF4

emily2749 7y ago

xliiv 7y ago

I did this https://github.com/xliiv/fui https://github.com/xliiv/fui This is something different, but maybe someone could say it's sort of "completions"? Fui also has a feature which allow to copy/dump a whole form (ctrl-k) string which can be pasted to bash.