4 ms·

Shell programmable completion highlights how slow the OS community can be to change/improve core tools (shell, coreutils, ttys). Completion should be specified

Shell programmable completion highlights how slow the OS community can be to change/improve core tools (shell, coreutils, ttys). Completion should be specified in the binary, and compatible with all shells. A similar interface could be designed for all scripting languages. It was obvious ~20 years ago that it makes no sense for each shell to specify their own completion system, and yet here we still are.

enriquto 8y ago
as a friend of mine uses to say in these cases: PR or GTFO ;)
mikepurvis 8y ago
You will be a fan of argcomplete, which makes some effort to standardize an approach for in-script completion: https://pypi.org/project/argcomplete https://pypi.org/project/argcomplete Of course, it still has to be bootstrapped using the conventional bash approaches, but it's a first step.
peff 8y ago
This makes sense for some completions (e.g., vanilla lists of options). But often the completion depends on other context that the binary doesn't need to know about. For example, I complete `git grep` patterns based on ctags. There's no reason git should know about ctags; it's only my personal completion that brings the two together.
- powercf 8y ago
  That is probably a very rare case, and any shell-agnostic-completion-protocol could be extensible to handle such a case. But, I think 99% of completions that people want/use on a daily basis could be covered by a machine-readable version of the usual "--help" option (with support for placement of paths, pids, strings, numbers etc.).
  - loa-in-backup 8y ago
    If it doesn't work reliably for all cases, there would soon be inevitable need for the kind of scripted completion we have now.
chubot 8y ago
I'm working in this area right now if you want to help :) http://www.oilshell.org/ http://www.oilshell.org/ I took a survey of what people use here: https://lobste.rs/s/z96uyr/survey_what_shell_completion_scripts_do https://lobste.rs/s/z96uyr/survey_what_shell_completion_scri... That produced a link to this old project which tried to do something like what you want: a completion system that works for multiple shells (in this case, by code generation from a DSL) https://github.com/mbrubeck/compleat https://github.com/mbrubeck/compleat It's easy for you to say "completion should be specified in the binary" without actually specifying HOW. That's not straightforward, although I agree it's a good goal. ----- Right now I'm implementing a bash-compatible completion API, and I'm actually getting pretty far without too much code. git-completion.bash is 3000 lines of code, so it's actually easier to emulate bash (with all its warts) than to write an entirely new completion system AND THEN write 3000+ lines of git completion in that hypothetical system. https://github.com/oilshell/oil/issues/175 https://github.com/oilshell/oil/issues/175 FWIW git is also the biggest completion script in the zsh source tree by a factor of 2 -- ~7000 lines IIRC with the next largest being around ~3000 lines. Eventually Oil should have something better than bash completion, but the first step is taking advantage of ~40K lines of existing code in the bash-completion [1] project. I also looked at emulating zsh but that would be a lot more work than emulating bash. [1] https://github.com/scop/bash-completion https://github.com/scop/bash-completion
- daveFNbuck 8y ago
  Are you building something to support multiple shells, or just yours?
  - chubot 8y ago
    Well, since I'm emulating bash to start, those completion scripts will support multiple shells :) However, I recently learned that zsh also emulates bash, e.g. the complete/compgen/compopt builtins, along with some global variables. So that is already technically true. Still, I'd like to define some kind of "nicer" protocol that binaries can implement to get completion. But that also sort of exists: I learned that when you do "ls --<TAB>" in bash right now, the script actually dynamically greps "ls --help" for flags! This is in contrast to how ZSH works -- it has a bunch of canned completions, which presumably suffer from a version skew problem. Someone out pointed out "npm completion" in this thread, which is new to me (try it; it prints a bash script). git is also adding 'git --listcmds' to help with completion. So the logic is partially in the binary, and partially in the shell completion script. I think I've seen other ad hoc mechanisms along those lines too. Ah yes I recall that there was XML output for the Google flag parser [1], which is meant to be used for completion, although I'm not sure how commonly it's used. So basically there are a lot of different systems. Hoping that someone will come along and produce a universal solution is probably wishful thinking. It's another "boiling the ocean" problem. [1] https://github.com/gflags/gflags https://github.com/gflags/gflags
lzybkr 8y ago
I'm optimistic about binaries providing completions, I think we'll see more of that as tools adopt frameworks like https://github.com/clap-rs/clap https://github.com/clap-rs/clap or https://pypi.org/project/argcomplete/ https://pypi.org/project/argcomplete/ which can provide completions for your favorite shell. But I do think many important/useful binaries will never provide completions, or provide sub-par completion that people will want to improve, so we can't ever count on completions from binaries. Recently I've been thinking some about how the LSP (https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/) might be useful for a CLI. A readline implementation could use something like a LSP for syntax coloring and completion. Imagine having a consistent editing in your favorite shell, repl, and editor.
ComputerGuru 8y ago
PowerShell does this.
jhayward 8y ago
> Completion should be specified in the binary, and compatible with all shells This is somewhat like the approach that DEC took with VAX/VMS in the late 1970's. The command line and arguments/parameters were declared external to the executable and known to the the shell (in this case, DCL) via the SET COMMAND command. It had its own little data description language. The shell did the parsing user interaction (recall, completion, etc) and provided utility functions to the binary to call to obtain params. So while the completion was not technically in the binary, a standard api was used to obtain it and the definitions were created side by side. Worked very nicely and was extensible by 3rd parties and could be used to incorporate "foreign" binaries as well.

enriquto 8y ago

as a friend of mine uses to say in these cases: PR or GTFO ;)

mikepurvis 8y ago

You will be a fan of argcomplete, which makes some effort to standardize an approach for in-script completion: https://pypi.org/project/argcomplete https://pypi.org/project/argcomplete Of course, it still has to be bootstrapped using the conventional bash approaches, but it's a first step.

peff 8y ago

This makes sense for some completions (e.g., vanilla lists of options). But often the completion depends on other context that the binary doesn't need to know about. For example, I complete `git grep` patterns based on ctags. There's no reason git should know about ctags; it's only my personal completion that brings the two together.

powercf 8y ago
That is probably a very rare case, and any shell-agnostic-completion-protocol could be extensible to handle such a case. But, I think 99% of completions that people want/use on a daily basis could be covered by a machine-readable version of the usual "--help" option (with support for placement of paths, pids, strings, numbers etc.).
- loa-in-backup 8y ago
  If it doesn't work reliably for all cases, there would soon be inevitable need for the kind of scripted completion we have now.

chubot 8y ago

I'm working in this area right now if you want to help :) http://www.oilshell.org/ http://www.oilshell.org/ I took a survey of what people use here: https://lobste.rs/s/z96uyr/survey_what_shell_completion_scripts_do https://lobste.rs/s/z96uyr/survey_what_shell_completion_scri... That produced a link to this old project which tried to do something like what you want: a completion system that works for multiple shells (in this case, by code generation from a DSL) https://github.com/mbrubeck/compleat https://github.com/mbrubeck/compleat It's easy for you to say "completion should be specified in the binary" without actually specifying HOW. That's not straightforward, although I agree it's a good goal. ----- Right now I'm implementing a bash-compatible completion API, and I'm actually getting pretty far without too much code. git-completion.bash is 3000 lines of code, so it's actually easier to emulate bash (with all its warts) than to write an entirely new completion system AND THEN write 3000+ lines of git completion in that hypothetical system. https://github.com/oilshell/oil/issues/175 https://github.com/oilshell/oil/issues/175 FWIW git is also the biggest completion script in the zsh source tree by a factor of 2 -- ~7000 lines IIRC with the next largest being around ~3000 lines. Eventually Oil should have something better than bash completion, but the first step is taking advantage of ~40K lines of existing code in the bash-completion [1] project. I also looked at emulating zsh but that would be a lot more work than emulating bash. [1] https://github.com/scop/bash-completion https://github.com/scop/bash-completion

daveFNbuck 8y ago
Are you building something to support multiple shells, or just yours?
- chubot 8y ago
  Well, since I'm emulating bash to start, those completion scripts will support multiple shells :) However, I recently learned that zsh also emulates bash, e.g. the complete/compgen/compopt builtins, along with some global variables. So that is already technically true. Still, I'd like to define some kind of "nicer" protocol that binaries can implement to get completion. But that also sort of exists: I learned that when you do "ls --<TAB>" in bash right now, the script actually dynamically greps "ls --help" for flags! This is in contrast to how ZSH works -- it has a bunch of canned completions, which presumably suffer from a version skew problem. Someone out pointed out "npm completion" in this thread, which is new to me (try it; it prints a bash script). git is also adding 'git --listcmds' to help with completion. So the logic is partially in the binary, and partially in the shell completion script. I think I've seen other ad hoc mechanisms along those lines too. Ah yes I recall that there was XML output for the Google flag parser [1], which is meant to be used for completion, although I'm not sure how commonly it's used. So basically there are a lot of different systems. Hoping that someone will come along and produce a universal solution is probably wishful thinking. It's another "boiling the ocean" problem. [1] https://github.com/gflags/gflags https://github.com/gflags/gflags

lzybkr 8y ago

I'm optimistic about binaries providing completions, I think we'll see more of that as tools adopt frameworks like https://github.com/clap-rs/clap https://github.com/clap-rs/clap or https://pypi.org/project/argcomplete/ https://pypi.org/project/argcomplete/ which can provide completions for your favorite shell. But I do think many important/useful binaries will never provide completions, or provide sub-par completion that people will want to improve, so we can't ever count on completions from binaries. Recently I've been thinking some about how the LSP (https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/) might be useful for a CLI. A readline implementation could use something like a LSP for syntax coloring and completion. Imagine having a consistent editing in your favorite shell, repl, and editor.

ComputerGuru 8y ago

PowerShell does this.

jhayward 8y ago

> Completion should be specified in the binary, and compatible with all shells This is somewhat like the approach that DEC took with VAX/VMS in the late 1970's. The command line and arguments/parameters were declared external to the executable and known to the the shell (in this case, DCL) via the SET COMMAND command. It had its own little data description language. The shell did the parsing user interaction (recall, completion, etc) and provided utility functions to the binary to call to obtain params. So while the completion was not technically in the binary, a standard api was used to obtain it and the definitions were created side by side. Worked very nicely and was extensible by 3rd parties and could be used to incorporate "foreign" binaries as well.