5 ms·

Carapace: A multi-shell completion library and binary

94 points by cab404 2y ago

bbkane 2y ago
I tried this but couldn't get the completions to load. I think it's a really good idea though (much faster than zsh script completions), so hopefully soon I can retry the install and either get it working or open an issue. One design aspect I do question is that completions are baked into the library. It'd be neat to have them in SQLite or something that could be updated independently (not sure how feasible that is though. I've never been any good at querying graphs like flag/command completions in SQLite).
- FrozenCow 2y ago
  Completions need to do arbitrary computation. A command like `git switch` needs to find what branches are available in the repository.
  - bbkane 2y ago
    True, but you can represent that in the db as a a CLI invocation to run in a subshell. The big gain from something like carapace or my theoretical SQLite-based completion system is faster startup time. I had to remove zsh-completions from my shell setup as it added too much to the startup time (https://github.com/bbkane/dotfiles/blob/master/zsh/README_notes.md#install-zsh-completions---replaced-with-carapace-bin https://github.com/bbkane/dotfiles/blob/master/zsh/README_no...)
- bPspGiJT8Y 2y ago
  > much faster than zsh script completions I'm curious where did you get that from. Edit: I read the note in your dotfiles repo. Yes calling `compinit` on each shell invocation is going to be really slow. That's not how you're supposed to do it, you could at least add the `-C` flag to cache the completions. Ideally you'd also use `zcompile` to compile the cache to ZSH word code. This puts my completions initializing time at ~20ms on a lower/mid-end laptop. Additionally you can do the trick `fish` does and defer the initialization of completions until the first hit of Tab key, so the impact on shell startup time is exactly 0.
  - bongobingo1 2y ago
    I have had this snippet in my .zshrc for years, autoload -U compinit && compinit { # Compile the completion dump to increase startup speed. Run in background. zcompdump="${ZDOTDIR:-$HOME}/.zcompdump" if [[ -s "$zcompdump" && (! -s "${zcompdump}.zwc" || "$zcompdump" -nt "${zcompdump}.zwc") ]]; then # if zcompdump file exists, and we don't have a compiled version or the # dump file is newer than the compiled file zcompile "$zcompdump" fi } &! Using `zmodload zsh/zprof`, I can see about 50% (16ms) of my start up is compinit (its about 28ms without the `zcompile`). Do you have any pointers for the "load on tab" idea? I didn't turn up any good results in DDG and LLMs were just hallucinating. BTW I believe `-C` will disable some cache checking, caching is enabled by default > To speed up the running of compinit, it can be made to produce a dumped configuration that will be read in on future invocations; this is the default, but can be turned off by calling compinit with the option -D. > ... > ... The check performed to see if there are new functions can be omitted by giving the option -C. In this case the dump file will only be created if there isn't one already.
    - bongobingo1 2y ago
      Unable to edit, but this is how I handled lazy loading the completions { # load compinit and rebind ^I (tab) to expand-or-complete, then compile # completions as bytecode if needed. lazyload-compinit() { autoload -Uz compinit # compinit will automatically cache completions to ~/.zcompdump compinit bindkey "^I" expand-or-complete { zcompdump="${ZDOTDIR:-$HOME}/.zcompdump" # if zcompdump file exists, and we don't have a compiled version or the # dump file is newer than the compiled file, update the bytecode. if [[ -s "$zcompdump" && (! -s "${zcompdump}.zwc" || "$zcompdump" -nt "${zcompdump}.zwc") ]]; then zcompile "$zcompdump" fi } &! # pretend we called this directly, instead of the lazy loader zle expand-or-complete } # mark the function as a zle widget zle -N lazyload-compinit bindkey "^I" lazyload-compinit }
    - bPspGiJT8Y 2y ago
      > BTW I believe `-C` will disable some cache checking, caching is enabled by default You're right, my memory has let me down. > Do you have any pointers for the "load on tab" idea? I didn't turn up any good results in DDG and LLMs were just hallucinating. The simplest implementation would be something like bindkey ^I init_completions init_completions () { # ... init logic here ... # rebind tab to complete bindkey ^I complete-word # actually do complete the initial request zle complete-word } Edit: I see now you already figured it out, yeah that's exactly what I meant
sigoden 2y ago
Other similar projects: https://github.com/sigoden/argc-completions https://github.com/sigoden/argc-completions https://github.com/microsoft/inshellisense https://github.com/microsoft/inshellisense
jchook 2y ago
> Custom completions can be defined using yaml files. The concept is awesome but it's curious to see YAML at the core of this project, rather than a Turing-complete, embeddable language like Lua.
- sevagh 2y ago
  YAML is "good enough" for a vast range of tools. It's a good thing that these maintainers are focusing on their core competency/goal of the project and not pandering to niche HN opinions on the stuff that doesn't matter.
arpowers 2y ago
Reminds of of Project Hail Mary. Never heard the word "carapace" before that book...
- Aachen 2y ago
  I suspected Brandon Sanderson made it up for his Mistborn series until hearing it in Project Hail Mary also. This is actually why I clicked the thread: was curious if anyone would make the reference!
- DeathArrow 2y ago
  In Romanian "carapace" means shell. It seems to mean the same in English, too, but I presume is seldom used: https://en.wikipedia.org/wiki/Carapace https://en.wikipedia.org/wiki/Carapace
  - drcongo 2y ago
    It's not uncommon in English, though I suspect it's most often used in technical fields like entomology.
    - madmountaingoat 2y ago
      I'm pretty sure you encounter in in science fiction as well, where some bug like alien has a carapace or a soldier has some armored exo-skeleton described by the word. Though I cannot think of any specific examples.
      - myrmidon 2y ago
        Quite common in videogames, too. A specific example would be the Carapace (armor) upgrades in Starcraft 2 (one faction, the Zerg, are basically insects). Also sometimes used for (humanoid) armor crafted from shells (fantasy settings). I think this is MUCH more prevalent in scifi/fantasy targeted at male audiences, and the data confirms it [1] [1]: https://osf.io/g4xrt/# https://osf.io/g4xrt/#!
logicprog 2y ago
I tried to use this to get better completions with nushell, but it just doesn't have the breadth and depth of completions that fish has as of yet, nor does it have automatic completion generation from manages (although it does have a tool for that) so I eventually gave up and just made fish itself my nushell's completion engine. However I'll be keeping my eye on it, because it's a really cool idea to have a shared completion engine for all shells, so that everyone can pool their work, and because of course having to install two entire shells just to use one of them is a bit silly so eventually I'd like to make the switch.
- bPspGiJT8Y 2y ago
  > so I eventually gave up and just made fish itself my nushell's completion engine That's an interesting idea. Do you have a link to show how you did this?
  - logicprog 2y ago
    Yeah! It's an officially supported thing with nushell, here are the docs: https://www.nushell.sh/cookbook/external_completers.html#fish-completer https://www.nushell.sh/cookbook/external_completers.html#fis... The completions don't work sometimes (for instance you don't get completion for some commands when running them with sudo) and it doesn't fall back on providing file completions if it doesn't know what other completions to provide, which is a bit annoying, but it generally works and I'm sure there are ways to fix that too.
Per_Bothner 2y ago
Completion for program P should be written and maintained by the "owner" of program P - and installed with program P. This is of course difficult when there are many different "shells" that each have their own "language" for specifying completions. A multi-shell completion library can help with this problem. To me it make sense that completion for program P should be handled by program P itself. That way, completions are unlikely to get out of sync with the application, and the completion handler can use the same option parser as the application. A way to do this is to use a special "hidden" switch to request completion. Specifically the DomTerm terminal emulator (https://domterm.org https://domterm.org) handles its own completions. Bash allows you to register a command that handles completions for some other command. The following tells bash that to handle completions for the domterm command it should call domterm with the magic "#complete-for-bash" option followed by the existing line and position. complete -o nospace -C 'domterm "#complete-for-bash" "$COMP_LINE" "$COMP_POINT"' domterm When domterm is run with "#complete-for-bash" as the first argument (chosen to avoids accidental use) it processes the remaining arguments, and prints out the completion suggestions to bash. I don't know how to do something similar for other shells; suggestions welcome. Some other shells (such as Fish) include help text as part of the suggested completions. Handling that would require a more complex protocol. Perhaps one that returs a result in JSON format? If a such a protocol were standardized (perhaps invoked by a "#completions-as-json" option?), we just need to solve a final piece of the puzzle: How does a shell know if a command understands the "#completions-as-json" option? My suggestion is that we define a new keyword-value pair in the "desktop" file specification; the shell would search for a P.desktop file,

94 points by cab404 2y ago

I tried this but couldn't get the completions to load. I think it's a really good idea though (much faster than zsh script completions), so hopefully soon I can retry the install and either get it working or open an issue. One design aspect I do question is that completions are baked into the library. It'd be neat to have them in SQLite or something that could be updated independently (not sure how feasible that is though. I've never been any good at querying graphs like flag/command completions in SQLite).

FrozenCow 2y ago
Completions need to do arbitrary computation. A command like `git switch` needs to find what branches are available in the repository.
- bbkane 2y ago
  True, but you can represent that in the db as a a CLI invocation to run in a subshell. The big gain from something like carapace or my theoretical SQLite-based completion system is faster startup time. I had to remove zsh-completions from my shell setup as it added too much to the startup time (https://github.com/bbkane/dotfiles/blob/master/zsh/README_notes.md#install-zsh-completions---replaced-with-carapace-bin https://github.com/bbkane/dotfiles/blob/master/zsh/README_no...)
bPspGiJT8Y 2y ago
> much faster than zsh script completions I'm curious where did you get that from. Edit: I read the note in your dotfiles repo. Yes calling `compinit` on each shell invocation is going to be really slow. That's not how you're supposed to do it, you could at least add the `-C` flag to cache the completions. Ideally you'd also use `zcompile` to compile the cache to ZSH word code. This puts my completions initializing time at ~20ms on a lower/mid-end laptop. Additionally you can do the trick `fish` does and defer the initialization of completions until the first hit of Tab key, so the impact on shell startup time is exactly 0.
- bongobingo1 2y ago
  I have had this snippet in my .zshrc for years, autoload -U compinit && compinit { # Compile the completion dump to increase startup speed. Run in background. zcompdump="${ZDOTDIR:-$HOME}/.zcompdump" if [[ -s "$zcompdump" && (! -s "${zcompdump}.zwc" || "$zcompdump" -nt "${zcompdump}.zwc") ]]; then # if zcompdump file exists, and we don't have a compiled version or the # dump file is newer than the compiled file zcompile "$zcompdump" fi } &! Using `zmodload zsh/zprof`, I can see about 50% (16ms) of my start up is compinit (its about 28ms without the `zcompile`). Do you have any pointers for the "load on tab" idea? I didn't turn up any good results in DDG and LLMs were just hallucinating. BTW I believe `-C` will disable some cache checking, caching is enabled by default > To speed up the running of compinit, it can be made to produce a dumped configuration that will be read in on future invocations; this is the default, but can be turned off by calling compinit with the option -D. > ... > ... The check performed to see if there are new functions can be omitted by giving the option -C. In this case the dump file will only be created if there isn't one already.
  - bongobingo1 2y ago
    Unable to edit, but this is how I handled lazy loading the completions { # load compinit and rebind ^I (tab) to expand-or-complete, then compile # completions as bytecode if needed. lazyload-compinit() { autoload -Uz compinit # compinit will automatically cache completions to ~/.zcompdump compinit bindkey "^I" expand-or-complete { zcompdump="${ZDOTDIR:-$HOME}/.zcompdump" # if zcompdump file exists, and we don't have a compiled version or the # dump file is newer than the compiled file, update the bytecode. if [[ -s "$zcompdump" && (! -s "${zcompdump}.zwc" || "$zcompdump" -nt "${zcompdump}.zwc") ]]; then zcompile "$zcompdump" fi } &! # pretend we called this directly, instead of the lazy loader zle expand-or-complete } # mark the function as a zle widget zle -N lazyload-compinit bindkey "^I" lazyload-compinit }
  - bPspGiJT8Y 2y ago
    > BTW I believe `-C` will disable some cache checking, caching is enabled by default You're right, my memory has let me down. > Do you have any pointers for the "load on tab" idea? I didn't turn up any good results in DDG and LLMs were just hallucinating. The simplest implementation would be something like bindkey ^I init_completions init_completions () { # ... init logic here ... # rebind tab to complete bindkey ^I complete-word # actually do complete the initial request zle complete-word } Edit: I see now you already figured it out, yeah that's exactly what I meant

sigoden 2y ago

Other similar projects: https://github.com/sigoden/argc-completions https://github.com/sigoden/argc-completions https://github.com/microsoft/inshellisense https://github.com/microsoft/inshellisense

jchook 2y ago

> Custom completions can be defined using yaml files. The concept is awesome but it's curious to see YAML at the core of this project, rather than a Turing-complete, embeddable language like Lua.

sevagh 2y ago
YAML is "good enough" for a vast range of tools. It's a good thing that these maintainers are focusing on their core competency/goal of the project and not pandering to niche HN opinions on the stuff that doesn't matter.

arpowers 2y ago

Reminds of of Project Hail Mary. Never heard the word "carapace" before that book...

Aachen 2y ago
I suspected Brandon Sanderson made it up for his Mistborn series until hearing it in Project Hail Mary also. This is actually why I clicked the thread: was curious if anyone would make the reference!
DeathArrow 2y ago
In Romanian "carapace" means shell. It seems to mean the same in English, too, but I presume is seldom used: https://en.wikipedia.org/wiki/Carapace https://en.wikipedia.org/wiki/Carapace
- drcongo 2y ago
  It's not uncommon in English, though I suspect it's most often used in technical fields like entomology.
  - madmountaingoat 2y ago
    I'm pretty sure you encounter in in science fiction as well, where some bug like alien has a carapace or a soldier has some armored exo-skeleton described by the word. Though I cannot think of any specific examples.
    - myrmidon 2y ago
      Quite common in videogames, too. A specific example would be the Carapace (armor) upgrades in Starcraft 2 (one faction, the Zerg, are basically insects). Also sometimes used for (humanoid) armor crafted from shells (fantasy settings). I think this is MUCH more prevalent in scifi/fantasy targeted at male audiences, and the data confirms it [1] [1]: https://osf.io/g4xrt/# https://osf.io/g4xrt/#!

logicprog 2y ago

I tried to use this to get better completions with nushell, but it just doesn't have the breadth and depth of completions that fish has as of yet, nor does it have automatic completion generation from manages (although it does have a tool for that) so I eventually gave up and just made fish itself my nushell's completion engine. However I'll be keeping my eye on it, because it's a really cool idea to have a shared completion engine for all shells, so that everyone can pool their work, and because of course having to install two entire shells just to use one of them is a bit silly so eventually I'd like to make the switch.

bPspGiJT8Y 2y ago
> so I eventually gave up and just made fish itself my nushell's completion engine That's an interesting idea. Do you have a link to show how you did this?
- logicprog 2y ago
  Yeah! It's an officially supported thing with nushell, here are the docs: https://www.nushell.sh/cookbook/external_completers.html#fish-completer https://www.nushell.sh/cookbook/external_completers.html#fis... The completions don't work sometimes (for instance you don't get completion for some commands when running them with sudo) and it doesn't fall back on providing file completions if it doesn't know what other completions to provide, which is a bit annoying, but it generally works and I'm sure there are ways to fix that too.

Per_Bothner 2y ago

Completion for program P should be written and maintained by the "owner" of program P - and installed with program P. This is of course difficult when there are many different "shells" that each have their own "language" for specifying completions. A multi-shell completion library can help with this problem. To me it make sense that completion for program P should be handled by program P itself. That way, completions are unlikely to get out of sync with the application, and the completion handler can use the same option parser as the application. A way to do this is to use a special "hidden" switch to request completion. Specifically the DomTerm terminal emulator (https://domterm.org https://domterm.org) handles its own completions. Bash allows you to register a command that handles completions for some other command. The following tells bash that to handle completions for the domterm command it should call domterm with the magic "#complete-for-bash" option followed by the existing line and position. complete -o nospace -C 'domterm "#complete-for-bash" "$COMP_LINE" "$COMP_POINT"' domterm When domterm is run with "#complete-for-bash" as the first argument (chosen to avoids accidental use) it processes the remaining arguments, and prints out the completion suggestions to bash. I don't know how to do something similar for other shells; suggestions welcome. Some other shells (such as Fish) include help text as part of the suggested completions. Handling that would require a more complex protocol. Perhaps one that returs a result in JSON format? If a such a protocol were standardized (perhaps invoked by a "#completions-as-json" option?), we just need to solve a final piece of the puzzle: How does a shell know if a command understands the "#completions-as-json" option? My suggestion is that we define a new keyword-value pair in the "desktop" file specification; the shell would search for a P.desktop file,