7 ms·

OpenAutoComplete: Shell-agnostic, cross-platform autocomplete specification

138 points by GamePad64 6y ago

freedomben 6y ago
If we could get some standards defined and adopted, that would be amazing. There are so many CLI tools that could really benefit from auto-complete, but don't yet have it because it is a huge pain to write. Often times the auto-complete is significantly more work than the original tool was, in a couple cases I've written it was an order of magnitude more work to write the shell completion. I have tried writing auto-complete scripts for various tools several times. I'm quite good with bash (if I may say so myself). I've written some fairly advanced scripts and have been doing it a long time. However, despite my existing experience, writing auto completion is still a major pain in the ass. This is an area that really needs improvement. I don't know if this project is the correct answer. I think some good discussion should be had. But I do want to say that the problem they are solving is real.
- GamePad64 6y ago
  I'have just posted something like "request for comments" thing, so we could talk and make a proper solution. I've tried to use PowerShell in Windows, and it was pure pain: the tools, that are written in PS work fine, everything else (like git, InnoSetup, CMake, everything else...) just didn't have autocompletion at all. Maybe, some tools, like git do have shell plugins (like posh-git), but the problem in general still persists. Also, I am using Poetry and Werf utilities, that have built-in generator for autocomplete (bash, zsh and fish), but it doesn't work very well and the developers had to write it by themselves. So, I've tried to make the spec as generic as possible, so it could support current compicated cases, like git (with its infinite number of commands and options), scp (with its remote file listing), some old Windows commands (like "format" for windows95 and so on). Also, this format could be used with a single-pass recursive generator, because each usage pattern is just a JSON array and all its properties can be defined in-place. So, it would be okay to use it with languages, like C, that have no JSON serializer built-in.
- DelightOne 6y ago
  I know what you mean. Go's spf13/cobra will soon have working zsh/bash/fish/powershell autocomplete generation! https://github.com/spf13/cobra/pull/1070 https://github.com/spf13/cobra/pull/1070
gfxgirl 6y ago
wish there was more autocomplete in powershell
- GamePad64 6y ago
  Well, before even writing anything, I've looked up into PS docs, and noticed Register-ArgumentCompleter cmdlet. It could definitely be used for autocompletion purposes, but we need something like a database of autocomplete definitions for common utilities to get started, and a tool to read such a database. So, I propose to create a format for autocomplete definitions and invite everyone to discuss it.
chubot 6y ago
I started down this path last year: https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos... but cut it out of the Oil project (for the forseeable future). Right now Oil takes the approach of running existing bash completion scripts, which solves the problem well enough (but leaves a few things to be desired) If someone comes up with a spec that works in at least one other shell, and some code, I'll be happy to merge it into Oil. Though, reading over this, it looks pretty early, and I don't think the "spec first" approach really works. That is, the spec should be derived from working code, and I believe you have to implement it in more than one shell to tell if it's feasible. Some other links I collected: https://github.com/oilshell/oil/wiki/Shell-Autocompletion https://github.com/oilshell/oil/wiki/Shell-Autocompletion
- GamePad64 6y ago
  Thanks for the reply! I've googled hard, but couldn't find anything like that. The problem I see is that Shellac is dynamic. So, it is pretty intrusive. This is how Click autocompletion works in Python. Also, I don't think, that Shellac is mutually exclusive with OpenAutoComplete. Being a Shell-to-autocompletion server protocol, it could use openautocomplete library as a data source.
  - GamePad64 6y ago
    Also, the accompanying code: https://github.com/openautocomplete/oactool https://github.com/openautocomplete/oactool It currently supports generating completion code for fish (experimental), but zsh and bash are similar to it, but a bit complicated, so it will be implemented later.
  - the_duke 6y ago
    > The problem I see is that Shellac is dynamic So OpenAutoComplete is purely static then? That makes it much less useful. Static completion is nice to have for tools you only use from time to time, or when fishing for that argument name, but if you don't know the name of an argument etc you will often have to use `man` anyway. The real benefit of completions are the dynamic ones, at least for me. If such a spec does not support dynamic completion, a lot of tools will need to do their own thing anyway. I realize that this makes it much more complex though.
Mizza 6y ago
Can't you just parse --help in the background so we as developers don't need to write anything at all?
- saagarjha 6y ago
  That’s not standardized at all, though.
  - Mizza 6y ago
    I don't think that really matters though. I'd wager that you can write 5 or 6 simple parsers and get 99% coverage of commands used.
    - TeddyDD 6y ago
      Fish does something like this (it scans manpages) but generated completion is mostly useless in more complex cli programs (for example: kubectl, docker). So fish is shipped with a bunch of manually written completions for many commands.
      - Mizza 6y ago
        Docker looks like it would be totally doable to me. On every page, all the options are in a standardly structured "Options:" section, all commands are in a standardly structured "Commands" section, and --help and the Options: parser will also work on the subcommands. It won't have typing, but it'd work. Do the most common argument parsing libraries from the standard libraries of the most popular languages and you're mostly there. As a maintainer, I'm sick of having to include all sorts of different stupid autocompleters and think that it should be the job of an intelligent shell.
        dastx 6y ago
        For things like docker simply parsing the help isn't enough. Docker's environment can get pretty complex, pretty quick. There are a thousand different arguments that require different kinds of inputs (think ports, mounts, images id, container I'd and much more). You cannot figure out what the argument type is from just the help or manpages. This is the case for many, many applications.
opk 6y ago
I've written quite a number of shell completions for zsh and the amount of variation in command arguments would require something much more flexible than this basic json specification. This might work fine for listing options and arguments but without more information, completions end up being worse than nothing because with dodgy definitions, you end up breaking things like basic file completion in certain contexts which can be very annoying for users.
novirium 6y ago
This sounds like a great idea, and I'm sure there's people that would be willing to help (me included). Is there somewhere organized already for people to discuss the development, or where to contribute (Gitter, Discord, IRC, etc.)? I couldn't find anything mentioned in the GitHub pages.
dpacmittal 6y ago
Why not use language server specs for something like this?
arendtio 6y ago
A few days ago, I came across kingpin [1] (a library for writing CLI applications in Go) which seems to have some kind of --completion-script-bash option which generates a completion script for you. It might be interesting in this context. [1] https://github.com/alecthomas/kingpin#bashzsh-shell-completion https://github.com/alecthomas/kingpin#bashzsh-shell-completi...
fetbaffe 6y ago
Dreaming about Microsoft implementing proper autocomplete hooks in cmd.exe.

138 points by GamePad64 6y ago

If we could get some standards defined and adopted, that would be amazing. There are so many CLI tools that could really benefit from auto-complete, but don't yet have it because it is a huge pain to write. Often times the auto-complete is significantly more work than the original tool was, in a couple cases I've written it was an order of magnitude more work to write the shell completion. I have tried writing auto-complete scripts for various tools several times. I'm quite good with bash (if I may say so myself). I've written some fairly advanced scripts and have been doing it a long time. However, despite my existing experience, writing auto completion is still a major pain in the ass. This is an area that really needs improvement. I don't know if this project is the correct answer. I think some good discussion should be had. But I do want to say that the problem they are solving is real.

GamePad64 6y ago
I'have just posted something like "request for comments" thing, so we could talk and make a proper solution. I've tried to use PowerShell in Windows, and it was pure pain: the tools, that are written in PS work fine, everything else (like git, InnoSetup, CMake, everything else...) just didn't have autocompletion at all. Maybe, some tools, like git do have shell plugins (like posh-git), but the problem in general still persists. Also, I am using Poetry and Werf utilities, that have built-in generator for autocomplete (bash, zsh and fish), but it doesn't work very well and the developers had to write it by themselves. So, I've tried to make the spec as generic as possible, so it could support current compicated cases, like git (with its infinite number of commands and options), scp (with its remote file listing), some old Windows commands (like "format" for windows95 and so on). Also, this format could be used with a single-pass recursive generator, because each usage pattern is just a JSON array and all its properties can be defined in-place. So, it would be okay to use it with languages, like C, that have no JSON serializer built-in.
DelightOne 6y ago
I know what you mean. Go's spf13/cobra will soon have working zsh/bash/fish/powershell autocomplete generation! https://github.com/spf13/cobra/pull/1070 https://github.com/spf13/cobra/pull/1070

gfxgirl 6y ago

wish there was more autocomplete in powershell

GamePad64 6y ago
Well, before even writing anything, I've looked up into PS docs, and noticed Register-ArgumentCompleter cmdlet. It could definitely be used for autocompletion purposes, but we need something like a database of autocomplete definitions for common utilities to get started, and a tool to read such a database. So, I propose to create a format for autocomplete definitions and invite everyone to discuss it.

chubot 6y ago

I started down this path last year: https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal https://github.com/oilshell/oil/wiki/Shellac-Protocol-Propos... but cut it out of the Oil project (for the forseeable future). Right now Oil takes the approach of running existing bash completion scripts, which solves the problem well enough (but leaves a few things to be desired) If someone comes up with a spec that works in at least one other shell, and some code, I'll be happy to merge it into Oil. Though, reading over this, it looks pretty early, and I don't think the "spec first" approach really works. That is, the spec should be derived from working code, and I believe you have to implement it in more than one shell to tell if it's feasible. Some other links I collected: https://github.com/oilshell/oil/wiki/Shell-Autocompletion https://github.com/oilshell/oil/wiki/Shell-Autocompletion

GamePad64 6y ago
Thanks for the reply! I've googled hard, but couldn't find anything like that. The problem I see is that Shellac is dynamic. So, it is pretty intrusive. This is how Click autocompletion works in Python. Also, I don't think, that Shellac is mutually exclusive with OpenAutoComplete. Being a Shell-to-autocompletion server protocol, it could use openautocomplete library as a data source.
- GamePad64 6y ago
  Also, the accompanying code: https://github.com/openautocomplete/oactool https://github.com/openautocomplete/oactool It currently supports generating completion code for fish (experimental), but zsh and bash are similar to it, but a bit complicated, so it will be implemented later.
- the_duke 6y ago
  > The problem I see is that Shellac is dynamic So OpenAutoComplete is purely static then? That makes it much less useful. Static completion is nice to have for tools you only use from time to time, or when fishing for that argument name, but if you don't know the name of an argument etc you will often have to use `man` anyway. The real benefit of completions are the dynamic ones, at least for me. If such a spec does not support dynamic completion, a lot of tools will need to do their own thing anyway. I realize that this makes it much more complex though.

Mizza 6y ago

Can't you just parse --help in the background so we as developers don't need to write anything at all?

saagarjha 6y ago
That’s not standardized at all, though.
- Mizza 6y ago
  I don't think that really matters though. I'd wager that you can write 5 or 6 simple parsers and get 99% coverage of commands used.
  - TeddyDD 6y ago
    Fish does something like this (it scans manpages) but generated completion is mostly useless in more complex cli programs (for example: kubectl, docker). So fish is shipped with a bunch of manually written completions for many commands.
    - Mizza 6y ago
      Docker looks like it would be totally doable to me. On every page, all the options are in a standardly structured "Options:" section, all commands are in a standardly structured "Commands" section, and --help and the Options: parser will also work on the subcommands. It won't have typing, but it'd work. Do the most common argument parsing libraries from the standard libraries of the most popular languages and you're mostly there. As a maintainer, I'm sick of having to include all sorts of different stupid autocompleters and think that it should be the job of an intelligent shell.
      - dastx 6y ago
        For things like docker simply parsing the help isn't enough. Docker's environment can get pretty complex, pretty quick. There are a thousand different arguments that require different kinds of inputs (think ports, mounts, images id, container I'd and much more). You cannot figure out what the argument type is from just the help or manpages. This is the case for many, many applications.

opk 6y ago

I've written quite a number of shell completions for zsh and the amount of variation in command arguments would require something much more flexible than this basic json specification. This might work fine for listing options and arguments but without more information, completions end up being worse than nothing because with dodgy definitions, you end up breaking things like basic file completion in certain contexts which can be very annoying for users.

novirium 6y ago

This sounds like a great idea, and I'm sure there's people that would be willing to help (me included). Is there somewhere organized already for people to discuss the development, or where to contribute (Gitter, Discord, IRC, etc.)? I couldn't find anything mentioned in the GitHub pages.

dpacmittal 6y ago

Why not use language server specs for something like this?

arendtio 6y ago

A few days ago, I came across kingpin [1] (a library for writing CLI applications in Go) which seems to have some kind of --completion-script-bash option which generates a completion script for you. It might be interesting in this context. [1] https://github.com/alecthomas/kingpin#bashzsh-shell-completion https://github.com/alecthomas/kingpin#bashzsh-shell-completi...

fetbaffe 6y ago

Dreaming about Microsoft implementing proper autocomplete hooks in cmd.exe.