8 ms·

How Bash Completion Works

299 points by cpatuzzo 7y ago

scriptease 7y ago
Awesome
techntoke 7y ago
Is there anything like completion for ash within Alpine containers, or do you have to install Bash?
- unixhero 7y ago
  To be fair. Bash does not have a large footprint.
  - roryrjb 7y ago
    As an alternative, as mentioned already in response to another comment is oksh (https://github.com/ibara/oksh https://github.com/ibara/oksh), a portable version of OpenBSD's pdksh. It's in the same Bourne shell lineage as bash but is lighter and the completion is simpler: you have to customise it if you want it to work for you, but it's very easy and quick to do.
- shakna 7y ago
  I don't believe Almquist ever incorporated tab completion. Korn did, so a number of more limited shells may support it, but probably not ash.
pininja 7y ago
Great tutorial! I’ve always wondered how this is implemented, and how I could add it to my scripts. The interface seems simple enough.
sdan 7y ago
Love the implementation. Can't help but think about another "startup" or project that uses Machine Learning to predict your bash commands... which in most if not all cases is probably not necessary XD.
- dmd 7y ago
  Like this? https://tabnine.com/blog/deep/ https://tabnine.com/blog/deep/
  - sdan 7y ago
    Yikes. Yes. Not sure if other developers agree with me on this: I probably don't need autocompletion, I'm not writing an essay; I'm writing code where every character is a lot more impactful than forgetting a semicolon here or there. For this reason, having autocompletion could probably be an annoyance than helpful.
    - hnlmorg 7y ago
      Personally I'm the exact opposite - weirdly for the same reasons you cited too. When programming I can usually predict what will come next or the compiler will usually slap me if I've done something dumb. Whereas it's easier to miss something in a shell because everything is a little more "golfed" and mistakes can go unnoticed for a while if you're being down-right careless. I'd love it if there was a way to do a "test run" of a command line where it prints what it would do without actually touching the file system (for example).
      - sdan 7y ago
        Pretty sure a simple "dry run" is what you're looking for. Not sure what programs support it, but I know Docker for one does.
        hnlmorg 7y ago
        Sorry yes, "dry run". There are definitely ways of doing it on a command by command basis but what I was more thinking about was a dry run on an entire pipeline. It's not a use case I tend to run into much these days though because there are usually better tools.
karatinversion 7y ago
Nice article, it’s usually hard to keep a reader’s interest when writing about shell. I will nitpick that COMP_LINE et al are shell variables, not environment variables - you can tell by the fact you don’t need to export the “return value”. (Scare quotes, as the function’s exit code, manually specifiable with “return NN”, is also called its return value.)
- p_ronto 7y ago
  I think they are env vars. complete is a bash built-in. It is executed in the process of the shell. You only need to export for child processes.
djohnston 7y ago
Cool read. I was recently wondering how this worked. I was using zsh but I assume the implementation is similar.
mdaniel 7y ago
> `function _fizzbuzz () {` I find that extra keyword distracting, when [POSIX.2 doesn't require it][1] and some folks going so far as to say to [actively avoid it][2] Is there some good reason for it, or people are just used to a programming language requiring `fun` before a declaration, so it's programmer muscle memory? _1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... _2: http://mywiki.wooledge.org/BashPitfalls#function_foo.28.29 http://mywiki.wooledge.org/BashPitfalls#function_foo.28.29
- eurg 7y ago
  Simple keywords for definitions can make grepping for stuff much easier. Given that shell allows other forms, it isn't that helpful in practice. I still use it because my eyes find start-of-definition faster with common patterns that are not just special-chars. YMMV. Edit: I usually don't care for portability, and just use bash. Shell is so ugly already, I refuse to write my personal stuff in portability hell mode. Rather level up my Civilization skills instead.
nfoz 7y ago
I always disable shell completion after it burned me a few times: 1. Completion that blocks the shell, by doing a lookup across a network (e.g. to complete a remote path). Can hang for several seconds. 2. Completion that gives me a misleading/incorrect view of the filesystem, by "intelligently" filtering which files/filetypes it will let me complete. For example, if I have foo.mp3 and foo.txt, and a media-player command tab-completes and only "sees" the mp3, but I actually wanted to know that the .txt was there as well (and in some cases, open it with the media player!) 3. When the completion has a wrong view of the options provided by a program (e.g. completion is not aware of some useful options). Instead of looking at the man-page, I've been tricked by a bad completion-set. I would love a better shell with good completion, but I need to avoid these types of problems. Does anyone else run into this?
- gpderetta 7y ago
  I've run into all of these, but, FWIW, I still prefer smart autocompletion in bash by a large margin. Somehow I learned to avoid the bad areas I guess.
- roryrjb 7y ago
  Some of this could be down to the completions that are provided. A package such as bash-completions comes with completions for a lot of common software. One specific solution to your problem, well a suggestion is to use oksh (https://github.com/ibara/oksh https://github.com/ibara/oksh) a portable version of pdksh (public domain Korn shell) that comes from OpenBSD. The reason why I recommend this shell is that it has programmable completion, although it is much simpler than what's provided by bash and that aside from the built in file name completion it won't come with anything else and you won't find (aside from people's dotfiles on Github) any packages that provide you with anything. Basically you have to customise it to your needs, but because the completion is so simple, whenever you find yourself in a situation where you'd want completion, spend a few minutes adding it to your .kshrc and move on.
- hnlmorg 7y ago
  Coincidentally just this week I've working to resolve point 1 in my own shell (https://github.com/lmorg/murex https://github.com/lmorg/murex). The compromise I came up with was: - The shell would attempt to return all suggestions immediately - However a subset of functions which are known to be slower queries (like recursive directory look ups on larger file system hierarchies) are given a "soft timeout" -- where after that timeout is reached, the slower faster subset of functions are returned as the suggestions while the slower subset continues to run in the background - The slower subset are also given a "hard timeout" where when that point is reached, if the function still hasn't completed then it is just killed. Any results is has produced (if any) by that point are appended to the auto-completion suggestions. - If the slower subset finished before the hard timeout then it is appended to the suggestions. If it finishes before the soft timeout then it's just part of the suggestions. At the moment it's only been implemented for file and directory suggestions. The fast function is just any files and directories in the current directory level; where as the slow function will return a larger directory hierarchy (for quick navigating akin to fzf). But the plan is to extend it further to support other dynamic completers as well. This wouldn't completely solve your point though. If the "fast" query actually runs slow (eg you're trying to complete when inside a network mounted filesystem when the network is dropping) then the shell would still hang. At some point I'll add timeouts on them as well. But I am inching towards that goal. Be warned though, because this feature is less than a week old, it's still only in a feature branch. :) Your point on 3 is apt too. One of the things I built early on was man page parsing for auto-completion suggestions. I've since learned that I'm not alone in that regard either (Fish does it as well -- and from some of the demo's I've watched it looks like they've done a nicer job there too). More recently I've also been writing tools that parse binaries for flags as well (in instances where you download a statically compiled binary - eg terraform - and thus don't have any corresponding man pages). That work is very much in it's infancy though.
vunie 7y ago
My issue with bash completion is that it requires a completion script (i.e executing `complete`) for each command you want it to complete. The shell cannot automatically deduce appropriate completions when possible. This problem is not specific to bash. Fish and other shells can't automatically complete commands either. There is simply no standard way to detect what type of auto-completion a command supports. I know this comment is a not about the OP, but I've wasted so much time dealing with this that I feel compelled to post this here. I wish shell developers would form some sort of consortium (like XDG) where they can agree on standard solutions to issues like this. No one benefits from having developers waste time manually writing and testing completion scripts for each project (times the number of shells they want to support). Side note: I remember that there was a project that modified bash internals to detect automatic completion support. It did this by scanning for the presence of a magic string in the first N bytes of a file (or in some ELF section of a binary). If that magic string was present, bash would automatically generate a completion set by passing `--_complete` to the command. I think this is a simple and elegant solution to this problem and one I hope shell developers would consider.
- shakna 7y ago
  > The shell cannot automatically deduce appropriate completions when possible. I don't think this even _can_ be possible for a large number of programs. For some small number of programs it _might_ be possible, if somehow (ignoring how for now) you were exposed some standard interfaces like getopt or argparse and didn't have to care about argument ordering. But a ton of other programs, with extremely complicated interfaces, simply wouldn't be possible. And these include commonly used programs. Auto-completing awk would require a full language parser at the least, for example. Another, ffmpeg, has a dizzying array of options, and the order of those options can completely change the intent of the command, and what options are allowed to follow without being ignored silently. Even if we had perfect interface detection, we could only probably generate a subset of auto-completion options because that particular problem might not _always_ be solveable.
  - vunie 7y ago
    You raise a good point about solvable completion, and one that I agree with. However, automatic completion support doesn't have to cover every use case. It only needs to be good enough. Complex applications can fall back to providing their own completion scripts. One point I'm not clear on is why you think such a scheme wouldn't be feasible for a large number of commands. Completion wouldn't be invoked until the fist argument (the command's name) is complete just like how bash doesn't scan the system's completion directory until it knows what command is being invoked. And as for completion for the command's arguments, the command is only ran once to generate the list of completion candidates. Note that there are projects that already do exactly this like [1] . [1]:https://github.com/kislyuk/argcomplete https://github.com/kislyuk/argcomplete
jsnwhte 7y ago
Thank you!
Someone 7y ago
TLDR version: duct tape and hundreds of people helping prevent the contraption from falling apart :-) That’s not the scary part, though. The scary part is that those completion scripts run with the user’s privileges, unsandboxed. I could find only a single CVE (https://www.cvedetails.com/cve/CVE-2018-7738/ https://www.cvedetails.com/cve/CVE-2018-7738/), but I think it would be wise to sandbox these scripts.
- shakna 7y ago
  If you sandbox, can you actually guarantee completion? Some more complicated commands might depend on the contents of a file, or the permissions of the file, and an environment variable at the same time to find valid options. Others may require accessing a list of processes running under the active user, which could only be accessible when running as the user in some circumstances. Tacking on any kind of permission system will probably break thirty-odd years of programs, and would likely be a hack because of how POSIX is expected to behave. Duct tape it certainly is, but backwards compatibility isn't something to hate either. We run code all day everyday. That completion runs code shouldn't be a surprise, and you need to judge whether or not you trust it before using it.

299 points by cpatuzzo 7y ago

scriptease 7y ago

Awesome

techntoke 7y ago

Is there anything like completion for ash within Alpine containers, or do you have to install Bash?

unixhero 7y ago
To be fair. Bash does not have a large footprint.
- roryrjb 7y ago
  As an alternative, as mentioned already in response to another comment is oksh (https://github.com/ibara/oksh https://github.com/ibara/oksh), a portable version of OpenBSD's pdksh. It's in the same Bourne shell lineage as bash but is lighter and the completion is simpler: you have to customise it if you want it to work for you, but it's very easy and quick to do.
shakna 7y ago
I don't believe Almquist ever incorporated tab completion. Korn did, so a number of more limited shells may support it, but probably not ash.

pininja 7y ago

Great tutorial! I’ve always wondered how this is implemented, and how I could add it to my scripts. The interface seems simple enough.

sdan 7y ago

Love the implementation. Can't help but think about another "startup" or project that uses Machine Learning to predict your bash commands... which in most if not all cases is probably not necessary XD.

dmd 7y ago
Like this? https://tabnine.com/blog/deep/ https://tabnine.com/blog/deep/
- sdan 7y ago
  Yikes. Yes. Not sure if other developers agree with me on this: I probably don't need autocompletion, I'm not writing an essay; I'm writing code where every character is a lot more impactful than forgetting a semicolon here or there. For this reason, having autocompletion could probably be an annoyance than helpful.
  - hnlmorg 7y ago
    Personally I'm the exact opposite - weirdly for the same reasons you cited too. When programming I can usually predict what will come next or the compiler will usually slap me if I've done something dumb. Whereas it's easier to miss something in a shell because everything is a little more "golfed" and mistakes can go unnoticed for a while if you're being down-right careless. I'd love it if there was a way to do a "test run" of a command line where it prints what it would do without actually touching the file system (for example).
    - sdan 7y ago
      Pretty sure a simple "dry run" is what you're looking for. Not sure what programs support it, but I know Docker for one does.
      - hnlmorg 7y ago
        Sorry yes, "dry run". There are definitely ways of doing it on a command by command basis but what I was more thinking about was a dry run on an entire pipeline. It's not a use case I tend to run into much these days though because there are usually better tools.

karatinversion 7y ago

Nice article, it’s usually hard to keep a reader’s interest when writing about shell. I will nitpick that COMP_LINE et al are shell variables, not environment variables - you can tell by the fact you don’t need to export the “return value”. (Scare quotes, as the function’s exit code, manually specifiable with “return NN”, is also called its return value.)

p_ronto 7y ago
I think they are env vars. complete is a bash built-in. It is executed in the process of the shell. You only need to export for child processes.

djohnston 7y ago

Cool read. I was recently wondering how this worked. I was using zsh but I assume the implementation is similar.

mdaniel 7y ago

> `function _fizzbuzz () {` I find that extra keyword distracting, when [POSIX.2 doesn't require it][1] and some folks going so far as to say to [actively avoid it][2] Is there some good reason for it, or people are just used to a programming language requiring `fun` before a declaration, so it's programmer muscle memory? _1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... _2: http://mywiki.wooledge.org/BashPitfalls#function_foo.28.29 http://mywiki.wooledge.org/BashPitfalls#function_foo.28.29

eurg 7y ago
Simple keywords for definitions can make grepping for stuff much easier. Given that shell allows other forms, it isn't that helpful in practice. I still use it because my eyes find start-of-definition faster with common patterns that are not just special-chars. YMMV. Edit: I usually don't care for portability, and just use bash. Shell is so ugly already, I refuse to write my personal stuff in portability hell mode. Rather level up my Civilization skills instead.

nfoz 7y ago

I always disable shell completion after it burned me a few times: 1. Completion that blocks the shell, by doing a lookup across a network (e.g. to complete a remote path). Can hang for several seconds. 2. Completion that gives me a misleading/incorrect view of the filesystem, by "intelligently" filtering which files/filetypes it will let me complete. For example, if I have foo.mp3 and foo.txt, and a media-player command tab-completes and only "sees" the mp3, but I actually wanted to know that the .txt was there as well (and in some cases, open it with the media player!) 3. When the completion has a wrong view of the options provided by a program (e.g. completion is not aware of some useful options). Instead of looking at the man-page, I've been tricked by a bad completion-set. I would love a better shell with good completion, but I need to avoid these types of problems. Does anyone else run into this?

gpderetta 7y ago
I've run into all of these, but, FWIW, I still prefer smart autocompletion in bash by a large margin. Somehow I learned to avoid the bad areas I guess.
roryrjb 7y ago
Some of this could be down to the completions that are provided. A package such as bash-completions comes with completions for a lot of common software. One specific solution to your problem, well a suggestion is to use oksh (https://github.com/ibara/oksh https://github.com/ibara/oksh) a portable version of pdksh (public domain Korn shell) that comes from OpenBSD. The reason why I recommend this shell is that it has programmable completion, although it is much simpler than what's provided by bash and that aside from the built in file name completion it won't come with anything else and you won't find (aside from people's dotfiles on Github) any packages that provide you with anything. Basically you have to customise it to your needs, but because the completion is so simple, whenever you find yourself in a situation where you'd want completion, spend a few minutes adding it to your .kshrc and move on.
hnlmorg 7y ago
Coincidentally just this week I've working to resolve point 1 in my own shell (https://github.com/lmorg/murex https://github.com/lmorg/murex). The compromise I came up with was: - The shell would attempt to return all suggestions immediately - However a subset of functions which are known to be slower queries (like recursive directory look ups on larger file system hierarchies) are given a "soft timeout" -- where after that timeout is reached, the slower faster subset of functions are returned as the suggestions while the slower subset continues to run in the background - The slower subset are also given a "hard timeout" where when that point is reached, if the function still hasn't completed then it is just killed. Any results is has produced (if any) by that point are appended to the auto-completion suggestions. - If the slower subset finished before the hard timeout then it is appended to the suggestions. If it finishes before the soft timeout then it's just part of the suggestions. At the moment it's only been implemented for file and directory suggestions. The fast function is just any files and directories in the current directory level; where as the slow function will return a larger directory hierarchy (for quick navigating akin to fzf). But the plan is to extend it further to support other dynamic completers as well. This wouldn't completely solve your point though. If the "fast" query actually runs slow (eg you're trying to complete when inside a network mounted filesystem when the network is dropping) then the shell would still hang. At some point I'll add timeouts on them as well. But I am inching towards that goal. Be warned though, because this feature is less than a week old, it's still only in a feature branch. :) Your point on 3 is apt too. One of the things I built early on was man page parsing for auto-completion suggestions. I've since learned that I'm not alone in that regard either (Fish does it as well -- and from some of the demo's I've watched it looks like they've done a nicer job there too). More recently I've also been writing tools that parse binaries for flags as well (in instances where you download a statically compiled binary - eg terraform - and thus don't have any corresponding man pages). That work is very much in it's infancy though.

vunie 7y ago

My issue with bash completion is that it requires a completion script (i.e executing `complete`) for each command you want it to complete. The shell cannot automatically deduce appropriate completions when possible. This problem is not specific to bash. Fish and other shells can't automatically complete commands either. There is simply no standard way to detect what type of auto-completion a command supports. I know this comment is a not about the OP, but I've wasted so much time dealing with this that I feel compelled to post this here. I wish shell developers would form some sort of consortium (like XDG) where they can agree on standard solutions to issues like this. No one benefits from having developers waste time manually writing and testing completion scripts for each project (times the number of shells they want to support). Side note: I remember that there was a project that modified bash internals to detect automatic completion support. It did this by scanning for the presence of a magic string in the first N bytes of a file (or in some ELF section of a binary). If that magic string was present, bash would automatically generate a completion set by passing `--_complete` to the command. I think this is a simple and elegant solution to this problem and one I hope shell developers would consider.

shakna 7y ago
> The shell cannot automatically deduce appropriate completions when possible. I don't think this even _can_ be possible for a large number of programs. For some small number of programs it _might_ be possible, if somehow (ignoring how for now) you were exposed some standard interfaces like getopt or argparse and didn't have to care about argument ordering. But a ton of other programs, with extremely complicated interfaces, simply wouldn't be possible. And these include commonly used programs. Auto-completing awk would require a full language parser at the least, for example. Another, ffmpeg, has a dizzying array of options, and the order of those options can completely change the intent of the command, and what options are allowed to follow without being ignored silently. Even if we had perfect interface detection, we could only probably generate a subset of auto-completion options because that particular problem might not _always_ be solveable.
- vunie 7y ago
  You raise a good point about solvable completion, and one that I agree with. However, automatic completion support doesn't have to cover every use case. It only needs to be good enough. Complex applications can fall back to providing their own completion scripts. One point I'm not clear on is why you think such a scheme wouldn't be feasible for a large number of commands. Completion wouldn't be invoked until the fist argument (the command's name) is complete just like how bash doesn't scan the system's completion directory until it knows what command is being invoked. And as for completion for the command's arguments, the command is only ran once to generate the list of completion candidates. Note that there are projects that already do exactly this like [1] . [1]:https://github.com/kislyuk/argcomplete https://github.com/kislyuk/argcomplete

jsnwhte 7y ago

Thank you!

Someone 7y ago

TLDR version: duct tape and hundreds of people helping prevent the contraption from falling apart :-) That’s not the scary part, though. The scary part is that those completion scripts run with the user’s privileges, unsandboxed. I could find only a single CVE (https://www.cvedetails.com/cve/CVE-2018-7738/ https://www.cvedetails.com/cve/CVE-2018-7738/), but I think it would be wise to sandbox these scripts.

shakna 7y ago
If you sandbox, can you actually guarantee completion? Some more complicated commands might depend on the contents of a file, or the permissions of the file, and an environment variable at the same time to find valid options. Others may require accessing a list of processes running under the active user, which could only be accessible when running as the user in some circumstances. Tacking on any kind of permission system will probably break thirty-odd years of programs, and would likely be a hack because of how POSIX is expected to behave. Duct tape it certainly is, but backwards compatibility isn't something to hate either. We run code all day everyday. That completion runs code shouldn't be a surprise, and you need to judge whether or not you trust it before using it.