13 ms·

Better bash in 15 minutes

475 points by wsxiaoys 13y ago

rshm 13y ago
'set -o nounset' is a must have. I Just suffered this script from Samsung Printer setting Utility. sudo ./uninstall wiped the /opt DEST_PATH=/opt/$VENDOR/$DEST_DIRNAME #remove destination VERSION=`cat "$DEST_PATH/bin/.version"` if rm -fr "$DEST_PATH"; then echo "INFO: $APP_NAME (ver.$VERSION) has been uninstalled successfully." ...
- laurent123456 13y ago
  Wow, nounset or not, doing a `rm -rf` on a variable without any check is quite irresponsible. Especially if they expect the script to be run as sudo.
- jacquesm 13y ago
  Just checking if .version exists and if it doesn't exiting with an error code would be a huge improvement, checking if $DEST_PATH is set is a must as well. Scary to see a big company like Samsung release code this bad.
earless1 13y ago
I'm glad this included the "Signs you should not be using a bash script" section. Bash is a very good solution for many cases, but it becomes downright unruly when dealing with a lot of string manipulation and more complex objects.
- chubot 13y ago
  your script is longer than a few hundred lines of code you need data structures beyond simple arrays you have a hard time working around quoting issues you do a lot of string manipulation you do not have much need for invoking other programs or pipe-lining them you worry about performance It's not a sign you shouldn't be using bash. It's a sign you shouldn't be using ONLY bash. People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man). You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes. You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution. Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.
  - japhyr 13y ago
    I think I might have accidentally down voted you. That was helpful feedback for me.
    - chubot 13y ago
      No problem... the way I think about is that Python/Perl/Ruby are for data structures, logic and control flow; while shell is for configuration and data flow. As mentioned above, things that belong in shell, and NOT in Python: - file system paths (these are parameters to Python scripts, not hard coded) - port numbers - user names - other config values - concurrency -- you can get really far with xargs -P, and sometimes & - pipelines (try doing this in Python without shelling out; it's horrible) Things that belong in Python and NOT shell: - complex control flow (e.g. I rarely write loops in bash; never a nested loop) - data structures (bash has hash tables, but I never use them) - arithmetic - parsing and cleaning individual lines or fields of data Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.
      - rix0r 13y ago
        File system paths may belong in the shell, but what about paths with spaces? To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(
        chubot 13y ago
        Here are the quoting rules I use: - Avoid paths with spaces :) - If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". - Always use "$@". I've never found a reason to use $@ or $* unquoted. I also write unit tests (in shell) if there is something unusual I know I have to handle. That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple. There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code. I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.
- yason 13y ago
  I generally, whatever I'm writing, start with bash or considering bash. Usually that means I start with bash, excluding the obvious cases such as graphics or doing heavy data processing where I know from start that line-based approach isn't enough. Then, when bash isn't enough, I write the difficult parts in Python as standalone utilities and use those from bash. Consequently, I already have a library of little tools in my ~/bin that are generic enough so that I can just use them from bash right away. But every now and then I need to write a special tool that my bash program can use. Most of my programs "finish" after this stage and they can live for years as a completed bash script calling helper programs. They do their job and there's nothing to add. As long as the program continues to "live on" I just make changes to the bash script or the helper programs. If at some point I need features that bash can't offer, such as more speed or more complex data processing, I begin to consider rewriting my program in one language. But this is only after the first revision of the program is already complete and I know exactly what to do when porting. Thus, porting the program is initially only about rewriting in another language, not developing it further at the same time. The first months or years of using the bash script turn into a prototyping stage, and this allows my rewrite be concise, well-thought and clear-partitioned. In other words, the rewrite becomes a new stable baseline that contains only the best of my bash prototyping, and only solves a perfectly scoped problem. As soon as the port becomes a drop-in replacement for my original bash program I switch to using it and start a development branch to support the new features, if any. Usually the newly gained speed is enough already. And usually the rewrite happens using C or Scheme. There's rarely enough point in rewriting the script in Python which may already be used to some extent in utility programs that the script calls. At that point I don't want to extend the program but lift it onto a new platform to cut some old baggage that's not needed anymore. There's a certain kind of joy rewriting in a new language something that you know completely before hand. This allows you to only work with the language since the underlying problem is already solved. Coincidentally, this new C or Scheme program might then, one day in the future, be called from a new bash script...
  - ejstronge 13y ago
    Do you have any examples of a tool/workflow that's made its way from bash to bash+python to C?
  - deleted 13y ago
- dllthomas 13y ago
  God yes. I spent a year maintaining ~90k lines of bash. Interesting experience...
  - migrantgeek 13y ago
    up-voted just to ease the pain
    - dllthomas 13y ago
      Thanks, man. Actually, it was kinda neat for the first 3 months or so, to have reason to dive deep and flesh out the dark corners of my bash knowledge. There's a few things I picked up that have served me well since - and I'd lived in the shell for years beforehand... The balance wasn't kinda neat.
- dvirsky 13y ago
  I'd go as far as saying "having more than a dozen lines". Bash syntax for anything other than basic execution and piping a few commands, is a disaster IMHO. Anything that requires logic should be written in a "real" scripting language. Python is my choice when I need a bit of logic in my scripts, but it's just a personal choice of course. A good addition for simplifying execution is the "sh" library that lets you call system commands easily as if they were python functions: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/
  - massysett 13y ago
    Agreed; I would also say that you should run from Bash not when you need anything more than a simple array, but when you need any kind of array, period. The advice on functions also gives me pause because that helps you write longer Bash scripts...not a good idea. Functions are of limited use because they can't really return values (just an exit status.) If a function has to return something, it generally prints it to stdout...which requires use of command substitution to get the result...nasty. Shells are good for interactive use and for starting other programs, but it's a lot easier to take a full-featured language and make it work like a shell than it is to take a shell and make it behave like a full-featured language. I'd take this blog post, add considerably to the "when you shouldn't use Bash" list, and put that at the top of the blog post, and then say "but if you absolutely must use Bash for something that is not trivial, here are some things that can make the process manageably nasty instead of truly horrid."
dsfadadsffd 13y ago
There is no need for long set flags, e.g. use set -e and not: set -o err etc.
- nfm 13y ago
  They are functionally equivalent, but the longhand versions are clearly more readable/greppable/google-able. The longhand versions have the exact same benefits as calling a variable eg. `target_file` instead of `a`.
  - stormbrew 13y ago
    Usually true but not always. Calling a for loop iterator "index" instead of "i" just adds unnecessary noise, for example. I think "set -e" in bash should become common on the same level, it's pretty rare that you really want a script to continue after an unguarded error return.
    - dsfadadsffd 13y ago
      Yes, and you can always set +e if you need to undo it.
      - stormbrew 13y ago
        I do, however, really despise that you use - to turn these things on and + to turn them off. Without the + you just think of it as being like command line flags, but then you run into +e and the whole thing goes topsy turvy.
    - gmjosack 13y ago
      I call my for loop variables idx. I never use single letter variable names, though I won't call someone out for it in a small block. Being able to highlight a variable in my editor, even when it's only used in a 5 line block of code is useful. Highlighting all letter i's is not.
      - 13y ago
sleepydog 13y ago
I found this web page, from the author of musl libc, very insightful: http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html Shell scripts are great, I use and write them every day (and quite advanced ones, too). But it's very hard to make a shell script robust. Unfortunately it's hard to find a replacement that is stable and installed everywhere. Perl is pretty close. And python too, if you are careful about making your script compatible with all the different versions.
narsil 13y ago
Another useful capability I use to do cleanup is `trap`. function cleanup { ... } trap cleanup EXIT See more here: http://linux.die.net/Bash-Beginners-Guide/sect_12_02.html http://linux.die.net/Bash-Beginners-Guide/sect_12_02.html
- eik3_de 13y ago
  to clean the command prompt line after CTRL-C: trap "{ echo; exit 1; }" INT
  - e12e 13y ago
    I don't know, if you're fearing a ctrl-c in the middle of one of those newfangled moving, colourized progress bars (hello, npm), "reset" might be more appropriate?
    - eik3_de 13y ago
      sure, if you do coloring/bold/underline/inverse in the script, reset would be appropriate. but I wouldn't do it by default, because it's quite slow ('time reset' takes 1s here)
      - TorKlingberg 13y ago
        My reset also takes suspiciously close to one second. Turns out it sleeps for one second at the end, probably to give the terminal time for something.
      - shabble 13y ago
        "tputs reset" has much the same effect (I think; there are some cases I've noticed it not fully restoring things) without the delay.
- pavel_lishin 13y ago
  Is there a way to determine whether the exit occurred due to the bash script finishing successfully, or due to an error (via set -o errexit)?
kirubakaran 13y ago
Use: #!/usr/bin/env bash Instead of: #!/bin/bash This makes the script more portable as you don't rely on bash (or any executable) to be in /bin. http://en.wikipedia.org/wiki/Shebang_(Unix)#Portability http://en.wikipedia.org/wiki/Shebang_(Unix)#Portability
- sgentle 13y ago
  Are there any situations where you wouldn't be able to find bash in /bin/bash? I haven't ever seen such a system, but I'd like to know if it's something I might run into... The other thing is, if you're targeting obscure systems, wouldn't it be just as likely that there wouldn't be a /usr/bin/env, /usr/ might not be mounted, or that bash might not be installed at all? I suppose what I'm asking is: for practical purposes, is the lowest common denominator /usr/bin/env or is it /bin/sh?
  - clarry 13y ago
    > Are there any situations where you wouldn't be able to find bash in /bin/bash? Yes there are. OpenBSD for instance installs third party software under /usr/local. Bash is not a part of the base system. env(1) on the other hand is a POSIX standard utility and comes with the OS.
  - kirubakaran 13y ago
    [in addition to what clarry said] Habit of using /usr/bin/env becomes more important in other use-cases, such as invoking Python interpreter.
    - nnnnni 13y ago
      Exactly. I often use "#!/usr/bin/env python2" to specify python 2.x on my systems where python 3 is the default.
  - thwarted 13y ago
    The lowest common denominator is /bin/sh. But bash has a lot of niceties, and the POSIX shell can get the job done, but that lowest common denominator is pretty low. You're still going to be dealing with annoying differences on the platforms anyway. I don't use env on my shebang lines specifically because it's then PATH ordering dependent (there used to be a thing where /usr/local/bin and GNU tools were last in root's path, but first in non-root users' path). I'm more confident (perhaps incorrectly) that /bin/bash or /usr/local/bin/bash is what I expect it is vs the first thing found in PATH that is named bash is what I think it is (however, this applies moreso to coreutils-like things, that have different SysV vs BSD semantics/options, vs a shell such as bash, which is known everywhere to be GNU bash). Some tools, like ssh, can propagate environment settings based on local, remote or config file settings, and I'd rather not be surprised. This used to be a bigger deal on systems that put reasonable (where "reasonable" == "what you're used to") tools in /usr/ucb or /opt/gnu, rather than system paths. If you're going to create something that's intended to be "portable", you've got bigger fish to fry than if and where bash is available, and it's wise to abstract system differences to different scripts (run.linux.sh, run.freebsd.sh, run.osx.sh, run.aix.sh, etc) than try to keep everything in one massive script anyway.
voltagex_ 13y ago
I also like http://google-styleguide.googlecode.com/svn/trunk/shell.xml http://google-styleguide.googlecode.com/svn/trunk/shell.xml but of course some things that work well for Google might not work for you.
dingaling 13y ago
complete -r disables Bash 'smart tab completion', which in theory is a great idea ( use tab to complete arguments or only list files applicable to the program ) but which never seems to work properly for me. Disabling it saves a lot of frustrated tab-banging.
- dfc 13y ago
  Something is terribly wrong with your setup if command line completion is not working.
  - aidenn0 13y ago
    A lot of installs come with over-complete smart completion configurations that make <tab> take several seconds (or even 10s of seconds) to complete in fairly common situations.
    - dfc 13y ago
      Please give me one or two examples of "fairly common situations" where it takes >=20 seconds for bash to respond to the tab.
      - claudius 13y ago
        Filename completion on remote systems using ssh/scp. It is a fairly common situation for me, although I wouldn’t say it takes 20s. Maybe 2-5s?
        shabble 13y ago
        Setting up SSH multiplexing with ControlPersist[1] can help quite a bit here, since after the first connection you don't have to go through the init/connection phase for subsequent completions. You can even preemptively fire up a master connection to commonly accessed hosts to avoid the initial delay. [1] https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing
mateuszf 13y ago
Nice here document feature I have found recently is heredoc with pipe, e.g. cat <<REQUEST_BODY | { "from" : 0, "size" : 40 } REQUEST_BODY curl http://localhost -d @- It allows to pass heredoc text to standard input of next command.
- lotheac 13y ago
  This is a prime example of useless use of cat. Heredoc already means "pass this as stdin", there's no need to pipe it. Your example without cat: curl http://localhost -d @- <<REQUEST_BODY { "from" : 0, "size" : 40 } REQUEST_BODY
  - mateuszf 13y ago
    Well, maybe in that case it is. But I like to separate the commands so that the data flow looks sequential. That way it makes more sense to me.
  - zimpenfish 13y ago
    Much as I hate the cargocult invocation of "UUOC", this is actually nicer because it reads more sanely - the HEREDOC is properly linked to the curl whereas the previous has the curl seemingly adrift on its own line unattached to the HEREDOC.
  - pdkl95 13y ago
    I like modularity: request_body() { cat <<REQUEST_BODY | { "from" : 0, "size" : 40 } REQUEST_BODY } get_url() { curl http://localhost -d @- } request_body | get_url I find it helps readability when you come back to it a year later. Of course, it's also easy to parameterize the body, if needed. /readability sometimes trumps YAGNI
- zimpenfish 13y ago
  My issue with this is that unless you're maintaining a lot of context and understanding the precise weirdness of HEREDOC piping, that looks at first glance like you're catting the HEREDOC to STDOUT and then running a random curl command. It's clever, sure, but it's harder to read and maintain.
cynik_ 13y ago
I'd really recommend using `set -x` or `bash -x script` to sanity check all the commands and expected output. See http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03...
- scott_karana 13y ago
  The article mentioned both -x and -v, didn't it? ;)
rtpg 13y ago
are we going to get a better bash at one point? I've always felt like the only thing bash scripts are good at describing is I/O redirection. But conditionals, dealing with variables, pretty much everything else is frustrating and error-prone I use fish as my main shell and its slightly better, but just testing things on variables can be a huge mess.
- nfoz 13y ago
  I'm making shok (http://shok.io http://shok.io) as an alternative, but it's in its infancy (read: currently useless). The basics should be usable in a few months. shok proposes that the solution to awkward-shell-syntax is to syntactically separate the programming language from program-invocation.
- dllthomas 13y ago
  Well, we've consistently gotten a better bash as bash development has proceeded. As for something that removes the warts and doesn't cut out anything too important and winds up with adoption - it'll be interesting to see...
- Goosey 13y ago
  I enjoy using fish as my main shell, but as soon as I ran into a "curl oneliner install" that failed in fish (in my case Homebrew's at the bottom of http://brew.sh/ http://brew.sh/) and required me to jump into bash. I enjoy fish so much I continue to use it, but I am in a state of fear that I will at some point encounter some failing shell script that leaves my system in a broken state. Do you have any recommendations to make fish play better with shell scripts intended for bash?
  - hollerith 13y ago
    In fish, type "bash" then return. Now paste in the curl oneliner and hit return. Finally, type "exit" to exit bash and go back to fish.
  - cpenner461 13y ago
    I used fish for about 6 months or so and loved it - with this exception. Initially I'd just hop into bash to do whatever I needed (as another comment suggests), but what was the "last straw" for me was not being to have some of the convenience functions that virtualenvwrapper exposes for working with python virtualenvs. My solution was to switch to zsh/oh-my-zsh - I'm pretty happy with it so far, although I do miss a few things from fish (namely the auto suggest/complete when typing previous commands).
ygra 13y ago
I love the very last list »Signs that you should not be using a bash script«. That should be a required part of every language/tool introduction/tutorial. So very often people lose track of when to use what tools. (Although admittedly, so very often people are forced into some tools by external constraints.)
oneandoneis2 13y ago
A link on HN about improving bash and it wasn't instructions on how to install zsh. I'm pleasantly surprised :)
- weaksauce 13y ago
  I like bash and all, but a well tuned zsh is amazing. I was hesitant for a while but it really improved my workflow in the shell.
  - oneandoneis2 13y ago
    I switched from bash to zsh some times ago, and miss it when it's not there.. but TBH, the only real difference for me is zsh can turn 'cd s/l/f/p[tab]' into 'cd some/long/file/path' - all other features I'm aware of either exist in Bash too, or aren't useful for me. What are some of the differences that made such a difference for you?
    - foxhill 13y ago
      off the top of my head: - tab completion on a shit-ton of stuff (configure scripts, makefiles, ssh hosts, aptitude packages, git branches) - partially entered command? push up to look at similar commands you have previously entered - immediate propagation of zsh_history to all open shells you could (correctly) argue that you can do all that in bash. of course, shell scripting is turing complete, so yes, you can. but with zsh, it's already there, and ready.
      - valarauca1 13y ago
        tab completion is a default feature of GNU BASH
        weaksauce 13y ago
        it tab completes more than bash does. switches, parameters, servers defined in your .ssh/config file etc. it's really impressive. coupled with the jump/mark script[1] it's amazing. mark . marks the current directory in a folder full of symlinks. jump <tab> expands to a list of your favorite folders. super simple and extremely low overhead. [1] http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-filesystem-from-the-command-line.html http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-f...
phaemon 13y ago
If you're using: set -o errexit # or set -e if you prefer Then you probably also want: set -o pipefail Otherwise, it only checks that the last command succeeds, so something like: ls *.ssjkfle | wc -l will actually continue as success despite the "ls" failing.
zx2c4 13y ago
If you'd like to see a decently written piece of bash that incorporates many of these suggestions, check out pass, the standard unix password manager. Project page: http://www.zx2c4.com/projects/password-store/ http://www.zx2c4.com/projects/password-store/ Source: http://git.zx2c4.com/password-store/tree/src/password-store.sh http://git.zx2c4.com/password-store/tree/src/password-store....
dllthomas 13y ago
One thing I've liked is throwing ${PIPESTATUS[*]} at the front of my PS1.
hyp0 13y ago
Excellent, bash the good parts. More than 15 minutes though. Googling bashlint, shlint turns up some discussion (bash -n, ksh -n, zsh -n, some github projects), but I doubt they cover this article's specifics - though most (all?) of it could be automatically checked. I think some could be automatically added (e.g. set -o nounset) - perhaps a bash-subset (or coffeescript-style language) possible...
- notfoss 13y ago
  I have had good results with Syntastic for Vim. Running :SyntasticInfo lists 'sh' as the linting program for bash scripts.
- curiousbiped 13y ago
  Try out shellcheck - there's an online checker at http://www.shellcheck.net/ http://www.shellcheck.net/, and if you like it, the source for it is on github at https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck.
njharman 13y ago
>> This will take care of two very common errors: >> Referencing undefined variables (which default to "") >> Ignoring failing commands Better is subjective... About half my scripts depend on those features. For default arguments, and fail early.
- skywhopper 13y ago
  He's advocating fail early with "set -e". And as for the other, you can allow overrides with syntax like: COULD_BE_SET=${COULD_BE_SET:altvalue}
Karunamon 13y ago
Try moving all bash code into functions leaving only global variable/constant definitions and a call to “main” at the top-level. One of my main complaints with bash.. the file is evaluated in order - you can't call a function on the line before it's declared. This fails: #!/usr/bin/env bash bar='bar' foofunction foofunction(){ echo 'foo' echo $bar } Basically you have to write your entire script in reverse, and i'm unaware of a good way to get around it.
- 14113 13y ago
  C is the same, but at least with C it's possible to declare functions before providing definitions. Is it possible that it's the same in bash?
- xrt 13y ago
  I really hate writing things in reverse, but a decent style in Bash is to put everything in functions, w/ a main() at the top: #! /usr/bin/env bash set -e -u bar='bar' main() { foofunction } foofunction() { echo 'foo' echo $bar } main "$@"
- nfoz 13y ago
  Well, what would you do instead? It's an interpreted language. When you type "foofunction" at a prompt, you don't want it to wait around in case you define that to have meaning later. You probably want the interpreter to be smart: "Am I loading a script from a file, or am I receiving instructions interactively on the command-line?" But now there's two modes of execution, and code in a bash script won't work if you type it into the prompt yourself. That's a bit uncomfortable.
knyt 13y ago
The author uses ${} a lot more than I see in most code. Is it helpful to always use the ${var} syntax instead of simply writing $var? I can see universal application of ${} being advantageous in avoiding accidental "$foo_bar where you meant ${foo}_bar" situations, and ${} makes it clearer that you're referencing a variable. The only cost would seem to be more typing.
celebril 13y ago
Or you can just use Zsh, which is superior in any way to Bash. ;)
- vacri 13y ago
  Except existing deployment spread :)

475 points by wsxiaoys 13y ago

rshm 13y ago

'set -o nounset' is a must have. I Just suffered this script from Samsung Printer setting Utility. sudo ./uninstall wiped the /opt DEST_PATH=/opt/$VENDOR/$DEST_DIRNAME #remove destination VERSION=`cat "$DEST_PATH/bin/.version"` if rm -fr "$DEST_PATH"; then echo "INFO: $APP_NAME (ver.$VERSION) has been uninstalled successfully." ...

laurent123456 13y ago
Wow, nounset or not, doing a `rm -rf` on a variable without any check is quite irresponsible. Especially if they expect the script to be run as sudo.
jacquesm 13y ago
Just checking if .version exists and if it doesn't exiting with an error code would be a huge improvement, checking if $DEST_PATH is set is a must as well. Scary to see a big company like Samsung release code this bad.

earless1 13y ago

I'm glad this included the "Signs you should not be using a bash script" section. Bash is a very good solution for many cases, but it becomes downright unruly when dealing with a lot of string manipulation and more complex objects.

chubot 13y ago
your script is longer than a few hundred lines of code you need data structures beyond simple arrays you have a hard time working around quoting issues you do a lot of string manipulation you do not have much need for invoking other programs or pipe-lining them you worry about performance It's not a sign you shouldn't be using bash. It's a sign you shouldn't be using ONLY bash. People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man). You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes. You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution. Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.
- japhyr 13y ago
  I think I might have accidentally down voted you. That was helpful feedback for me.
  - chubot 13y ago
    No problem... the way I think about is that Python/Perl/Ruby are for data structures, logic and control flow; while shell is for configuration and data flow. As mentioned above, things that belong in shell, and NOT in Python: - file system paths (these are parameters to Python scripts, not hard coded) - port numbers - user names - other config values - concurrency -- you can get really far with xargs -P, and sometimes & - pipelines (try doing this in Python without shelling out; it's horrible) Things that belong in Python and NOT shell: - complex control flow (e.g. I rarely write loops in bash; never a nested loop) - data structures (bash has hash tables, but I never use them) - arithmetic - parsing and cleaning individual lines or fields of data Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.
    - rix0r 13y ago
      File system paths may belong in the shell, but what about paths with spaces? To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(
      - chubot 13y ago
        Here are the quoting rules I use: - Avoid paths with spaces :) - If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". - Always use "$@". I've never found a reason to use $@ or $* unquoted. I also write unit tests (in shell) if there is something unusual I know I have to handle. That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple. There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code. I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.
yason 13y ago
I generally, whatever I'm writing, start with bash or considering bash. Usually that means I start with bash, excluding the obvious cases such as graphics or doing heavy data processing where I know from start that line-based approach isn't enough. Then, when bash isn't enough, I write the difficult parts in Python as standalone utilities and use those from bash. Consequently, I already have a library of little tools in my ~/bin that are generic enough so that I can just use them from bash right away. But every now and then I need to write a special tool that my bash program can use. Most of my programs "finish" after this stage and they can live for years as a completed bash script calling helper programs. They do their job and there's nothing to add. As long as the program continues to "live on" I just make changes to the bash script or the helper programs. If at some point I need features that bash can't offer, such as more speed or more complex data processing, I begin to consider rewriting my program in one language. But this is only after the first revision of the program is already complete and I know exactly what to do when porting. Thus, porting the program is initially only about rewriting in another language, not developing it further at the same time. The first months or years of using the bash script turn into a prototyping stage, and this allows my rewrite be concise, well-thought and clear-partitioned. In other words, the rewrite becomes a new stable baseline that contains only the best of my bash prototyping, and only solves a perfectly scoped problem. As soon as the port becomes a drop-in replacement for my original bash program I switch to using it and start a development branch to support the new features, if any. Usually the newly gained speed is enough already. And usually the rewrite happens using C or Scheme. There's rarely enough point in rewriting the script in Python which may already be used to some extent in utility programs that the script calls. At that point I don't want to extend the program but lift it onto a new platform to cut some old baggage that's not needed anymore. There's a certain kind of joy rewriting in a new language something that you know completely before hand. This allows you to only work with the language since the underlying problem is already solved. Coincidentally, this new C or Scheme program might then, one day in the future, be called from a new bash script...
- ejstronge 13y ago
  Do you have any examples of a tool/workflow that's made its way from bash to bash+python to C?
- deleted 13y ago
dllthomas 13y ago
God yes. I spent a year maintaining ~90k lines of bash. Interesting experience...
- migrantgeek 13y ago
  up-voted just to ease the pain
  - dllthomas 13y ago
    Thanks, man. Actually, it was kinda neat for the first 3 months or so, to have reason to dive deep and flesh out the dark corners of my bash knowledge. There's a few things I picked up that have served me well since - and I'd lived in the shell for years beforehand... The balance wasn't kinda neat.
dvirsky 13y ago
I'd go as far as saying "having more than a dozen lines". Bash syntax for anything other than basic execution and piping a few commands, is a disaster IMHO. Anything that requires logic should be written in a "real" scripting language. Python is my choice when I need a bit of logic in my scripts, but it's just a personal choice of course. A good addition for simplifying execution is the "sh" library that lets you call system commands easily as if they were python functions: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/
- massysett 13y ago
  Agreed; I would also say that you should run from Bash not when you need anything more than a simple array, but when you need any kind of array, period. The advice on functions also gives me pause because that helps you write longer Bash scripts...not a good idea. Functions are of limited use because they can't really return values (just an exit status.) If a function has to return something, it generally prints it to stdout...which requires use of command substitution to get the result...nasty. Shells are good for interactive use and for starting other programs, but it's a lot easier to take a full-featured language and make it work like a shell than it is to take a shell and make it behave like a full-featured language. I'd take this blog post, add considerably to the "when you shouldn't use Bash" list, and put that at the top of the blog post, and then say "but if you absolutely must use Bash for something that is not trivial, here are some things that can make the process manageably nasty instead of truly horrid."

dsfadadsffd 13y ago

There is no need for long set flags, e.g. use set -e and not: set -o err etc.

nfm 13y ago
They are functionally equivalent, but the longhand versions are clearly more readable/greppable/google-able. The longhand versions have the exact same benefits as calling a variable eg. `target_file` instead of `a`.
- stormbrew 13y ago
  Usually true but not always. Calling a for loop iterator "index" instead of "i" just adds unnecessary noise, for example. I think "set -e" in bash should become common on the same level, it's pretty rare that you really want a script to continue after an unguarded error return.
  - dsfadadsffd 13y ago
    Yes, and you can always set +e if you need to undo it.
    - stormbrew 13y ago
      I do, however, really despise that you use - to turn these things on and + to turn them off. Without the + you just think of it as being like command line flags, but then you run into +e and the whole thing goes topsy turvy.
  - gmjosack 13y ago
    I call my for loop variables idx. I never use single letter variable names, though I won't call someone out for it in a small block. Being able to highlight a variable in my editor, even when it's only used in a 5 line block of code is useful. Highlighting all letter i's is not.
    - 13y ago

sleepydog 13y ago

I found this web page, from the author of musl libc, very insightful: http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html Shell scripts are great, I use and write them every day (and quite advanced ones, too). But it's very hard to make a shell script robust. Unfortunately it's hard to find a replacement that is stable and installed everywhere. Perl is pretty close. And python too, if you are careful about making your script compatible with all the different versions.

narsil 13y ago

Another useful capability I use to do cleanup is `trap`. function cleanup { ... } trap cleanup EXIT See more here: http://linux.die.net/Bash-Beginners-Guide/sect_12_02.html http://linux.die.net/Bash-Beginners-Guide/sect_12_02.html

eik3_de 13y ago
to clean the command prompt line after CTRL-C: trap "{ echo; exit 1; }" INT
- e12e 13y ago
  I don't know, if you're fearing a ctrl-c in the middle of one of those newfangled moving, colourized progress bars (hello, npm), "reset" might be more appropriate?
  - eik3_de 13y ago
    sure, if you do coloring/bold/underline/inverse in the script, reset would be appropriate. but I wouldn't do it by default, because it's quite slow ('time reset' takes 1s here)
    - TorKlingberg 13y ago
      My reset also takes suspiciously close to one second. Turns out it sleeps for one second at the end, probably to give the terminal time for something.
    - shabble 13y ago
      "tputs reset" has much the same effect (I think; there are some cases I've noticed it not fully restoring things) without the delay.
pavel_lishin 13y ago
Is there a way to determine whether the exit occurred due to the bash script finishing successfully, or due to an error (via set -o errexit)?

kirubakaran 13y ago

Use: #!/usr/bin/env bash Instead of: #!/bin/bash This makes the script more portable as you don't rely on bash (or any executable) to be in /bin. http://en.wikipedia.org/wiki/Shebang_(Unix)#Portability http://en.wikipedia.org/wiki/Shebang_(Unix)#Portability

sgentle 13y ago
Are there any situations where you wouldn't be able to find bash in /bin/bash? I haven't ever seen such a system, but I'd like to know if it's something I might run into... The other thing is, if you're targeting obscure systems, wouldn't it be just as likely that there wouldn't be a /usr/bin/env, /usr/ might not be mounted, or that bash might not be installed at all? I suppose what I'm asking is: for practical purposes, is the lowest common denominator /usr/bin/env or is it /bin/sh?
- clarry 13y ago
  > Are there any situations where you wouldn't be able to find bash in /bin/bash? Yes there are. OpenBSD for instance installs third party software under /usr/local. Bash is not a part of the base system. env(1) on the other hand is a POSIX standard utility and comes with the OS.
- kirubakaran 13y ago
  [in addition to what clarry said] Habit of using /usr/bin/env becomes more important in other use-cases, such as invoking Python interpreter.
  - nnnnni 13y ago
    Exactly. I often use "#!/usr/bin/env python2" to specify python 2.x on my systems where python 3 is the default.
- thwarted 13y ago
  The lowest common denominator is /bin/sh. But bash has a lot of niceties, and the POSIX shell can get the job done, but that lowest common denominator is pretty low. You're still going to be dealing with annoying differences on the platforms anyway. I don't use env on my shebang lines specifically because it's then PATH ordering dependent (there used to be a thing where /usr/local/bin and GNU tools were last in root's path, but first in non-root users' path). I'm more confident (perhaps incorrectly) that /bin/bash or /usr/local/bin/bash is what I expect it is vs the first thing found in PATH that is named bash is what I think it is (however, this applies moreso to coreutils-like things, that have different SysV vs BSD semantics/options, vs a shell such as bash, which is known everywhere to be GNU bash). Some tools, like ssh, can propagate environment settings based on local, remote or config file settings, and I'd rather not be surprised. This used to be a bigger deal on systems that put reasonable (where "reasonable" == "what you're used to") tools in /usr/ucb or /opt/gnu, rather than system paths. If you're going to create something that's intended to be "portable", you've got bigger fish to fry than if and where bash is available, and it's wise to abstract system differences to different scripts (run.linux.sh, run.freebsd.sh, run.osx.sh, run.aix.sh, etc) than try to keep everything in one massive script anyway.

voltagex_ 13y ago

I also like http://google-styleguide.googlecode.com/svn/trunk/shell.xml http://google-styleguide.googlecode.com/svn/trunk/shell.xml but of course some things that work well for Google might not work for you.

dingaling 13y ago

complete -r disables Bash 'smart tab completion', which in theory is a great idea ( use tab to complete arguments or only list files applicable to the program ) but which never seems to work properly for me. Disabling it saves a lot of frustrated tab-banging.

dfc 13y ago
Something is terribly wrong with your setup if command line completion is not working.
- aidenn0 13y ago
  A lot of installs come with over-complete smart completion configurations that make <tab> take several seconds (or even 10s of seconds) to complete in fairly common situations.
  - dfc 13y ago
    Please give me one or two examples of "fairly common situations" where it takes >=20 seconds for bash to respond to the tab.
    - claudius 13y ago
      Filename completion on remote systems using ssh/scp. It is a fairly common situation for me, although I wouldn’t say it takes 20s. Maybe 2-5s?
      - shabble 13y ago
        Setting up SSH multiplexing with ControlPersist[1] can help quite a bit here, since after the first connection you don't have to go through the init/connection phase for subsequent completions. You can even preemptively fire up a master connection to commonly accessed hosts to avoid the initial delay. [1] https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing

mateuszf 13y ago

Nice here document feature I have found recently is heredoc with pipe, e.g. cat <<REQUEST_BODY | { "from" : 0, "size" : 40 } REQUEST_BODY curl http://localhost -d @- It allows to pass heredoc text to standard input of next command.

lotheac 13y ago
This is a prime example of useless use of cat. Heredoc already means "pass this as stdin", there's no need to pipe it. Your example without cat: curl http://localhost -d @- <<REQUEST_BODY { "from" : 0, "size" : 40 } REQUEST_BODY
- mateuszf 13y ago
  Well, maybe in that case it is. But I like to separate the commands so that the data flow looks sequential. That way it makes more sense to me.
- zimpenfish 13y ago
  Much as I hate the cargocult invocation of "UUOC", this is actually nicer because it reads more sanely - the HEREDOC is properly linked to the curl whereas the previous has the curl seemingly adrift on its own line unattached to the HEREDOC.
- pdkl95 13y ago
  I like modularity: request_body() { cat <<REQUEST_BODY | { "from" : 0, "size" : 40 } REQUEST_BODY } get_url() { curl http://localhost -d @- } request_body | get_url I find it helps readability when you come back to it a year later. Of course, it's also easy to parameterize the body, if needed. /readability sometimes trumps YAGNI
zimpenfish 13y ago
My issue with this is that unless you're maintaining a lot of context and understanding the precise weirdness of HEREDOC piping, that looks at first glance like you're catting the HEREDOC to STDOUT and then running a random curl command. It's clever, sure, but it's harder to read and maintain.

cynik_ 13y ago

I'd really recommend using `set -x` or `bash -x script` to sanity check all the commands and expected output. See http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03...

scott_karana 13y ago
The article mentioned both -x and -v, didn't it? ;)

rtpg 13y ago

are we going to get a better bash at one point? I've always felt like the only thing bash scripts are good at describing is I/O redirection. But conditionals, dealing with variables, pretty much everything else is frustrating and error-prone I use fish as my main shell and its slightly better, but just testing things on variables can be a huge mess.

nfoz 13y ago
I'm making shok (http://shok.io http://shok.io) as an alternative, but it's in its infancy (read: currently useless). The basics should be usable in a few months. shok proposes that the solution to awkward-shell-syntax is to syntactically separate the programming language from program-invocation.
dllthomas 13y ago
Well, we've consistently gotten a better bash as bash development has proceeded. As for something that removes the warts and doesn't cut out anything too important and winds up with adoption - it'll be interesting to see...
Goosey 13y ago
I enjoy using fish as my main shell, but as soon as I ran into a "curl oneliner install" that failed in fish (in my case Homebrew's at the bottom of http://brew.sh/ http://brew.sh/) and required me to jump into bash. I enjoy fish so much I continue to use it, but I am in a state of fear that I will at some point encounter some failing shell script that leaves my system in a broken state. Do you have any recommendations to make fish play better with shell scripts intended for bash?
- hollerith 13y ago
  In fish, type "bash" then return. Now paste in the curl oneliner and hit return. Finally, type "exit" to exit bash and go back to fish.
- cpenner461 13y ago
  I used fish for about 6 months or so and loved it - with this exception. Initially I'd just hop into bash to do whatever I needed (as another comment suggests), but what was the "last straw" for me was not being to have some of the convenience functions that virtualenvwrapper exposes for working with python virtualenvs. My solution was to switch to zsh/oh-my-zsh - I'm pretty happy with it so far, although I do miss a few things from fish (namely the auto suggest/complete when typing previous commands).

ygra 13y ago

I love the very last list »Signs that you should not be using a bash script«. That should be a required part of every language/tool introduction/tutorial. So very often people lose track of when to use what tools. (Although admittedly, so very often people are forced into some tools by external constraints.)

oneandoneis2 13y ago

A link on HN about improving bash and it wasn't instructions on how to install zsh. I'm pleasantly surprised :)

weaksauce 13y ago
I like bash and all, but a well tuned zsh is amazing. I was hesitant for a while but it really improved my workflow in the shell.
- oneandoneis2 13y ago
  I switched from bash to zsh some times ago, and miss it when it's not there.. but TBH, the only real difference for me is zsh can turn 'cd s/l/f/p[tab]' into 'cd some/long/file/path' - all other features I'm aware of either exist in Bash too, or aren't useful for me. What are some of the differences that made such a difference for you?
  - foxhill 13y ago
    off the top of my head: - tab completion on a shit-ton of stuff (configure scripts, makefiles, ssh hosts, aptitude packages, git branches) - partially entered command? push up to look at similar commands you have previously entered - immediate propagation of zsh_history to all open shells you could (correctly) argue that you can do all that in bash. of course, shell scripting is turing complete, so yes, you can. but with zsh, it's already there, and ready.
    - valarauca1 13y ago
      tab completion is a default feature of GNU BASH
      - weaksauce 13y ago
        it tab completes more than bash does. switches, parameters, servers defined in your .ssh/config file etc. it's really impressive. coupled with the jump/mark script[1] it's amazing. mark . marks the current directory in a folder full of symlinks. jump <tab> expands to a list of your favorite folders. super simple and extremely low overhead. [1] http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-filesystem-from-the-command-line.html http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-f...

phaemon 13y ago

If you're using: set -o errexit # or set -e if you prefer Then you probably also want: set -o pipefail Otherwise, it only checks that the last command succeeds, so something like: ls *.ssjkfle | wc -l will actually continue as success despite the "ls" failing.

zx2c4 13y ago

If you'd like to see a decently written piece of bash that incorporates many of these suggestions, check out pass, the standard unix password manager. Project page: http://www.zx2c4.com/projects/password-store/ http://www.zx2c4.com/projects/password-store/ Source: http://git.zx2c4.com/password-store/tree/src/password-store.sh http://git.zx2c4.com/password-store/tree/src/password-store....

dllthomas 13y ago

One thing I've liked is throwing ${PIPESTATUS[*]} at the front of my PS1.

hyp0 13y ago

Excellent, bash the good parts. More than 15 minutes though. Googling bashlint, shlint turns up some discussion (bash -n, ksh -n, zsh -n, some github projects), but I doubt they cover this article's specifics - though most (all?) of it could be automatically checked. I think some could be automatically added (e.g. set -o nounset) - perhaps a bash-subset (or coffeescript-style language) possible...

notfoss 13y ago
I have had good results with Syntastic for Vim. Running :SyntasticInfo lists 'sh' as the linting program for bash scripts.
curiousbiped 13y ago
Try out shellcheck - there's an online checker at http://www.shellcheck.net/ http://www.shellcheck.net/, and if you like it, the source for it is on github at https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck.

njharman 13y ago

>> This will take care of two very common errors: >> Referencing undefined variables (which default to "") >> Ignoring failing commands Better is subjective... About half my scripts depend on those features. For default arguments, and fail early.

skywhopper 13y ago
He's advocating fail early with "set -e". And as for the other, you can allow overrides with syntax like: COULD_BE_SET=${COULD_BE_SET:altvalue}

Karunamon 13y ago

Try moving all bash code into functions leaving only global variable/constant definitions and a call to “main” at the top-level. One of my main complaints with bash.. the file is evaluated in order - you can't call a function on the line before it's declared. This fails: #!/usr/bin/env bash bar='bar' foofunction foofunction(){ echo 'foo' echo $bar } Basically you have to write your entire script in reverse, and i'm unaware of a good way to get around it.

14113 13y ago
C is the same, but at least with C it's possible to declare functions before providing definitions. Is it possible that it's the same in bash?
xrt 13y ago
I really hate writing things in reverse, but a decent style in Bash is to put everything in functions, w/ a main() at the top: #! /usr/bin/env bash set -e -u bar='bar' main() { foofunction } foofunction() { echo 'foo' echo $bar } main "$@"
nfoz 13y ago
Well, what would you do instead? It's an interpreted language. When you type "foofunction" at a prompt, you don't want it to wait around in case you define that to have meaning later. You probably want the interpreter to be smart: "Am I loading a script from a file, or am I receiving instructions interactively on the command-line?" But now there's two modes of execution, and code in a bash script won't work if you type it into the prompt yourself. That's a bit uncomfortable.

knyt 13y ago

The author uses ${} a lot more than I see in most code. Is it helpful to always use the ${var} syntax instead of simply writing $var? I can see universal application of ${} being advantageous in avoiding accidental "$foo_bar where you meant ${foo}_bar" situations, and ${} makes it clearer that you're referencing a variable. The only cost would seem to be more typing.

celebril 13y ago

Or you can just use Zsh, which is superior in any way to Bash. ;)

vacri 13y ago
Except existing deployment spread :)