9 ms·

Defensive Bash programming

127 points by urlwolf 12y ago

mooism2 12y ago
The first rule of defensive bash programming should be: quote everything. Incredibly, the article doesn't mention quoting at all, doesn't even use it silently in examples.
- twic 12y ago
  He also doesn't set -eu or -o pipefail.
- chrismonsanto 12y ago
  I always recommend people use zsh for scripting instead of bash--it gets word splitting right. Most of my sysadminny scripts start with #!/bin/zsh these days. It would be really nice if Linux distros came with zsh installed automatically! For some reason the shell quoting behavior never seems to get brought up when comparing zsh and bash. It is, IMO, the most important distinction between the two shells.
  - Florin_Andrei 12y ago
    As someone who's never used zsh - you don't need to quote stuff in zsh?
    - dozzie 12y ago
      It depends on options set. There is one that causes word splitting only to happen when explicitly asked for.
    - chrismonsanto 12y ago
      By default, no. Word splitting only happens when you request it: ${=words}. It reminds me of ,@list in Lisp, which is a warm and fuzzy feeling.
  - kondor6c 12y ago
    That's the tricky thing about 'sysadminny' scripts is that they generally need to be written for the least common denominator, which usually ends up being a very old version of $(software). I frequently have to use Python 2.4 to reach many of the RHEL-ish 5 machines in our environment.
- moe 12y ago
  Actually the first rule of bash programming should be: Don't use bash. If your script is longer than 10 lines then you should probably write it in a less error prone language.
dozzie 12y ago
Yes, of course. Use more bashisms, wait for bash to change its behaviour in those bashisms (it already happened and was not that rare) and happy debugging.
- mhd 12y ago
  Given the current glacial pace of bash development? The problem is that the only other 'standard' shell scripts can rely on is POSIX/Bourne, which is a bit anemic. Back in the days, this was the realm where the Korn shell was supposed to reign. And amongst commercial unices, it actually did help (the license of ksh88/93 prohibited widespread BSD/Linux use). But sadly most of the features that zsh/bash copied or invented themselves concern the UI, not programming capabilities. I've seen some pretty big programs in ksh93 and it was half-way decent. Then again, quite often you shelled out to awk/grep/ed and had to fight their incompatibilities... I don't really miss those days. On the other hand, sometimes I wish Linux scripters/programmers would be forced to use a slightly incompatible system now and then so that they don't assume that the whole world is GNU. I'm not misanthropic enough to say that said system should be IBM AIX, though.
  - dozzie 12y ago
    > the only other 'standard' shell scripts can rely on is POSIX/Bourne Have you seen POSIX? Have you read Bourne shell documentation? I haven't. The former is expensive, the latter is not that easy to get. I have read Single UNIX Specification, though, which is available on-line. > But sadly most of the features that zsh/bash copied or invented themselves concern the UI, not programming capabilities. Shell is not intended for regular programming. It's intended for small automation scripts. If you need to write anything bigger, choosing shell over Perl, Python or Ruby is a fundamentally bad idea. BTDTGTT. And do you know how much of SUS-guaranteed shell syntax you use, anyway? Do you know what is a bashism and what is in the specification, so you can claim SUS-compliant shell has too weak language?
    - clarry 12y ago
      > Have you seen POSIX? Have you read Bourne shell documentation? I haven't. The former is expensive, the latter is not that easy to get. I have read Single UNIX Specification, though, which is available on-line. What is the distinction you are trying to make? Single UNIX® Specification, Version 4, 2013 Edition Technically identical to IEEE Std 1003.1, 2013 Edition and ISO/IEC 9945:2009 including ISO/IEC 9945:2009/Cor 1:2013(E), with the addition of X/Open Curses. https://www2.opengroup.org/ogsys/catalog/t101 https://www2.opengroup.org/ogsys/catalog/t101 IEEE Std 1003.1 is also known as POSIX.1. You can read it online. It documents the shell command language. http://ddg.gg/?q=1003.1-2013 http://ddg.gg/?q=1003.1-2013 See also the POSIX FAQ: http://www.opengroup.org/austin/papers/posix_faq.html http://www.opengroup.org/austin/papers/posix_faq.html EDIT: Source to bourne shell's manual is here, http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/man/man1/sh.1 http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/man/man1... There's also a PDF version, look in volume 1: http://plan9.bell-labs.com/7thEdMan/bswv7.html http://plan9.bell-labs.com/7thEdMan/bswv7.html
iamthebest 12y ago
Corporate gateway seems to not like this domain: This web site ( www.kfirlavi.com ) has been blocked because it has been determined by Web Reputation Filters to be a security threat to your computer or the corporate network. This web site has been associated with malware/spyware. Reputation Score for www.kfirlavi.com: -7.1 Reputation scores can range from -10 (worst) through 10 (best).
- josephlord 12y ago
  I can't see anything there that looks troubling. Maybe it was hacked at some point or the domain used to be used for evil before the current owner got it. Or your corporate filter is having an off day or is just rubbish...
q3k 12y ago
And yet he pipes from ls...
- unhammer 12y ago
  And all those functions wrapping [[, ugh. maybe I should start doing this in Python def plus (a,b): return a+b
- obiterdictum 12y ago
  What is wrong with piping `ls`?
  - dockd 12y ago
    It's usually redundant. for i in * <--> for i in `ls`
  - q3k 12y ago
    http://mywiki.wooledge.org/ParsingLs http://mywiki.wooledge.org/ParsingLs
  - massysett 12y ago
    Usually it is more pristine (from a theoretical standpoint) to use filename generation, aka globbing. You don't need a subprocess.
nemasu 12y ago
Defining a function for is_file as [[ -f $blah ]]; seems like defining a function for var++ as increment_one() ...
- girvo 12y ago
  I get why that seems silly, and I write shell all day (for provisioning Vagrant machines where I don't need Puppet. I'm still yet to work Ansible properly...), so I know exactly what that command does. But to some of my coworkers who don't write shell very often at all, it would make the script far clearer (which is useful as they need to be able to edit it on the fly) to those who don't know shell
  - datashaman 12y ago
    Encouraging coworkers to learn the proper syntax seems a better choice than hiding it with personally chosen abstractions (albeit more readable).
    - girvo 12y ago
      I don't disagree, but in some contexts that trade off is worth it, in my opinion.
    - myg204 12y ago
      Yeah, definitely better to keep it idiomatic. Also: $ test -f "$file" is pretty clear I think.
    - lukeschlather 12y ago
      The "proper" syntax is dumb. I've never considered writing functions to get around it, but seeing this makes me want to contribute some builtins to bash.
      - datashaman 12y ago
        The correct answer. ^^^
vlisivka 12y ago
Check out bash-modules: https://github.com/vlisivka/bash-modules https://github.com/vlisivka/bash-modules
borplk 12y ago
If you do have the choice, the first rule of defensive bash programming is to not program in bash. But otherwise it was a great article.
rquirk 12y ago
You can use the pipe | as a continuation at the end of a line. No need to use the backslash escape then. i.e. you never should need to use | \ at the end of a line.
- warrick 12y ago
  Right, this is a style choice. He explicitly mentions '| \' as a bad example. https://google-styleguide.googlecode.com/svn/trunk/shell.xml?showone=Pipelines#Pipelines https://google-styleguide.googlecode.com/svn/trunk/shell.xml...
cthalupa 12y ago
Generally, bash is for quick and dirty things I want to automate. I'll go to perl or python if I need anything more complex. The amount of effort put into these examples is already way higher than my personal sniff test for "Should I be doing this in something besides bash?"
- oneeyedpigeon 12y ago
  That might be chicken-and-egg. By following more rigorous practices, bash might become more acceptable to you for more complex tasks.
  - Alupis 12y ago
    Such as managing complex build processes.
  - massysett 12y ago
    Nope. The amount of rigor required to sanitize Bash or any other descendant of sh or csh is not worth it. There are problems such as the lack of real return values from functions that no amount of discipline can solve, and the rigor required to solve things like crazy word splitting is too hard to consistently get right.
danielweber 12y ago
I freaking love "set -x" and wish every language had an equivalent. When I switch from bash to another language I miss "set -x" deep in my soul.
- planckscnst 12y ago
  ruby http://unroller.rubyforge.org/ http://unroller.rubyforge.org/ python https://docs.python.org/2/library/trace.html https://docs.python.org/2/library/trace.html perl http://search.cpan.org/~mjd/Devel-Trace-0.10/Trace.pm http://search.cpan.org/~mjd/Devel-Trace-0.10/Trace.pm
- lordlarm 12y ago
  "set -x" enables debug-mode, for anyone else heading off to Google to investigate. So given the script: #!/bin/bash set -x echo "Hello World" The output would be: + echo 'Hello World!' Hello World! Where the "+ ..." is the command executed. And it can be disabled with "set +x".
- vdm 12y ago
  I agree, and refer to it as a 'call trace'. Not to be confused with stack traces, which are the same thing inside out.
vog 12y ago
The article mentions so many topics, but misses almost all important ones. * First of all, use proper quoting. There are so many possibilities for file names, command line arguments, etc. that every unquoted usage of a variable is essentially a security risk. * Then, start your script with "set -e", which stops the script whenever one of the commands fail, instead of blindly continuing and messing things up. This is the most important option for robust shell scripts. * Also use "set -u" which makes the script stop on undefined variables. This includes $1, $2, etc., so it provides checks for missing arguments for free. * In addition to "set -e", also set "set -o pipefail", otherwise a pipe will only break if the last command fails, while with "set -o pipefail" the pipe fails whenever any command of the pipe fails. * After that, you may continue with spacing issues in "for" loops, and that you should not pipe the "find" output directly (instead, use either "-print0" + "xargs -0", or use "-exec"), and similar stuff. When you got all of this right, and only then!, you may start worrying about the (relatively) minor issues mentioned in the article.
- mietek 12y ago
  Unfortunately, you cannot depend on `set -e` (or `set -o errexit`) to work as expected in all cases. This usually means any non-trivial bash script is forced to do its own manual error handling, e.g. via ` || die`. http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105
  - CameronNemo 12y ago
    Exactly. It makes you think about what you want to happen if that command fails. Die hard, or try to salvage it. You have to write shell scripts with set -e in mind from the get-go, and that is a good thing.
    - mietek 12y ago
      It would be a good thing if `set -e` worked as expected. The point of the FAQ is that `set -e` does not catch every failure, so you need to do your own checking anyway. http://mywiki.wooledge.org/BashFAQ/105/Answers http://mywiki.wooledge.org/BashFAQ/105/Answers
Sir_Cmpwn 12y ago
I thought that UPPER_CASE variables were a bad idea? Doesn't the bash world generally accept that you should use uppercase only for enviornment variables, and lowercase for variables in the script's context?
- Anthony-G 12y ago
  The author specifies using upper case for global variables – not for local variables. I’d consider that to be a useful technique and practice it myself. I also agree with minimising the use of global variables in the first place.
pertinhower 12y ago
Is threesome kind of secret code embedded in the typos?

127 points by urlwolf 12y ago

The first rule of defensive bash programming should be: quote everything. Incredibly, the article doesn't mention quoting at all, doesn't even use it silently in examples.

twic 12y ago
He also doesn't set -eu or -o pipefail.
chrismonsanto 12y ago
I always recommend people use zsh for scripting instead of bash--it gets word splitting right. Most of my sysadminny scripts start with #!/bin/zsh these days. It would be really nice if Linux distros came with zsh installed automatically! For some reason the shell quoting behavior never seems to get brought up when comparing zsh and bash. It is, IMO, the most important distinction between the two shells.
- Florin_Andrei 12y ago
  As someone who's never used zsh - you don't need to quote stuff in zsh?
  - dozzie 12y ago
    It depends on options set. There is one that causes word splitting only to happen when explicitly asked for.
  - chrismonsanto 12y ago
    By default, no. Word splitting only happens when you request it: ${=words}. It reminds me of ,@list in Lisp, which is a warm and fuzzy feeling.
- kondor6c 12y ago
  That's the tricky thing about 'sysadminny' scripts is that they generally need to be written for the least common denominator, which usually ends up being a very old version of $(software). I frequently have to use Python 2.4 to reach many of the RHEL-ish 5 machines in our environment.
moe 12y ago
Actually the first rule of bash programming should be: Don't use bash. If your script is longer than 10 lines then you should probably write it in a less error prone language.

dozzie 12y ago

Yes, of course. Use more bashisms, wait for bash to change its behaviour in those bashisms (it already happened and was not that rare) and happy debugging.

mhd 12y ago
Given the current glacial pace of bash development? The problem is that the only other 'standard' shell scripts can rely on is POSIX/Bourne, which is a bit anemic. Back in the days, this was the realm where the Korn shell was supposed to reign. And amongst commercial unices, it actually did help (the license of ksh88/93 prohibited widespread BSD/Linux use). But sadly most of the features that zsh/bash copied or invented themselves concern the UI, not programming capabilities. I've seen some pretty big programs in ksh93 and it was half-way decent. Then again, quite often you shelled out to awk/grep/ed and had to fight their incompatibilities... I don't really miss those days. On the other hand, sometimes I wish Linux scripters/programmers would be forced to use a slightly incompatible system now and then so that they don't assume that the whole world is GNU. I'm not misanthropic enough to say that said system should be IBM AIX, though.
- dozzie 12y ago
  > the only other 'standard' shell scripts can rely on is POSIX/Bourne Have you seen POSIX? Have you read Bourne shell documentation? I haven't. The former is expensive, the latter is not that easy to get. I have read Single UNIX Specification, though, which is available on-line. > But sadly most of the features that zsh/bash copied or invented themselves concern the UI, not programming capabilities. Shell is not intended for regular programming. It's intended for small automation scripts. If you need to write anything bigger, choosing shell over Perl, Python or Ruby is a fundamentally bad idea. BTDTGTT. And do you know how much of SUS-guaranteed shell syntax you use, anyway? Do you know what is a bashism and what is in the specification, so you can claim SUS-compliant shell has too weak language?
  - clarry 12y ago
    > Have you seen POSIX? Have you read Bourne shell documentation? I haven't. The former is expensive, the latter is not that easy to get. I have read Single UNIX Specification, though, which is available on-line. What is the distinction you are trying to make? Single UNIX® Specification, Version 4, 2013 Edition Technically identical to IEEE Std 1003.1, 2013 Edition and ISO/IEC 9945:2009 including ISO/IEC 9945:2009/Cor 1:2013(E), with the addition of X/Open Curses. https://www2.opengroup.org/ogsys/catalog/t101 https://www2.opengroup.org/ogsys/catalog/t101 IEEE Std 1003.1 is also known as POSIX.1. You can read it online. It documents the shell command language. http://ddg.gg/?q=1003.1-2013 http://ddg.gg/?q=1003.1-2013 See also the POSIX FAQ: http://www.opengroup.org/austin/papers/posix_faq.html http://www.opengroup.org/austin/papers/posix_faq.html EDIT: Source to bourne shell's manual is here, http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/man/man1/sh.1 http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/man/man1... There's also a PDF version, look in volume 1: http://plan9.bell-labs.com/7thEdMan/bswv7.html http://plan9.bell-labs.com/7thEdMan/bswv7.html

iamthebest 12y ago

Corporate gateway seems to not like this domain: This web site ( www.kfirlavi.com ) has been blocked because it has been determined by Web Reputation Filters to be a security threat to your computer or the corporate network. This web site has been associated with malware/spyware. Reputation Score for www.kfirlavi.com: -7.1 Reputation scores can range from -10 (worst) through 10 (best).

josephlord 12y ago
I can't see anything there that looks troubling. Maybe it was hacked at some point or the domain used to be used for evil before the current owner got it. Or your corporate filter is having an off day or is just rubbish...

q3k 12y ago

And yet he pipes from ls...

unhammer 12y ago
And all those functions wrapping [[, ugh. maybe I should start doing this in Python def plus (a,b): return a+b
obiterdictum 12y ago
What is wrong with piping `ls`?
- dockd 12y ago
  It's usually redundant. for i in * <--> for i in `ls`
- q3k 12y ago
  http://mywiki.wooledge.org/ParsingLs http://mywiki.wooledge.org/ParsingLs
- massysett 12y ago
  Usually it is more pristine (from a theoretical standpoint) to use filename generation, aka globbing. You don't need a subprocess.

nemasu 12y ago

Defining a function for is_file as [[ -f $blah ]]; seems like defining a function for var++ as increment_one() ...

girvo 12y ago
I get why that seems silly, and I write shell all day (for provisioning Vagrant machines where I don't need Puppet. I'm still yet to work Ansible properly...), so I know exactly what that command does. But to some of my coworkers who don't write shell very often at all, it would make the script far clearer (which is useful as they need to be able to edit it on the fly) to those who don't know shell
- datashaman 12y ago
  Encouraging coworkers to learn the proper syntax seems a better choice than hiding it with personally chosen abstractions (albeit more readable).
  - girvo 12y ago
    I don't disagree, but in some contexts that trade off is worth it, in my opinion.
  - myg204 12y ago
    Yeah, definitely better to keep it idiomatic. Also: $ test -f "$file" is pretty clear I think.
  - lukeschlather 12y ago
    The "proper" syntax is dumb. I've never considered writing functions to get around it, but seeing this makes me want to contribute some builtins to bash.
    - datashaman 12y ago
      The correct answer. ^^^

vlisivka 12y ago

Check out bash-modules: https://github.com/vlisivka/bash-modules https://github.com/vlisivka/bash-modules

borplk 12y ago

If you do have the choice, the first rule of defensive bash programming is to not program in bash. But otherwise it was a great article.

rquirk 12y ago

You can use the pipe | as a continuation at the end of a line. No need to use the backslash escape then. i.e. you never should need to use | \ at the end of a line.

warrick 12y ago
Right, this is a style choice. He explicitly mentions '| \' as a bad example. https://google-styleguide.googlecode.com/svn/trunk/shell.xml?showone=Pipelines#Pipelines https://google-styleguide.googlecode.com/svn/trunk/shell.xml...

cthalupa 12y ago

Generally, bash is for quick and dirty things I want to automate. I'll go to perl or python if I need anything more complex. The amount of effort put into these examples is already way higher than my personal sniff test for "Should I be doing this in something besides bash?"

oneeyedpigeon 12y ago
That might be chicken-and-egg. By following more rigorous practices, bash might become more acceptable to you for more complex tasks.
- Alupis 12y ago
  Such as managing complex build processes.
- massysett 12y ago
  Nope. The amount of rigor required to sanitize Bash or any other descendant of sh or csh is not worth it. There are problems such as the lack of real return values from functions that no amount of discipline can solve, and the rigor required to solve things like crazy word splitting is too hard to consistently get right.

danielweber 12y ago

I freaking love "set -x" and wish every language had an equivalent. When I switch from bash to another language I miss "set -x" deep in my soul.

planckscnst 12y ago
ruby http://unroller.rubyforge.org/ http://unroller.rubyforge.org/ python https://docs.python.org/2/library/trace.html https://docs.python.org/2/library/trace.html perl http://search.cpan.org/~mjd/Devel-Trace-0.10/Trace.pm http://search.cpan.org/~mjd/Devel-Trace-0.10/Trace.pm
lordlarm 12y ago
"set -x" enables debug-mode, for anyone else heading off to Google to investigate. So given the script: #!/bin/bash set -x echo "Hello World" The output would be: + echo 'Hello World!' Hello World! Where the "+ ..." is the command executed. And it can be disabled with "set +x".
vdm 12y ago
I agree, and refer to it as a 'call trace'. Not to be confused with stack traces, which are the same thing inside out.

vog 12y ago

The article mentions so many topics, but misses almost all important ones. * First of all, use proper quoting. There are so many possibilities for file names, command line arguments, etc. that every unquoted usage of a variable is essentially a security risk. * Then, start your script with "set -e", which stops the script whenever one of the commands fail, instead of blindly continuing and messing things up. This is the most important option for robust shell scripts. * Also use "set -u" which makes the script stop on undefined variables. This includes $1, $2, etc., so it provides checks for missing arguments for free. * In addition to "set -e", also set "set -o pipefail", otherwise a pipe will only break if the last command fails, while with "set -o pipefail" the pipe fails whenever any command of the pipe fails. * After that, you may continue with spacing issues in "for" loops, and that you should not pipe the "find" output directly (instead, use either "-print0" + "xargs -0", or use "-exec"), and similar stuff. When you got all of this right, and only then!, you may start worrying about the (relatively) minor issues mentioned in the article.

mietek 12y ago
Unfortunately, you cannot depend on `set -e` (or `set -o errexit`) to work as expected in all cases. This usually means any non-trivial bash script is forced to do its own manual error handling, e.g. via ` || die`. http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105
- CameronNemo 12y ago
  Exactly. It makes you think about what you want to happen if that command fails. Die hard, or try to salvage it. You have to write shell scripts with set -e in mind from the get-go, and that is a good thing.
  - mietek 12y ago
    It would be a good thing if `set -e` worked as expected. The point of the FAQ is that `set -e` does not catch every failure, so you need to do your own checking anyway. http://mywiki.wooledge.org/BashFAQ/105/Answers http://mywiki.wooledge.org/BashFAQ/105/Answers

Sir_Cmpwn 12y ago

I thought that UPPER_CASE variables were a bad idea? Doesn't the bash world generally accept that you should use uppercase only for enviornment variables, and lowercase for variables in the script's context?

Anthony-G 12y ago
The author specifies using upper case for global variables – not for local variables. I’d consider that to be a useful technique and practice it myself. I also agree with minimising the use of global variables in the first place.

pertinhower 12y ago

Is threesome kind of secret code embedded in the typos?