5 ms·

The article mentions so many topics, but misses almost all important ones. * First of all, use proper quoting. There are so many possibilities for file names,

by vog 12y ago

The article mentions so many topics, but misses almost all important ones. * First of all, use proper quoting. There are so many possibilities for file names, command line arguments, etc. that every unquoted usage of a variable is essentially a security risk. * Then, start your script with "set -e", which stops the script whenever one of the commands fail, instead of blindly continuing and messing things up. This is the most important option for robust shell scripts. * Also use "set -u" which makes the script stop on undefined variables. This includes $1, $2, etc., so it provides checks for missing arguments for free. * In addition to "set -e", also set "set -o pipefail", otherwise a pipe will only break if the last command fails, while with "set -o pipefail" the pipe fails whenever any command of the pipe fails. * After that, you may continue with spacing issues in "for" loops, and that you should not pipe the "find" output directly (instead, use either "-print0" + "xargs -0", or use "-exec"), and similar stuff. When you got all of this right, and only then!, you may start worrying about the (relatively) minor issues mentioned in the article.

mietek 12y ago
Unfortunately, you cannot depend on `set -e` (or `set -o errexit`) to work as expected in all cases. This usually means any non-trivial bash script is forced to do its own manual error handling, e.g. via ` || die`. http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105
- CameronNemo 12y ago
  Exactly. It makes you think about what you want to happen if that command fails. Die hard, or try to salvage it. You have to write shell scripts with set -e in mind from the get-go, and that is a good thing.
  - mietek 12y ago
    It would be a good thing if `set -e` worked as expected. The point of the FAQ is that `set -e` does not catch every failure, so you need to do your own checking anyway. http://mywiki.wooledge.org/BashFAQ/105/Answers http://mywiki.wooledge.org/BashFAQ/105/Answers
- mturmon 12y ago
  That's interesting, but I disagree. The cases in the linked article are uncommon. Most of the time set -e does what you want. And to the examples in that page: I am unconvinced by examples 1, 3, and 4. In (1), it's your business to know what the exit status of let is, and behave accordingly. In (3), you wrote the function f(), so it's your business to make its exit status correct. Similar for (4). These are unconvincing examples that seem to deliberately walk into minefields.
  - mietek 12y ago
    FWIW, I can assure you I did not deliberately walk into minefields while developing my recent bash project, and yet I've struggled with `errexit` and `errtrace` to the point where I've ceased to use them, preferring to just ` || die` everything which may possibly fail. Although you may say that simply choosing to use bash is self-defeating already... Have you reviewed the examples on this page, linked from the FAQ? http://fvue.nl/wiki/Bash:_Error_handling http://fvue.nl/wiki/Bash:_Error_handling
    - mturmon 12y ago
      Respectfully, this is nuts. You consider it a problem with 'set -e' that, if someone runs your script via % bash foo.sh then a '-e' on the shebang line of foo.sh will be ignored. That's a more general issue, separate from -e. You can say users who do that deserve what they get (not unreasonable IMHO), or you can use a 'set -e' within the script. But don't say it's a problem with -e. It's actually a problem with running command interpreters on arbitrary files. I can't read any more of these examples. I think you need more experience with shell, or maybe we have incommensurate approaches/expectations. I'm not of the view that using bash/sh in itself is a bad choice. There are lots of problems that it's well-suited for, and I myself use sh scripts a lot (with a linear flow that can go on for hundreds of lines). In particular, I don't think python is a good replacement for my use cases.
      - mietek 12y ago
        What are you talking about? Are you reading the same page as me? Let me quote http://fvue.nl/wiki/Bash:_Error_handling http://fvue.nl/wiki/Bash:_Error_handling Caveat 1: #!/bin/bash -e echo begin (false) echo end Executing the script above gives: $ ./caveat1.sh begin end $ This has nothing to do with "ignoring the '-e' on the shebang line." This is bash ignoring the exit status of the explicit subshell, even though `errexit` is set.
tieTYT 12y ago
I wrote my first shell script that actually matters a few weeks ago (in other words, I'm a total newb). Its purpose is to validate the configuration of another app. It seems to me that "set -o pipefail" would get in my way, because I'm often doing things like this: adapter_properties_home_community_id=$(cat ${EXTERNAL_VARIABLE}/adapter.properties \ | grep -P '^HomeCommunityId=' | sed 's/HomeCommunityId=//g' \ || err "line $LINENO: Could not find HomeCommunityId in the adaper.properties") If I "set -o pipefail", won't any failure in this chain of pipes halt the program? That will prevent the user from seeing the useful error message that says "Could not find HomeCommunityId in the adaper.properties" I'm also concerned about "set -u". That EXTERNAL_VARIABLE is set external to my validator script and available to the shell. Any appliance with the app should have that variable set. I have a check to make sure it's set and it's valid, but if I had "set -u", wouldn't that prevent me from checking its value because it hasn't been defined in the script? What's the work around to this, because I see how "set -u" would be very useful in most other situations.
- sigil 12y ago
  Some specific remarks on your shell script at the end... > If I "set -o pipefail", won't any failure in this chain of pipes halt the program? No, due to your use of the || operator: foo | bar || baz ...as a whole will always exit 0, even if foo or baz exits non-zero. The precedence there is actually ( foo | bar ) || baz I.e, the pipe operator binds tighter than any of the logic operators. Also, `set -o pipefail` doesn't cause your shell script to exit on the first error; you need `set -e` or explicit exit status checking for that. What `set -o pipefail` actually does is change the way exit status for a pipeline as a whole works. Consider the following pipeline: foo | bar | baz Without `set -o pipefail`, if either foo or bar exit non-zero, but baz exits zero, this pipeline will appear to have succeeded, because by default a pipeline gets the exit status of its last element. Ignoring errors is bad. So `set -o pipefail` makes the pipeline take the exit status of the last element that failed (if any failed). Regarding `set -u`, I don't personally use it, I just check that kind of stuff at the beginning of the script. Now, on to your shell script example: in practice you'll never see an error because both grep and sed return true even when no matching or substitution is done. You also have a useless use of cat [1] at the beginning; just use grep directly on that file. I would suggest simply doing sed -ne 's/^HomeCommunityId=//p' "${EXTERNAL_VARIABLE}"/adapter.properties for grabbing that variable. Then see if the variable is empty with a `-z` test. Hope this helps. Shell is weird but definitely worth learning! [1] http://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat http://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat
lomnakkus 12y ago
You're absolutely right, and this very list makes me think: Just program any non-trivial script in Python and use "subprocess" + "argparse". Heck, maybe even use Python if it's trivial -- since things have a tendency to accrue complexity over time you might as well avoid the "management won't let me rewrite in $DIFFERENT_LANGUAGE" trap.

by vog 12y ago

mietek 12y ago

Unfortunately, you cannot depend on `set -e` (or `set -o errexit`) to work as expected in all cases. This usually means any non-trivial bash script is forced to do its own manual error handling, e.g. via ` || die`. http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105

CameronNemo 12y ago
Exactly. It makes you think about what you want to happen if that command fails. Die hard, or try to salvage it. You have to write shell scripts with set -e in mind from the get-go, and that is a good thing.
- mietek 12y ago
  It would be a good thing if `set -e` worked as expected. The point of the FAQ is that `set -e` does not catch every failure, so you need to do your own checking anyway. http://mywiki.wooledge.org/BashFAQ/105/Answers http://mywiki.wooledge.org/BashFAQ/105/Answers
mturmon 12y ago
That's interesting, but I disagree. The cases in the linked article are uncommon. Most of the time set -e does what you want. And to the examples in that page: I am unconvinced by examples 1, 3, and 4. In (1), it's your business to know what the exit status of let is, and behave accordingly. In (3), you wrote the function f(), so it's your business to make its exit status correct. Similar for (4). These are unconvincing examples that seem to deliberately walk into minefields.
- mietek 12y ago
  FWIW, I can assure you I did not deliberately walk into minefields while developing my recent bash project, and yet I've struggled with `errexit` and `errtrace` to the point where I've ceased to use them, preferring to just ` || die` everything which may possibly fail. Although you may say that simply choosing to use bash is self-defeating already... Have you reviewed the examples on this page, linked from the FAQ? http://fvue.nl/wiki/Bash:_Error_handling http://fvue.nl/wiki/Bash:_Error_handling
  - mturmon 12y ago
    Respectfully, this is nuts. You consider it a problem with 'set -e' that, if someone runs your script via % bash foo.sh then a '-e' on the shebang line of foo.sh will be ignored. That's a more general issue, separate from -e. You can say users who do that deserve what they get (not unreasonable IMHO), or you can use a 'set -e' within the script. But don't say it's a problem with -e. It's actually a problem with running command interpreters on arbitrary files. I can't read any more of these examples. I think you need more experience with shell, or maybe we have incommensurate approaches/expectations. I'm not of the view that using bash/sh in itself is a bad choice. There are lots of problems that it's well-suited for, and I myself use sh scripts a lot (with a linear flow that can go on for hundreds of lines). In particular, I don't think python is a good replacement for my use cases.
    - mietek 12y ago
      What are you talking about? Are you reading the same page as me? Let me quote http://fvue.nl/wiki/Bash:_Error_handling http://fvue.nl/wiki/Bash:_Error_handling Caveat 1: #!/bin/bash -e echo begin (false) echo end Executing the script above gives: $ ./caveat1.sh begin end $ This has nothing to do with "ignoring the '-e' on the shebang line." This is bash ignoring the exit status of the explicit subshell, even though `errexit` is set.

tieTYT 12y ago

I wrote my first shell script that actually matters a few weeks ago (in other words, I'm a total newb). Its purpose is to validate the configuration of another app. It seems to me that "set -o pipefail" would get in my way, because I'm often doing things like this: adapter_properties_home_community_id=$(cat ${EXTERNAL_VARIABLE}/adapter.properties \ | grep -P '^HomeCommunityId=' | sed 's/HomeCommunityId=//g' \ || err "line $LINENO: Could not find HomeCommunityId in the adaper.properties") If I "set -o pipefail", won't any failure in this chain of pipes halt the program? That will prevent the user from seeing the useful error message that says "Could not find HomeCommunityId in the adaper.properties" I'm also concerned about "set -u". That EXTERNAL_VARIABLE is set external to my validator script and available to the shell. Any appliance with the app should have that variable set. I have a check to make sure it's set and it's valid, but if I had "set -u", wouldn't that prevent me from checking its value because it hasn't been defined in the script? What's the work around to this, because I see how "set -u" would be very useful in most other situations.

sigil 12y ago
Some specific remarks on your shell script at the end... > If I "set -o pipefail", won't any failure in this chain of pipes halt the program? No, due to your use of the || operator: foo | bar || baz ...as a whole will always exit 0, even if foo or baz exits non-zero. The precedence there is actually ( foo | bar ) || baz I.e, the pipe operator binds tighter than any of the logic operators. Also, `set -o pipefail` doesn't cause your shell script to exit on the first error; you need `set -e` or explicit exit status checking for that. What `set -o pipefail` actually does is change the way exit status for a pipeline as a whole works. Consider the following pipeline: foo | bar | baz Without `set -o pipefail`, if either foo or bar exit non-zero, but baz exits zero, this pipeline will appear to have succeeded, because by default a pipeline gets the exit status of its last element. Ignoring errors is bad. So `set -o pipefail` makes the pipeline take the exit status of the last element that failed (if any failed). Regarding `set -u`, I don't personally use it, I just check that kind of stuff at the beginning of the script. Now, on to your shell script example: in practice you'll never see an error because both grep and sed return true even when no matching or substitution is done. You also have a useless use of cat [1] at the beginning; just use grep directly on that file. I would suggest simply doing sed -ne 's/^HomeCommunityId=//p' "${EXTERNAL_VARIABLE}"/adapter.properties for grabbing that variable. Then see if the variable is empty with a `-z` test. Hope this helps. Shell is weird but definitely worth learning! [1] http://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat http://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat

lomnakkus 12y ago

You're absolutely right, and this very list makes me think: Just program any non-trivial script in Python and use "subprocess" + "argparse". Heck, maybe even use Python if it's trivial -- since things have a tendency to accrue complexity over time you might as well avoid the "management won't let me rewrite in $DIFFERENT_LANGUAGE" trap.