4 ms·

You should never use set -e. Because shell scripts use exit status both to indicate failure and as a kind of a boolean, this option has to try to guess your int

by pkkm 4y ago

You should never use set -e. Because shell scripts use exit status both to indicate failure and as a kind of a boolean, this option has to try to guess your intent by using a bunch of rules that are convoluted and can differ between shells, or even between versions. It's a matter of time before they do something you don't expect. Putting || exit or || return everywhere really is the better option if you can't switch to a better language like Python. I highly recommend https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls. It should be required reading before anyone writes a shell script.

Too 4y ago
No no no. Shell scripts without -e is one of the biggest footguns since computers were invented. Second being shell scripts with -e enabled ;) As others mentioned -u and pipefail are also highly recommended. Even mundane shit like ‘cd’ can fail where you least expect it, requiring || exit on practically every line of the script. Forget one place and boom. Defeating the premise of “it’s just a short “simple” script so let’s not write it in a sane language”. If you want to write serious scripts and spend weeks scrutinizing every line, NASA style, it is perhaps better to explicitly handle every potential error. But good luck doing that in a large team of junior web developers just glueing stuff together in the build pipeline. The pitfalls with -e linked above are mostly related to functions, if your script reached the size of requiring functions you have kind of lost already.
- pkkm 4y ago
  > Even mundane shit like ‘cd’ can fail where you least expect it, requiring || exit on practically every line of the script. Forget one place and boom. This is true, but a much better solution is for everyone to use shellcheck. You can require that all shell scripts pass it without warnings and that every shellcheck disable comment has a good explanation attached; the first part can be automated with a pre-commit hook. In return, you get a script that is explicit and predictable rather than relying on a set of rules which you don't understand, and which may come back to bite you later when you change shells or even bash versions. > spend weeks scrutinizing every line, NASA style This is a colossal exaggeration. I've switched from set -e to putting || return or || exit everywhere and it barely takes more time after I got in the habit. On the rare occasions I get distracted and forget, shellcheck reminds me. I have my editor configured to run it on the fly and underline lines with warnings.
  - Too 4y ago
    While shellcheck is a fantastic tool, it doesn't test for lack of error handling generally. In this area it only has a predefined set of known pitfalls, like "cd". Try running this through shellcheck and it will happily let it through. curl icanhazip.com > myip.txt cat myip.txt Taking you back to the every line must be scrutinized problem. Maybe you can do it. Maybe i can do it. But does the rest of your large organization of diverse experiences have the discipline to do it? In my experience, from multiple organizations of different sizes - no. Even with shellcheck and mandatory reviews it just doesn't work. It's tiring to for the 100th time remind someone that curl can fail and arguments must be quoted, you worry being labeled as the know-it-all-besserwisser that shoots down even simple scripts, that's assuming the reviewer spots the errors in the first place. What's worse is that the code doesn't immediately tell you if errors has been considered or not, making it very difficult to read existing scripts. Shellcheck btw has an optional check for the case of functions used in combination with errexit https://www.shellcheck.net/wiki/SC2310 https://www.shellcheck.net/wiki/SC2310
    - pkkm 4y ago
      Thanks for sharing where you're coming from. My opinion comes from maintaining shell scripts either alone or in small teams of people who learn quickly. Perhaps the optimal solution is different when the team is large. I still think that manual error handling is best if you're disciplined enough to do it, but I'll make sure to qualify this view next time I'm talking about it.
acdha 4y ago
This is terrible advice. -e doesn’t prevent you from adding more careful error handling but it does catch a significant majority of the problems which will, with absolute certainty, happen because nobody ever adds “|| exit” everywhere it’s necessary. -e and shellcheck should be mandatory for any script which can’t be written in Python. Once you’re over a screen or two of code, it’s almost always shorter to write in Python anyway and it’ll be easier to understand.
- pkkm 4y ago
  If you already use shellcheck and know to add || exit, there's no need to rely on a half-solution that may fail in unintuitive and hard to predict ways. Counting on convoluted "automagical" rules to guess your intent correctly every time is how you write unreliable programs. set -e just gives you a false sense of security. > Once you’re over a screen or two of code, it’s almost always shorter to write in Python anyway That I agree with.
  - acdha 4y ago
    I would argue that it’s better to start with -e and deal with the handful of edge cases where something like grep is expected to return a non-zero value rather than have to remember to check every command. We now have half a century of evidence suggesting that won’t work reliably for all but the most conscientious shell scripters.

by pkkm 4y ago

Too 4y ago

No no no. Shell scripts without -e is one of the biggest footguns since computers were invented. Second being shell scripts with -e enabled ;) As others mentioned -u and pipefail are also highly recommended. Even mundane shit like ‘cd’ can fail where you least expect it, requiring || exit on practically every line of the script. Forget one place and boom. Defeating the premise of “it’s just a short “simple” script so let’s not write it in a sane language”. If you want to write serious scripts and spend weeks scrutinizing every line, NASA style, it is perhaps better to explicitly handle every potential error. But good luck doing that in a large team of junior web developers just glueing stuff together in the build pipeline. The pitfalls with -e linked above are mostly related to functions, if your script reached the size of requiring functions you have kind of lost already.

pkkm 4y ago
> Even mundane shit like ‘cd’ can fail where you least expect it, requiring || exit on practically every line of the script. Forget one place and boom. This is true, but a much better solution is for everyone to use shellcheck. You can require that all shell scripts pass it without warnings and that every shellcheck disable comment has a good explanation attached; the first part can be automated with a pre-commit hook. In return, you get a script that is explicit and predictable rather than relying on a set of rules which you don't understand, and which may come back to bite you later when you change shells or even bash versions. > spend weeks scrutinizing every line, NASA style This is a colossal exaggeration. I've switched from set -e to putting || return or || exit everywhere and it barely takes more time after I got in the habit. On the rare occasions I get distracted and forget, shellcheck reminds me. I have my editor configured to run it on the fly and underline lines with warnings.
- Too 4y ago
  While shellcheck is a fantastic tool, it doesn't test for lack of error handling generally. In this area it only has a predefined set of known pitfalls, like "cd". Try running this through shellcheck and it will happily let it through. curl icanhazip.com > myip.txt cat myip.txt Taking you back to the every line must be scrutinized problem. Maybe you can do it. Maybe i can do it. But does the rest of your large organization of diverse experiences have the discipline to do it? In my experience, from multiple organizations of different sizes - no. Even with shellcheck and mandatory reviews it just doesn't work. It's tiring to for the 100th time remind someone that curl can fail and arguments must be quoted, you worry being labeled as the know-it-all-besserwisser that shoots down even simple scripts, that's assuming the reviewer spots the errors in the first place. What's worse is that the code doesn't immediately tell you if errors has been considered or not, making it very difficult to read existing scripts. Shellcheck btw has an optional check for the case of functions used in combination with errexit https://www.shellcheck.net/wiki/SC2310 https://www.shellcheck.net/wiki/SC2310
  - pkkm 4y ago
    Thanks for sharing where you're coming from. My opinion comes from maintaining shell scripts either alone or in small teams of people who learn quickly. Perhaps the optimal solution is different when the team is large. I still think that manual error handling is best if you're disciplined enough to do it, but I'll make sure to qualify this view next time I'm talking about it.

acdha 4y ago

This is terrible advice. -e doesn’t prevent you from adding more careful error handling but it does catch a significant majority of the problems which will, with absolute certainty, happen because nobody ever adds “|| exit” everywhere it’s necessary. -e and shellcheck should be mandatory for any script which can’t be written in Python. Once you’re over a screen or two of code, it’s almost always shorter to write in Python anyway and it’ll be easier to understand.

pkkm 4y ago
If you already use shellcheck and know to add || exit, there's no need to rely on a half-solution that may fail in unintuitive and hard to predict ways. Counting on convoluted "automagical" rules to guess your intent correctly every time is how you write unreliable programs. set -e just gives you a false sense of security. > Once you’re over a screen or two of code, it’s almost always shorter to write in Python anyway That I agree with.
- acdha 4y ago
  I would argue that it’s better to start with -e and deal with the handful of edge cases where something like grep is expected to return a non-zero value rather than have to remember to check every command. We now have half a century of evidence suggesting that won’t work reliably for all but the most conscientious shell scripters.