11 ms·

Bash Error Handling

287 points by sohkamyung 6y ago

JdeBP 6y ago
I wonder how much Clippy is to blame for the visual motif used in this comic strip. Common motifs, elsewhere, for a shell are a dollar sign and an underscore or a greater than sign and an underscore. (The latter is somewhat odd for a shell, given that it more resembles the prompts on Microsoft/IBM command interpreters, and not the PS1 prompts of Unix shells, which are commonly dollar symbols, hashes, or percent signs rather than greater than.) * https://www.redbubble.com/i/sticker/zsh-by-zoerab/20363330.EJUG5 https://www.redbubble.com/i/sticker/zsh-by-zoerab/20363330.E... * https://commons.wikimedia.org/wiki/File:Bash_Logo_black_and_white_icon_only.svg https://commons.wikimedia.org/wiki/File:Bash_Logo_black_and_... * https://commons.wikimedia.org/wiki/File:PowerShell_5.0_icon.png https://commons.wikimedia.org/wiki/File:PowerShell_5.0_icon.... * https://icon-library.net/icon/commands-icon-5.html https://icon-library.net/icon/commands-icon-5.html * https://icon-library.com/icon/bash-icon-10.html https://icon-library.com/icon/bash-icon-10.html * https://dribbble.com/shots/6101482-Bash-Automation https://dribbble.com/shots/6101482-Bash-Automation
pferde 6y ago
Also known as "bash strict mode". It's a godsend if you're unlucky enough to have to deal with bash scripting. http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ EDIT: The comic strip would be better in three rows of two panels - a row for each set flag.
pzmarzly 6y ago
Another useful flag is -x, which enables tracing (Bash will print each command before executing it). You can disable it again with set +x (same goes for +e, +u and afaik +o pipefail) Also, please use shellcheck - https://www.shellcheck.net/ https://www.shellcheck.net/ EDIT: Also, please don't modify scripts while they are running.
- aidenn0 6y ago
  Here's a workaround for modifying scripts while they are running; it works because function bodies are read and parsed entirely when they are defined. main() { script goes here exit } main "$@"
- 0xmohit 6y ago
  Specifying PS4 when using the -x flag can be even more helpful while debugging. The variable PS4 denotes the value is the prompt printed before the command line is echoed when the -x option is set and defaults to : followed by space. A number of useful debugging tips are listed at <https://wiki.bash-hackers.org/scripting/debuggingtips https://wiki.bash-hackers.org/scripting/debuggingtips>.
mehrdadn 6y ago
Wait until you realize set -e; (false && true); echo hi does nothing, but set -e; false && true ; echo hi prints hi.
- Sebb767 6y ago
  It's a bit surprising at first, but I'd like to think most people deeper into bash are aware of Subshells.
  - alanbernstein 6y ago
    I know $() invokes a subshell, but does () ? My guess was that the () affects the order of operations between ; and &&, so the first line is three commands, while the second line is two.
    - aidenn0 6y ago
      It's a subshell. Order of operations is unchanged, and if you just want a block of commands you can use { }
    - asicsp 6y ago
      See https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Command-Grouping https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.... "Placing a list of commands between parentheses causes a subshell environment to be created" "Placing a list of commands between curly braces causes the list to be executed in the current shell context"
      - gravitas 6y ago
        It's horribly more complex than just that, the person you're replying to is "more correct" because of the rules of `set -e` which is critical in this thread; a singular command followed by && is treated differently than a compound command. Pull up `man bash` and search `/set.*abef` to read the rather long and involved paragraph for the `-e` option, running in a subshell is only part of it.
exdsq 6y ago
Other tips I've found after a year or two of using Bash for many things it shouldn't be used for: - Use shellcheck (static analysis/linter) https://www.shellcheck.net/ https://www.shellcheck.net/ - Use shunit2 (unit tests) https://github.com/kward/shunit2 https://github.com/kward/shunit2 - Use 'local' or 'readonly' to annotate your variables - Trap ctrl+c to gracefully exit (details here https://www.tothenew.com/blog/foolproof-your-bash-script-some-best-practices/ https://www.tothenew.com/blog/foolproof-your-bash-script-som...) - Stick to long-form options for readability (--delete over -d for example) - #!/usr/bin/env > #!/bin/bash for portability - Consider setting variable values in a file and importing it at the top of your script to improve refactoring - You will eventually forget how your scripts work - seriously consider if Bash is your best option for anything that needs to last a while!
- mehrdadn 6y ago
  Do you want to trap INT, or EXIT?
  - exdsq 6y ago
    EXIT is trapped in the same way as 0, it's something that happens when your shell exits. Ctrl+C sends the SIGINT signal but you can catch it with INT too. You want to do the latter because your gracefully exiting the script, if you want to have some cleanup after that you could trap EXIT (for deleting tmp files or something).
    - jwilk 6y ago
      Writing good SIGINT handler is a bit tricky. You should re-raise the signal, so that bash notices that the program was actually interrupted: https://www.cons.org/cracauer/sigint.html https://www.cons.org/cracauer/sigint.html (section "How to be a proper program") Most of the time, especially for short lived script, adding signal handlers is not worth the trouble.
- asicsp 6y ago
  Some more useful links: * https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * https://mywiki.wooledge.org/BashGuide/Practices https://mywiki.wooledge.org/BashGuide/Practices * https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls * https://devmanual.gentoo.org/tools-reference/bash/index.html https://devmanual.gentoo.org/tools-reference/bash/index.html
loevborg 6y ago
`-o pipefail` is not without its gotchas bash -eo pipefail -c "cat /etc/services | head -1; echo never got here"
- aidenn0 6y ago
  I hate pipefail. I find set -e also has some gotchas. '|| die' added to the end of each line is uglier but more obvious in many cases.
- deleted 6y ago
  [deleted]
- stonewareslord 6y ago
  What is the gotcha here? You’re telling it to pipefail and exit on fail, so it exits.
  - tux1968 6y ago
    The gotcha is perhaps failing to recognize that cat will actually fail in this simple pipeline because its output is truncated "unexpectedly" by head exiting after completing its job properly. (Maybe you could argue that head should be a good citizen and read its input to completion) Where this will work as expected: bash -eo pipefail -c "head < /etc/services -1|cat ; echo got here"
    - gray_-_wolf 6y ago
      But head is a being a good citizen by doing only what it needs to do and not anything else. Otherwise things like `grep X <5GBfile | head -n1` would process the whole file and not just until the first match.
polyrand 6y ago
In Bash 4.4+ you can use: shopt -s inherit_errexit 2>/dev/null || true To make subprocesses inherit the errexit flag.
- chinigo 6y ago
  Lifesaver! Without this, error handling in bash functions is a gigantic, unintuitive pain. Thank you for bringing this to my attention.
  - amelius 6y ago
    And this boilerplate isn't going to be a pain?
- kasabali 6y ago
  inherit_errexit If set, command substitution inherits the value of the errexit option, instead of unsetting it in the subshell environment. This option is enabled when posix mode is enabled. So must I presume older versions were already doing that without needing an option set?
harisund 6y ago
I thought people shouldn't post anything about bash on HN? The minute you post something about bash immediately you will draw out a whole bunch of folks from the wood works talking about how bash sucks and should never be used for anything more than 3 or 4 lines and how they replaced bash with python or some thing else, immediately in turn drawing out a bunch of other folks talking about how bash should be replaced with power shell and how you can parse objects better ... .
- michaelcampbell 6y ago
  Any programming topic spawns a bunch of comments about readability, or examples of code-golf. It's the nature of things.
- exdsq 6y ago
  I had a really fun project earlier in the year prototyping a load testing tool for a blockchain in Bash while 4 other developers wrote a ‘better’ one in Haskell. Bash can get results quickly, although it’s not maintainable! Still, a decent kloc or two of bash with performance results within the sprint.
- heresie-dabord 6y ago
  I had become annoyed by the Python bigots who will tell us about how easy to read their language is because its notation is clean, how all its functionality is "intuitive", how any combination of Python-based Rube-Goldberg Machine systems is the best. But then came the PowerShell people...
asicsp 6y ago
Others in this on-going series: * https://wizardzines.com/comics/environment-variables https://wizardzines.com/comics/environment-variables * https://wizardzines.com/comics/brackets-cheatsheet https://wizardzines.com/comics/brackets-cheatsheet * https://wizardzines.com/comics/bash-quotes https://wizardzines.com/comics/bash-quotes * https://wizardzines.com/comics/bash-if-statements/ https://wizardzines.com/comics/bash-if-statements/
GolDDranks 6y ago
I almost always write POSIX shell instead of bash for compatibility; it would be nice to see collections for tips and tricks specifically for POSIX shell. I know, for example, that -o pipefail doesn't exist in plain POSIX shell. I wonder what's are the best practices when you can't use it.
- fpoling 6y ago
  I also write my scripts to stay /bin/sh compatible. If this is not enough, then a real scripting language should be used, not bash. But I very much agree that lack of pipefail is painful. If I know that output on the left of the pipe is small, I read it into a variable and then use printf | right part. If the output can be big, I use a helper function to emulate it that I copy-paste.
- asicsp 6y ago
  >it would be nice to see collections for tips and tricks specifically for POSIX shell Check out https://freebsdfrau.gitbook.io/serious-shell-programming/ https://freebsdfrau.gitbook.io/serious-shell-programming/
- sigzero 6y ago
  > it would be nice to see collections for tips and tricks specifically for POSIX shell https://www.shellscript.sh/ https://www.shellscript.sh/ There is a Facebook group and I have emailed the author as well with questions. Nice guy.
- m463 6y ago
  ok some low-grade tips: cleanup="true" later: if $cleanup; then a ; else b ; fi or slightly uglier $cleanup && rm -f mytempfile I like one-line functions: die() { echo "$@" 1>&2; exit 1; }
- peterwwillis 6y ago
  Read the ash or dash man pages and just use what they give you. That's pretty much Bourne shell which is pretty much POSIX.
- still_grokking 6y ago
_where 6y ago
If it’s not by default there’s a reason. Bash is literally running commands in a shell session. Think terminal session. When a command fails, would you want the terminal session to end? That’d be annoying. Same theory for unset variable. Referencing an undefined variable shouldn’t break your session. Why initialize it anyway? It’s more code to change if you don’t use it if you have to initialize it when it might not be needed. And, you’d have to call the script with A= just to check A wasn’t defined, and in the process now you have A assigned to an empty string, instead of only defaulting to one when called, which uses more memory and execution time. The pipeline doesn’t die because && and || and parens are seriously helpful for one-liners. Don’t think of it as a script. Think of it as a script for a shell.
- pwdisswordfish4 6y ago
  Sometimes the only reason is backwards compatibility.
- Waterluvian 6y ago
  Thinking of it that way, it becomes clear why you would want to switch to something like Python for more complex workflows. Bash is brilliantly useful for a lot of things. I’m not bashing it.
- IshKebab 6y ago
  A compelling argument for why Bash scripts shouldn't exist at all.
  - chubot 6y ago
    https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-di...
    - IshKebab 6y ago
      Ah yes the "nothing can ever change" argument. It's the reason we still code everything in FORTRAN and COBOL.
tpoacher 6y ago
I'm surprised nobody's mentioned [expect](https://likegeeks.com/expect-command/ https://likegeeks.com/expect-command/) yet in this thread :)
x87678r 6y ago
bash is not fit for any moderately complex task. I dont understand its popularity. Normally sane people who like unit testing, CICD, SOLID principles, quality tools end up with a bunch of crappy scripts holding everything together. Please avoid.
- chubot 6y ago
  https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-di...
- peterwwillis 6y ago
  It's fantastic for simple tasks, which is why so many people use it. It also turns out that many complex tasks can be reduced to a collection of simple tasks, otherwise known as "a bunch of crappy scripts". I encourage all my teams to avoid built-in CI/CD features and plugins and just script what they want in a Docker container. It ends up being easier to maintain, breaks less often, and is more portable.
- amelius 6y ago
  > I dont understand its popularity. I think the only reason people use the shell for scripting is that ls -lsa /tmp is simpler to write than execute(["ls", "-lsa", "/tmp"]) or even execute("ls -lsa /tmp") > Normally sane people who like unit testing, CICD, SOLID principles, quality tools end up with a bunch of crappy scripts holding everything together. Please avoid. Totally agree!
arendtio 6y ago
I like the notion of 'set -e' and at the same time I hate it. First, because it behaves inconsistently across shells/versions [1] and second, because it doesn't always work as expected. For example, when you depend on the 'set -e' behavior in a function and call the function from within a condition, the 'set -e' has no effect at all. So you better don't count on 'set -e'. But don't expect me to follow my own advice, as not using 'set -e' isn't a good option either... [1] https://www.in-ulm.de/~mascheck/various/set-e/ https://www.in-ulm.de/~mascheck/various/set-e/
- chubot 6y ago
  Yup, I am fixing this with Oil: https://news.ycombinator.com/item?id=24740842 https://news.ycombinator.com/item?id=24740842 (2 of 4 problems fixed so far, more problems welcome) Try it out on your shell scripts and let me know what happens :) OSH has the "broken" POSIX/bash behavior to maintain compatibility, while Oil opts you in to the better error semantics.
chubot 6y ago
I'm working on fixing errexit / set -e in Oil: https://github.com/oilshell/oil/issues/709 https://github.com/oilshell/oil/issues/709 Summary of problems: 1. The "if myfunc" problem -- error checking is skipped. This is specified by POSIX, but it's undesirable. 2. The f() { test -d /tmp && echo "exists" } problem. The exit code of the function is probably not what you expect! 3. the local x=$(false) problem. This happens in all shells because of the definition of $?. (local is a shell builtin with its own exit code, not part of the language) This one is already fixed with shopt -s more_errexit in Oil. 4. the x=$(false) problem. This is a bash problem with command substitution. For example, dash and zsh don't have this problem. Test case: bash -c 'set -e; x=$(false); echo should not get here' Newer versions of bash fix it with inherit_errexit. Oil also implements inherit_errexit and turns it on by default! (in Oil, not OSH) ----- So 1 and 2 relate to the confusing combination of shell functions and errexit. And 3 and 4 relate to command subs. Oil has fixed these with opt-in OSH shell options (on by default in Oil), but not 1 and 2. If you know of any other problems, please chime in on the bug!
- a1369209993 6y ago
  > 2. The f() { test -d /tmp && echo "exists" } problem. The exit code of the function is probably not what you expect! The exit code is 0 (assuming /tmp/, stdout, /bin/test and /bin/echo are all working correctly; with /tmp1 it's 1), as expected; is this referencing a bug in sh and/or bash that I've fixed locally and then forgotten about? (Also, I'm pretty sure it should be: f() { test -d /tmp && echo "exists"; } unless the parse error for missing ';' was your point (I haven't bothered to fix that one, but maybe Oil has).)
  - chubot 6y ago
    Yeah I didn't explain that very well. It's actually a variation on the problem here: https://news.ycombinator.com/item?id=24738274 https://news.ycombinator.com/item?id=24738274 except with a shell function rather than a subshell. Demo: https://github.com/oilshell/blog-code/blob/master/errexit https://github.com/oilshell/blog-code/blob/master/errexit /demo.sh#L21 Run ./demo.sh subshell-compare and func-compare. I have seen multiple people point it out as confusing. The problem is that in the first case, there are two exit codes: 1. false (suppressed on LHS of &&) 2. true And in the second case, there are 3: 1. false (suppressed on LHS of &&) 2. true 3. the exit code of the function/subshell, which is the exit code of the last statement, which is nonzero. This then CAUSES THE PROGRAM TO FAIL, whereas it didn't before. (edit: removed incorrect code) ----- So it's kind of like a "ghost" exit code created by the subshell/function! Which has an unexpected interaction with errexit. Does that make sense? I am not sure exactly how to fix it, but it will probably involve limiting && to stuff like this: if test -d /tmp && test -d /tmp/foo; then echo yes; fi And disallow it when standing alone, because it doesn't make much sense there, if errexit is on. Feel free to chime in bug with any possible solutions or more problems. ---- (And yes Oil fixes the brace problem with shopt -s parse_brace, which is on by default in Oil! Try it out and let me know if you like it :) )
major505 6y ago
This is usefull. I know a guy who didnt treat unset vairable, so in the script where he would remove with rm -rf /$dir-old-back his script removed all directories. The problem is that all his backups were in an external drive mounted in a directory, so he removed all his backups also. A hell month for him .
JeremyBanks 6y ago
https://dev.to/banks/stop-ignoring-errors-in-bash-3co5 https://dev.to/banks/stop-ignoring-errors-in-bash-3co5
cryptonector 6y ago
Except `set -e` is stupid because it is disabled all the way down when a command occurs in a conditional, meaning this: function foo { false; true; } foo || echo foo failed prints nothing! From the bash manual page: | If a compound command other than a subshell returns a non-zero status because a command failed while -e was being ignored, the shell does not exit. POSIX says the same thing, so this is true of all POSIX-y shells. This means you really have to check for errors you care about, and `set -e` is useless. Ugh!
lazyant 6y ago
I'd add TRAP DEBUG (running line by line) to the toolset
bewuethr 6y ago
set -e and set -u both come with their own pitfalls: https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 and https://mywiki.wooledge.org/BashFAQ/112 https://mywiki.wooledge.org/BashFAQ/112 I generally use them and think that overall, they have more benefits than drawbacks, but the odd time where I run into one of the pitfalls, debugging usually takes a while. ShellCheck already does a pretty good job of pointing out incorrect variable names, too.
westurner 6y ago
From https://twitter.com/b0rk/status/1312413117436104705 https://twitter.com/b0rk/status/1312413117436104705 : > TIL that you can use the "DEBUG" trap to step through a bash script line by line trap '(read -p "[$BASH_SOURCE:$LINENO] $BASH_COMMAND?")' DEBUG > [...] it does something very different than sh -x — sh -x will just print out lines, this stops before* every single line and lets you confirm that you want to run that line* >> you can also customize the prompt with set -x export PS4='+(${BASH_SOURCE}:${LINENO}) ' set -x With a markdown_escape function, could this make for something like a notebook with ```bash fenced code blocks with syntax highlighting?

287 points by sohkamyung 6y ago

I wonder how much Clippy is to blame for the visual motif used in this comic strip. Common motifs, elsewhere, for a shell are a dollar sign and an underscore or a greater than sign and an underscore. (The latter is somewhat odd for a shell, given that it more resembles the prompts on Microsoft/IBM command interpreters, and not the PS1 prompts of Unix shells, which are commonly dollar symbols, hashes, or percent signs rather than greater than.) * https://www.redbubble.com/i/sticker/zsh-by-zoerab/20363330.EJUG5 https://www.redbubble.com/i/sticker/zsh-by-zoerab/20363330.E... * https://commons.wikimedia.org/wiki/File:Bash_Logo_black_and_white_icon_only.svg https://commons.wikimedia.org/wiki/File:Bash_Logo_black_and_... * https://commons.wikimedia.org/wiki/File:PowerShell_5.0_icon.png https://commons.wikimedia.org/wiki/File:PowerShell_5.0_icon.... * https://icon-library.net/icon/commands-icon-5.html https://icon-library.net/icon/commands-icon-5.html * https://icon-library.com/icon/bash-icon-10.html https://icon-library.com/icon/bash-icon-10.html * https://dribbble.com/shots/6101482-Bash-Automation https://dribbble.com/shots/6101482-Bash-Automation

pferde 6y ago

Also known as "bash strict mode". It's a godsend if you're unlucky enough to have to deal with bash scripting. http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ EDIT: The comic strip would be better in three rows of two panels - a row for each set flag.

pzmarzly 6y ago

Another useful flag is -x, which enables tracing (Bash will print each command before executing it). You can disable it again with set +x (same goes for +e, +u and afaik +o pipefail) Also, please use shellcheck - https://www.shellcheck.net/ https://www.shellcheck.net/ EDIT: Also, please don't modify scripts while they are running.

aidenn0 6y ago
Here's a workaround for modifying scripts while they are running; it works because function bodies are read and parsed entirely when they are defined. main() { script goes here exit } main "$@"
0xmohit 6y ago
Specifying PS4 when using the -x flag can be even more helpful while debugging. The variable PS4 denotes the value is the prompt printed before the command line is echoed when the -x option is set and defaults to : followed by space. A number of useful debugging tips are listed at <https://wiki.bash-hackers.org/scripting/debuggingtips https://wiki.bash-hackers.org/scripting/debuggingtips>.

mehrdadn 6y ago

Wait until you realize set -e; (false && true); echo hi does nothing, but set -e; false && true ; echo hi prints hi.

Sebb767 6y ago
It's a bit surprising at first, but I'd like to think most people deeper into bash are aware of Subshells.
- alanbernstein 6y ago
  I know $() invokes a subshell, but does () ? My guess was that the () affects the order of operations between ; and &&, so the first line is three commands, while the second line is two.
  - aidenn0 6y ago
    It's a subshell. Order of operations is unchanged, and if you just want a block of commands you can use { }
  - asicsp 6y ago
    See https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Command-Grouping https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.... "Placing a list of commands between parentheses causes a subshell environment to be created" "Placing a list of commands between curly braces causes the list to be executed in the current shell context"
    - gravitas 6y ago
      It's horribly more complex than just that, the person you're replying to is "more correct" because of the rules of `set -e` which is critical in this thread; a singular command followed by && is treated differently than a compound command. Pull up `man bash` and search `/set.*abef` to read the rather long and involved paragraph for the `-e` option, running in a subshell is only part of it.

exdsq 6y ago

Other tips I've found after a year or two of using Bash for many things it shouldn't be used for: - Use shellcheck (static analysis/linter) https://www.shellcheck.net/ https://www.shellcheck.net/ - Use shunit2 (unit tests) https://github.com/kward/shunit2 https://github.com/kward/shunit2 - Use 'local' or 'readonly' to annotate your variables - Trap ctrl+c to gracefully exit (details here https://www.tothenew.com/blog/foolproof-your-bash-script-some-best-practices/ https://www.tothenew.com/blog/foolproof-your-bash-script-som...) - Stick to long-form options for readability (--delete over -d for example) - #!/usr/bin/env > #!/bin/bash for portability - Consider setting variable values in a file and importing it at the top of your script to improve refactoring - You will eventually forget how your scripts work - seriously consider if Bash is your best option for anything that needs to last a while!

mehrdadn 6y ago
Do you want to trap INT, or EXIT?
- exdsq 6y ago
  EXIT is trapped in the same way as 0, it's something that happens when your shell exits. Ctrl+C sends the SIGINT signal but you can catch it with INT too. You want to do the latter because your gracefully exiting the script, if you want to have some cleanup after that you could trap EXIT (for deleting tmp files or something).
  - jwilk 6y ago
    Writing good SIGINT handler is a bit tricky. You should re-raise the signal, so that bash notices that the program was actually interrupted: https://www.cons.org/cracauer/sigint.html https://www.cons.org/cracauer/sigint.html (section "How to be a proper program") Most of the time, especially for short lived script, adding signal handlers is not worth the trouble.
asicsp 6y ago
Some more useful links: * https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * https://mywiki.wooledge.org/BashGuide/Practices https://mywiki.wooledge.org/BashGuide/Practices * https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls * https://devmanual.gentoo.org/tools-reference/bash/index.html https://devmanual.gentoo.org/tools-reference/bash/index.html

loevborg 6y ago

`-o pipefail` is not without its gotchas bash -eo pipefail -c "cat /etc/services | head -1; echo never got here"

aidenn0 6y ago
I hate pipefail. I find set -e also has some gotchas. '|| die' added to the end of each line is uglier but more obvious in many cases.
deleted 6y ago
[deleted]
stonewareslord 6y ago
What is the gotcha here? You’re telling it to pipefail and exit on fail, so it exits.
- tux1968 6y ago
  The gotcha is perhaps failing to recognize that cat will actually fail in this simple pipeline because its output is truncated "unexpectedly" by head exiting after completing its job properly. (Maybe you could argue that head should be a good citizen and read its input to completion) Where this will work as expected: bash -eo pipefail -c "head < /etc/services -1|cat ; echo got here"
  - gray_-_wolf 6y ago
    But head is a being a good citizen by doing only what it needs to do and not anything else. Otherwise things like `grep X <5GBfile | head -n1` would process the whole file and not just until the first match.

polyrand 6y ago

In Bash 4.4+ you can use: shopt -s inherit_errexit 2>/dev/null || true To make subprocesses inherit the errexit flag.

chinigo 6y ago
Lifesaver! Without this, error handling in bash functions is a gigantic, unintuitive pain. Thank you for bringing this to my attention.
- amelius 6y ago
  And this boilerplate isn't going to be a pain?
kasabali 6y ago
inherit_errexit If set, command substitution inherits the value of the errexit option, instead of unsetting it in the subshell environment. This option is enabled when posix mode is enabled. So must I presume older versions were already doing that without needing an option set?

harisund 6y ago

I thought people shouldn't post anything about bash on HN? The minute you post something about bash immediately you will draw out a whole bunch of folks from the wood works talking about how bash sucks and should never be used for anything more than 3 or 4 lines and how they replaced bash with python or some thing else, immediately in turn drawing out a bunch of other folks talking about how bash should be replaced with power shell and how you can parse objects better ... .

michaelcampbell 6y ago
Any programming topic spawns a bunch of comments about readability, or examples of code-golf. It's the nature of things.
exdsq 6y ago
I had a really fun project earlier in the year prototyping a load testing tool for a blockchain in Bash while 4 other developers wrote a ‘better’ one in Haskell. Bash can get results quickly, although it’s not maintainable! Still, a decent kloc or two of bash with performance results within the sprint.
heresie-dabord 6y ago
I had become annoyed by the Python bigots who will tell us about how easy to read their language is because its notation is clean, how all its functionality is "intuitive", how any combination of Python-based Rube-Goldberg Machine systems is the best. But then came the PowerShell people...

asicsp 6y ago

Others in this on-going series: * https://wizardzines.com/comics/environment-variables https://wizardzines.com/comics/environment-variables * https://wizardzines.com/comics/brackets-cheatsheet https://wizardzines.com/comics/brackets-cheatsheet * https://wizardzines.com/comics/bash-quotes https://wizardzines.com/comics/bash-quotes * https://wizardzines.com/comics/bash-if-statements/ https://wizardzines.com/comics/bash-if-statements/

GolDDranks 6y ago

I almost always write POSIX shell instead of bash for compatibility; it would be nice to see collections for tips and tricks specifically for POSIX shell. I know, for example, that -o pipefail doesn't exist in plain POSIX shell. I wonder what's are the best practices when you can't use it.

fpoling 6y ago
I also write my scripts to stay /bin/sh compatible. If this is not enough, then a real scripting language should be used, not bash. But I very much agree that lack of pipefail is painful. If I know that output on the left of the pipe is small, I read it into a variable and then use printf | right part. If the output can be big, I use a helper function to emulate it that I copy-paste.
asicsp 6y ago
>it would be nice to see collections for tips and tricks specifically for POSIX shell Check out https://freebsdfrau.gitbook.io/serious-shell-programming/ https://freebsdfrau.gitbook.io/serious-shell-programming/
sigzero 6y ago
> it would be nice to see collections for tips and tricks specifically for POSIX shell https://www.shellscript.sh/ https://www.shellscript.sh/ There is a Facebook group and I have emailed the author as well with questions. Nice guy.
m463 6y ago
ok some low-grade tips: cleanup="true" later: if $cleanup; then a ; else b ; fi or slightly uglier $cleanup && rm -f mytempfile I like one-line functions: die() { echo "$@" 1>&2; exit 1; }
peterwwillis 6y ago
Read the ash or dash man pages and just use what they give you. That's pretty much Bourne shell which is pretty much POSIX.
still_grokking 6y ago

_where 6y ago

If it’s not by default there’s a reason. Bash is literally running commands in a shell session. Think terminal session. When a command fails, would you want the terminal session to end? That’d be annoying. Same theory for unset variable. Referencing an undefined variable shouldn’t break your session. Why initialize it anyway? It’s more code to change if you don’t use it if you have to initialize it when it might not be needed. And, you’d have to call the script with A= just to check A wasn’t defined, and in the process now you have A assigned to an empty string, instead of only defaulting to one when called, which uses more memory and execution time. The pipeline doesn’t die because && and || and parens are seriously helpful for one-liners. Don’t think of it as a script. Think of it as a script for a shell.

pwdisswordfish4 6y ago
Sometimes the only reason is backwards compatibility.
Waterluvian 6y ago
Thinking of it that way, it becomes clear why you would want to switch to something like Python for more complex workflows. Bash is brilliantly useful for a lot of things. I’m not bashing it.
IshKebab 6y ago
A compelling argument for why Bash scripts shouldn't exist at all.
- chubot 6y ago
  https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-di...
  - IshKebab 6y ago
    Ah yes the "nothing can ever change" argument. It's the reason we still code everything in FORTRAN and COBOL.

tpoacher 6y ago

I'm surprised nobody's mentioned [expect](https://likegeeks.com/expect-command/ https://likegeeks.com/expect-command/) yet in this thread :)

x87678r 6y ago

bash is not fit for any moderately complex task. I dont understand its popularity. Normally sane people who like unit testing, CICD, SOLID principles, quality tools end up with a bunch of crappy scripts holding everything together. Please avoid.

chubot 6y ago
https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts https://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-di...
peterwwillis 6y ago
It's fantastic for simple tasks, which is why so many people use it. It also turns out that many complex tasks can be reduced to a collection of simple tasks, otherwise known as "a bunch of crappy scripts". I encourage all my teams to avoid built-in CI/CD features and plugins and just script what they want in a Docker container. It ends up being easier to maintain, breaks less often, and is more portable.
amelius 6y ago
> I dont understand its popularity. I think the only reason people use the shell for scripting is that ls -lsa /tmp is simpler to write than execute(["ls", "-lsa", "/tmp"]) or even execute("ls -lsa /tmp") > Normally sane people who like unit testing, CICD, SOLID principles, quality tools end up with a bunch of crappy scripts holding everything together. Please avoid. Totally agree!

arendtio 6y ago

I like the notion of 'set -e' and at the same time I hate it. First, because it behaves inconsistently across shells/versions [1] and second, because it doesn't always work as expected. For example, when you depend on the 'set -e' behavior in a function and call the function from within a condition, the 'set -e' has no effect at all. So you better don't count on 'set -e'. But don't expect me to follow my own advice, as not using 'set -e' isn't a good option either... [1] https://www.in-ulm.de/~mascheck/various/set-e/ https://www.in-ulm.de/~mascheck/various/set-e/

chubot 6y ago
Yup, I am fixing this with Oil: https://news.ycombinator.com/item?id=24740842 https://news.ycombinator.com/item?id=24740842 (2 of 4 problems fixed so far, more problems welcome) Try it out on your shell scripts and let me know what happens :) OSH has the "broken" POSIX/bash behavior to maintain compatibility, while Oil opts you in to the better error semantics.

chubot 6y ago

I'm working on fixing errexit / set -e in Oil: https://github.com/oilshell/oil/issues/709 https://github.com/oilshell/oil/issues/709 Summary of problems: 1. The "if myfunc" problem -- error checking is skipped. This is specified by POSIX, but it's undesirable. 2. The f() { test -d /tmp && echo "exists" } problem. The exit code of the function is probably not what you expect! 3. the local x=$(false) problem. This happens in all shells because of the definition of $?. (local is a shell builtin with its own exit code, not part of the language) This one is already fixed with shopt -s more_errexit in Oil. 4. the x=$(false) problem. This is a bash problem with command substitution. For example, dash and zsh don't have this problem. Test case: bash -c 'set -e; x=$(false); echo should not get here' Newer versions of bash fix it with inherit_errexit. Oil also implements inherit_errexit and turns it on by default! (in Oil, not OSH) ----- So 1 and 2 relate to the confusing combination of shell functions and errexit. And 3 and 4 relate to command subs. Oil has fixed these with opt-in OSH shell options (on by default in Oil), but not 1 and 2. If you know of any other problems, please chime in on the bug!

a1369209993 6y ago
> 2. The f() { test -d /tmp && echo "exists" } problem. The exit code of the function is probably not what you expect! The exit code is 0 (assuming /tmp/, stdout, /bin/test and /bin/echo are all working correctly; with /tmp1 it's 1), as expected; is this referencing a bug in sh and/or bash that I've fixed locally and then forgotten about? (Also, I'm pretty sure it should be: f() { test -d /tmp && echo "exists"; } unless the parse error for missing ';' was your point (I haven't bothered to fix that one, but maybe Oil has).)
- chubot 6y ago
  Yeah I didn't explain that very well. It's actually a variation on the problem here: https://news.ycombinator.com/item?id=24738274 https://news.ycombinator.com/item?id=24738274 except with a shell function rather than a subshell. Demo: https://github.com/oilshell/blog-code/blob/master/errexit https://github.com/oilshell/blog-code/blob/master/errexit /demo.sh#L21 Run ./demo.sh subshell-compare and func-compare. I have seen multiple people point it out as confusing. The problem is that in the first case, there are two exit codes: 1. false (suppressed on LHS of &&) 2. true And in the second case, there are 3: 1. false (suppressed on LHS of &&) 2. true 3. the exit code of the function/subshell, which is the exit code of the last statement, which is nonzero. This then CAUSES THE PROGRAM TO FAIL, whereas it didn't before. (edit: removed incorrect code) ----- So it's kind of like a "ghost" exit code created by the subshell/function! Which has an unexpected interaction with errexit. Does that make sense? I am not sure exactly how to fix it, but it will probably involve limiting && to stuff like this: if test -d /tmp && test -d /tmp/foo; then echo yes; fi And disallow it when standing alone, because it doesn't make much sense there, if errexit is on. Feel free to chime in bug with any possible solutions or more problems. ---- (And yes Oil fixes the brace problem with shopt -s parse_brace, which is on by default in Oil! Try it out and let me know if you like it :) )

major505 6y ago

This is usefull. I know a guy who didnt treat unset vairable, so in the script where he would remove with rm -rf /$dir-old-back his script removed all directories. The problem is that all his backups were in an external drive mounted in a directory, so he removed all his backups also. A hell month for him .

JeremyBanks 6y ago

https://dev.to/banks/stop-ignoring-errors-in-bash-3co5 https://dev.to/banks/stop-ignoring-errors-in-bash-3co5

cryptonector 6y ago

Except `set -e` is stupid because it is disabled all the way down when a command occurs in a conditional, meaning this: function foo { false; true; } foo || echo foo failed prints nothing! From the bash manual page: | If a compound command other than a subshell returns a non-zero status because a command failed while -e was being ignored, the shell does not exit. POSIX says the same thing, so this is true of all POSIX-y shells. This means you really have to check for errors you care about, and `set -e` is useless. Ugh!

lazyant 6y ago

I'd add TRAP DEBUG (running line by line) to the toolset

bewuethr 6y ago

set -e and set -u both come with their own pitfalls: https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 and https://mywiki.wooledge.org/BashFAQ/112 https://mywiki.wooledge.org/BashFAQ/112 I generally use them and think that overall, they have more benefits than drawbacks, but the odd time where I run into one of the pitfalls, debugging usually takes a while. ShellCheck already does a pretty good job of pointing out incorrect variable names, too.

westurner 6y ago

From https://twitter.com/b0rk/status/1312413117436104705 https://twitter.com/b0rk/status/1312413117436104705 : > TIL that you can use the "DEBUG" trap to step through a bash script line by line trap '(read -p "[$BASH_SOURCE:$LINENO] $BASH_COMMAND?")' DEBUG > [...] it does something very different than sh -x — sh -x will just print out lines, this stops before* every single line and lets you confirm that you want to run that line* >> you can also customize the prompt with set -x export PS4='+(${BASH_SOURCE}:${LINENO}) ' set -x With a markdown_escape function, could this make for something like a notebook with ```bash fenced code blocks with syntax highlighting?