6 ms·

Debugging Bash Like a Sire (2023)

91 points by gfalcao 1y ago

t43562 1y ago
I never thought of the idea of printing out a stack trace. A logging function is an example of such a good idea that is so obvious that I didn't think of it :-) I use -e sometimes but I really dislike scripts that rely on it for all error handling instead of handling errors and logging them. https://www.shellcheck.net/ https://www.shellcheck.net/ ^^ this tool has proven very useful for avoiding some of the most silly mistakes and making my scripts better. If you're maintaining scripts with other people then it is a great way of getting people to fix things without directly criticising them.
- oneshtein 1y ago
  You can use bash-modules if you like stack traces in bash.
naruhodo 1y ago
Since no commenter has remarked upon it yet, I draw your attention to the reddit comment [1] mentioned in the article. It is received wisdom that you should begin your scripts with "set -euo pipefail", and that turns out to be a footgun. [1] https://www.reddit.com/r/commandline/comments/g1vsxk/the_first_two_statements_of_your_bash_script/fniifmk/ https://www.reddit.com/r/commandline/comments/g1vsxk/the_fir...
neves 1y ago
Nice article. Do you have any shell debugging tip not listed in the article?
mjd 1y ago
I've been writing scripts in Bourne shell since the 1980s and in Bash since whenever it came along, and I feel like the most important thing I've learned about it is: don't. Sure, it can be done, and even done well, but why? There are better languages. Every time I write a shell script that grows to more than about 20 lines I curse myself for not having written it in Python. The longer I have waited before throwing it away and redoing it, the more I curse. This article says nothing to change my mind. I could build logging and stack traces in Bash. I admire the author's ingenuity. But again, why?
- orhmeh09 1y ago
  Because it's better at the task than Python is.
  - bigstrat2003 1y ago
    The only thing bash is better at than Python is very short scripts, like 10ish lines. Everything else it sucks at, due to the horrible syntax and various bash footguns.
  - mjd 1y ago
    That's just the problem! It is better at the task. Until it isn't, and "isn't" comes much too soon.
    - hn92726819 1y ago
      I've seen this sentiment a lot here. "Once shell is >n lines, port to python". My experience has been different. Maybe half of the scripts I write are better off in python while the other half are exponentially longer in python than bash. For example, anything to do with json can be done in 1 line of readable jq, while it could be 1, 5, or 20 lines in python depending on the problem. I'd just like to put that out there because half of the time, the >n metric does not work for me at all. My shell scripts range from ~5-150 lines while python are 100+
      - sgarland 1y ago
        Same. It’s mostly because if I have a shell script, while I’ll add some comments for tricky bits if needed, and maybe a `-h` function, that’s about it. In Python, though, I use the language’s features to make it as readable and safe as possible. Types, docstrings, argparse, etc. My thinking is that if I’m going to take the time to use a “proper” language, then I should make it bulletproof, otherwise I’d just stick with shell. My personal decision matrix for when to switch to Python usually involves the relative comfort of the rest of my team in both languages, the likelihood that future maintenance or development of the script will be necessary, and whether or not I’m dealing with inputs may change (e.g. API responses, since sadly versioning isn’t always a guarantee).
Y_Y 1y ago
I find that `bash -x` actially gives such a good trace that I rarely need anything else. Coupled with the ancient wisdom "bash for quick and dirty, once it gets too fancy switch to python", I use bash a lot and find it manages really well without external tools. Shoutouts to shellcheck though, for catching a lot of edge cases.
- ygritte 1y ago
  Yep, shellcheck + syntastic in vim makes for a great bash "IDE".
  - HankB99 1y ago
    https://github.com/vim-syntastic/syntastic https://github.com/vim-syntastic/syntastic looks like it is no longer maintained. Is there something more recent for syntastic? Agreed WRT shellcheck.
    - dcassett 1y ago
      It mentions ALE [1] as the logical successor to syntastic. I've been using ALE for many years. [1] https://github.com/dense-analysis/ale https://github.com/dense-analysis/ale
toolmantim 1y ago
Reminds me of Keith Pitt’s (Buildkite founder) latest streams: https://twitch.tv/magickeith/ https://twitch.tv/magickeith/
chasil 1y ago
It is always helpful in recording if any bash behavior changes when it is not running in POSIX mode. This is most common in Debian and Ubuntu, where ash is /bin/sh, and /bin/bash does not run in POSIX mode by default. Some behavior of legacy bash of the '80s, prior to POSIX.2, can be surprising. https://w3.pppl.gov/info/bash/Bash_POSIX_Mode.html https://w3.pppl.gov/info/bash/Bash_POSIX_Mode.html
- cerved 1y ago
  (D)ash is not bash --posix
agumonkey 1y ago
I had zero idea bash exposed the stack this way.. i'm utterly stumped.
rook_line_sinkr 1y ago
Many commenters recommend shellcheck For actually _testing_ the scripts or its functions, I recommend ShellSpec https://github.com/shellspec https://github.com/shellspec
- stebian_dable 1y ago
  There’s also bashunit https://bashunit.typeddevs.com https://bashunit.typeddevs.com
hn92726819 1y ago
> I tend to skip the -u flag as bash scripts often interact with global variables that are set outside my scripts. What? If globals are set outside the scripts, -u still works. If the author means they may or may not be defined outside the script, the ${VAR:-} construct allows it to expand to nothing if unset (just throw VAR=${VAR:-} at the top if you don't want to edit the body) Also, I do not like the function return based on error code: function ... { ... (( check_level >= current_level )) } Unless I'm reading this wrong, this is a bad idea if using set -e. This is a function and it should instead: return $(( check_level < current_level ))
cassianoleal 1y ago
> I tend to skip the -u flag as bash scripts often interact with global variables that are set outside my scripts. That's throwing the baby out with the bathwater. Instead, default the optional global variables with something like: "${GLOBAL_VAR:-}" That will satisfy the optionality of the variable whilst keeping the check for the cases you actually want them.
deleted 1y ago
[deleted]
bmoyles 1y ago
You can also skip the subshell invocation of date by using %(fmt)T from bash's printf: %(fmt)T -output the date-time string resulting from using FMT as a format string for strftime(3) The man page provides a bit more detail: %(datefmt)T causes printf to output the date-time string resulting from using datefmt as a format string for strftime(3). The corresponding argument is an integer representing the number of seconds since the epoch. Two special argument values may be used: -1 represents the current time, and -2 represents the time the shell was invoked. If no argument is specified, conversion behaves as if -1 had been given. This is an exception to the usual printf behavior. With that, timestamp=$(date +'%y.%m.%d %H:%M:%S') becomes printf -v timestamp '%(%y.%m.%d %H:%M:%S)T' -1
- gkfasdfasdf 1y ago
  That would indeed be faster, looks like it requires bash 4.2+
kjellsbells 1y ago
Curious that after 30 years the idea of a CPAN-like bash repository hasnt taken root. How many personal reimplementations of logging in bash are there by now? I suppose what is really tripping people up is that bash can show up on all kinds of runtimes, some of which have the external tools one might need (jq, logger, etc) and some of which don't. So then you go searching for a minimum standard that can be expected to be present. Maybe POSIX or gnu coreutils. Reminds me of the shell horrors of the late 1990s where every script had to figure out if sh was really ksh and what variant of UNIX it was running on, and therefore what commands and options were available. I swear this was one of the great things about Perl when it came along, it just worked. In 2025, I kind of see the attraction of single binaries like Go does. Ship the binary and be done. It is very un-UNIX I suppose (not so much golfing as having the beer cart drive you to the hole) but then again its not 1985 any more.
gkfasdfasdf 1y ago
Why not leverage the bash 'caller' builtin? It's meant for printing stack traces, e.g. #!/bin/bash die() { local frame=0 while caller $frame; do ((++frame)); done echo "$*" exit 1 } f1() { die "*** an error occured ***"; } f2() { f1; } f3() { f2; } f3 Output 12 f1 ./callertest.sh 13 f2 ./callertest.sh 14 f3 ./callertest.sh 16 main ./callertest.sh *** an error occured *** Via: https://bash-hackers.gabe565.com/commands/builtin/caller/ https://bash-hackers.gabe565.com/commands/builtin/caller/

91 points by gfalcao 1y ago

I never thought of the idea of printing out a stack trace. A logging function is an example of such a good idea that is so obvious that I didn't think of it :-) I use -e sometimes but I really dislike scripts that rely on it for all error handling instead of handling errors and logging them. https://www.shellcheck.net/ https://www.shellcheck.net/ ^^ this tool has proven very useful for avoiding some of the most silly mistakes and making my scripts better. If you're maintaining scripts with other people then it is a great way of getting people to fix things without directly criticising them.

oneshtein 1y ago
You can use bash-modules if you like stack traces in bash.

naruhodo 1y ago

Since no commenter has remarked upon it yet, I draw your attention to the reddit comment [1] mentioned in the article. It is received wisdom that you should begin your scripts with "set -euo pipefail", and that turns out to be a footgun. [1] https://www.reddit.com/r/commandline/comments/g1vsxk/the_first_two_statements_of_your_bash_script/fniifmk/ https://www.reddit.com/r/commandline/comments/g1vsxk/the_fir...

neves 1y ago

Nice article. Do you have any shell debugging tip not listed in the article?

mjd 1y ago

I've been writing scripts in Bourne shell since the 1980s and in Bash since whenever it came along, and I feel like the most important thing I've learned about it is: don't. Sure, it can be done, and even done well, but why? There are better languages. Every time I write a shell script that grows to more than about 20 lines I curse myself for not having written it in Python. The longer I have waited before throwing it away and redoing it, the more I curse. This article says nothing to change my mind. I could build logging and stack traces in Bash. I admire the author's ingenuity. But again, why?

orhmeh09 1y ago
Because it's better at the task than Python is.
- bigstrat2003 1y ago
  The only thing bash is better at than Python is very short scripts, like 10ish lines. Everything else it sucks at, due to the horrible syntax and various bash footguns.
- mjd 1y ago
  That's just the problem! It is better at the task. Until it isn't, and "isn't" comes much too soon.
  - hn92726819 1y ago
    I've seen this sentiment a lot here. "Once shell is >n lines, port to python". My experience has been different. Maybe half of the scripts I write are better off in python while the other half are exponentially longer in python than bash. For example, anything to do with json can be done in 1 line of readable jq, while it could be 1, 5, or 20 lines in python depending on the problem. I'd just like to put that out there because half of the time, the >n metric does not work for me at all. My shell scripts range from ~5-150 lines while python are 100+
    - sgarland 1y ago
      Same. It’s mostly because if I have a shell script, while I’ll add some comments for tricky bits if needed, and maybe a `-h` function, that’s about it. In Python, though, I use the language’s features to make it as readable and safe as possible. Types, docstrings, argparse, etc. My thinking is that if I’m going to take the time to use a “proper” language, then I should make it bulletproof, otherwise I’d just stick with shell. My personal decision matrix for when to switch to Python usually involves the relative comfort of the rest of my team in both languages, the likelihood that future maintenance or development of the script will be necessary, and whether or not I’m dealing with inputs may change (e.g. API responses, since sadly versioning isn’t always a guarantee).

Y_Y 1y ago

I find that `bash -x` actially gives such a good trace that I rarely need anything else. Coupled with the ancient wisdom "bash for quick and dirty, once it gets too fancy switch to python", I use bash a lot and find it manages really well without external tools. Shoutouts to shellcheck though, for catching a lot of edge cases.

ygritte 1y ago
Yep, shellcheck + syntastic in vim makes for a great bash "IDE".
- HankB99 1y ago
  https://github.com/vim-syntastic/syntastic https://github.com/vim-syntastic/syntastic looks like it is no longer maintained. Is there something more recent for syntastic? Agreed WRT shellcheck.
  - dcassett 1y ago
    It mentions ALE [1] as the logical successor to syntastic. I've been using ALE for many years. [1] https://github.com/dense-analysis/ale https://github.com/dense-analysis/ale

toolmantim 1y ago

Reminds me of Keith Pitt’s (Buildkite founder) latest streams: https://twitch.tv/magickeith/ https://twitch.tv/magickeith/

chasil 1y ago

It is always helpful in recording if any bash behavior changes when it is not running in POSIX mode. This is most common in Debian and Ubuntu, where ash is /bin/sh, and /bin/bash does not run in POSIX mode by default. Some behavior of legacy bash of the '80s, prior to POSIX.2, can be surprising. https://w3.pppl.gov/info/bash/Bash_POSIX_Mode.html https://w3.pppl.gov/info/bash/Bash_POSIX_Mode.html

cerved 1y ago
(D)ash is not bash --posix

agumonkey 1y ago

I had zero idea bash exposed the stack this way.. i'm utterly stumped.

rook_line_sinkr 1y ago

Many commenters recommend shellcheck For actually _testing_ the scripts or its functions, I recommend ShellSpec https://github.com/shellspec https://github.com/shellspec

stebian_dable 1y ago
There’s also bashunit https://bashunit.typeddevs.com https://bashunit.typeddevs.com

hn92726819 1y ago

> I tend to skip the -u flag as bash scripts often interact with global variables that are set outside my scripts. What? If globals are set outside the scripts, -u still works. If the author means they may or may not be defined outside the script, the ${VAR:-} construct allows it to expand to nothing if unset (just throw VAR=${VAR:-} at the top if you don't want to edit the body) Also, I do not like the function return based on error code: function ... { ... (( check_level >= current_level )) } Unless I'm reading this wrong, this is a bad idea if using set -e. This is a function and it should instead: return $(( check_level < current_level ))

cassianoleal 1y ago

> I tend to skip the -u flag as bash scripts often interact with global variables that are set outside my scripts. That's throwing the baby out with the bathwater. Instead, default the optional global variables with something like: "${GLOBAL_VAR:-}" That will satisfy the optionality of the variable whilst keeping the check for the cases you actually want them.

deleted 1y ago

[deleted]

bmoyles 1y ago

You can also skip the subshell invocation of date by using %(fmt)T from bash's printf: %(fmt)T -output the date-time string resulting from using FMT as a format string for strftime(3) The man page provides a bit more detail: %(datefmt)T causes printf to output the date-time string resulting from using datefmt as a format string for strftime(3). The corresponding argument is an integer representing the number of seconds since the epoch. Two special argument values may be used: -1 represents the current time, and -2 represents the time the shell was invoked. If no argument is specified, conversion behaves as if -1 had been given. This is an exception to the usual printf behavior. With that, timestamp=$(date +'%y.%m.%d %H:%M:%S') becomes printf -v timestamp '%(%y.%m.%d %H:%M:%S)T' -1

gkfasdfasdf 1y ago
That would indeed be faster, looks like it requires bash 4.2+

kjellsbells 1y ago

Curious that after 30 years the idea of a CPAN-like bash repository hasnt taken root. How many personal reimplementations of logging in bash are there by now? I suppose what is really tripping people up is that bash can show up on all kinds of runtimes, some of which have the external tools one might need (jq, logger, etc) and some of which don't. So then you go searching for a minimum standard that can be expected to be present. Maybe POSIX or gnu coreutils. Reminds me of the shell horrors of the late 1990s where every script had to figure out if sh was really ksh and what variant of UNIX it was running on, and therefore what commands and options were available. I swear this was one of the great things about Perl when it came along, it just worked. In 2025, I kind of see the attraction of single binaries like Go does. Ship the binary and be done. It is very un-UNIX I suppose (not so much golfing as having the beer cart drive you to the hole) but then again its not 1985 any more.

gkfasdfasdf 1y ago

Why not leverage the bash 'caller' builtin? It's meant for printing stack traces, e.g. #!/bin/bash die() { local frame=0 while caller $frame; do ((++frame)); done echo "$*" exit 1 } f1() { die "*** an error occured ***"; } f2() { f1; } f3() { f2; } f3 Output 12 f1 ./callertest.sh 13 f2 ./callertest.sh 14 f3 ./callertest.sh 16 main ./callertest.sh *** an error occured *** Via: https://bash-hackers.gabe565.com/commands/builtin/caller/ https://bash-hackers.gabe565.com/commands/builtin/caller/