7 ms·
Bash Debugging
- bubblebeard 3y agoThis is neat, never considered one could step debug a bash script. Kudos for sharing!
- memco 3y agoGood stuff! I use set-x frequently and have used a similar thing to die (but Julia’s version is nicer). I’ll consider using the debugger thing but stepping through a bash script line by line sounds a bit tedious. Perhaps less so than having to reread a log and rerun the script a bunch.
- chatmasta 3y agoThe line-by-line debugging would probably only be useful for a particular section of your script that you're trying to fix. In that case, you can remove the trap at the end of it with `trap - DEBUG`
- halostatue 3y agoI often add a fail-unless function: fail-unless() { local result "$@" result=$? if ((result != 0)); then echo >2&1 "Failed ${result} with command '$*'." exit ${result} fi } That way, I know exactly what failed in the script.
- sureglymop 3y agoYou do e.g. `fail-unless somecommand`. The result (exit/return code) is captured in the function and based on that, the function logs and exits or not.
- ykonstant 3y agoYou probably meant to reply to BeefySwain, right?
- BeefySwain 3y agoHow does this work exactly? What calls that function and when?
- klysm 3y agoI always put set -euxo pipefail at the top of my bash scripts. It makes some conditional testing more difficult but it has paid for itself many times over just because of pipefail
- ddlsmurf 3y agoYou can also set it for a bunch of lines then deactivate it with `set +x`. It gets rather tedious otherwise...
- mellutussa 3y agoThis is a lifesaver. Although I save the -x until I really need to see all the garbage debug output.
- klysm 3y agoI find it’s usually worth the noise to catch weirdness
- mrAssHat 3y agohttps://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail https://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail
- klysm 3y agoThanks for the reference! Seems like a really good resource. I disagree with the reasoning about pipefail though. If I expect a command to return non-zero exit code I'd rather be explicit about it.
- Brian_K_White 3y agoI have almost that same die() in every script, except I call it abrt(). Maybe I'll switch to die() since it's shorter. Mine also prepends $0 and sometimes I use printf or echo -e so I can pass larger more complex messages with linefeeds and escape codes etc.
- asicsp 3y agoSee also: Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected? https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 What are the advantages and disadvantages of using set -u (or set -o nounset)? https://mywiki.wooledge.org/BashFAQ/112 https://mywiki.wooledge.org/BashFAQ/112 Safe ways to do things in bash https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... Better Bash Scripting in 15 Minutes https://robertmuth.blogspot.com/2012/08/better-bash-scripting-in-15-minutes.html https://robertmuth.blogspot.com/2012/08/better-bash-scriptin... Writing Robust Bash Shell Scripts https://www.davidpashley.com/articles/writing-robust-shell-scripts/ https://www.davidpashley.com/articles/writing-robust-shell-s...
- ndsipa_pomu 3y agoYou missed the most important tip - use ShellCheck on every script you write: https://www.shellcheck.net/ https://www.shellcheck.net/ Personally, I'm a big fan of BASH3 boilerplate: https://github.com/kvz/bash3boilerplate https://github.com/kvz/bash3boilerplate It's fine for BASH versions above v3 and provides decent logging though I typically extend the script so that I can pipe long running commands into its logging framework. It also ensures that you specify the "help" options correctly as it parses the usage information to process the command line arguments with support for short and long options.
- asicsp 3y agoOh yeah, shellcheck is a must have tool.
- ndsipa_pomu 3y agoI think of it as a gateway to writing better scripts. When you first run it and it highlights what it considers to be a problem, you end up reading why it considers it to be a problem and that clues you in on some of the many footguns that Bash has.
- jclulow 3y agoIt's actually possible to produce a stack trace of sorts, if you use a lot of bash functions. One possible implementation is: https://github.com/TritonDataCenter/sdc-headnode/blob/master/buildtools/lib/error_handler.sh https://github.com/TritonDataCenter/sdc-headnode/blob/master...
- drizzleword 3y agoAnother stack trace implementation [1] that allows you to write: some-command || fail "message" to produce a stack trace and exit the shell in case of non-zero exit status from some-command, or write some-command || softfail "message" || return $? in case you want to produce a stack trace and return from the function. [1]: https://github.com/runag/runag/blob/main/lib/fail.sh https://github.com/runag/runag/blob/main/lib/fail.sh
- colordrops 3y agoIs there a reason bash is still the de facto shell scripting language other than sheer momentum of legacy? I'm able to get what I need done in it, but it's clunky and the syntax is horrid. I guess it forces you to move to a proper language once scripts grow to a certain size/complexity, so perhaps it's by design?
- awesomeMilou 3y agoAre you sure it's bash? Most scripts on FreeBSD's are written for sh, which I feel is much more widely supported due to being part of the POSIX standard. Bash is just popular I think.
- HankB99 3y agoBash is pretty much expected to be installed on any Linux distro. On FreeBSD (and likely other BSDs) it is an optional install. If you want a script to run on either, use sh. If strictly Linux, bash is probably safe. Bash/sh is good for when you need to combine some commands and what needs to be done can be accomplished mostly by CLI commands with a little glue to tie them together. Some times it is surprising what can be accomplished. I wrote a program to import pictures from an SD card on Windows using C#, copying pictures to C:\Pictures\YYYY\MM\DD according to the EXIF data or failing that, file time stamp. I tried to port it to Linux but ran into problems trying to connect to the EXIF library. After struggling with that, I rewrote it using sh, some EXIF tool and various file utilities. It took 31 lines, about half of which were actual commands and the rest comments or white space. A much bigger project is a script to install Debian with root on ZFS. It's mostly a series of CLI commands with some variable substitution and conditionals depending on stuff like encrypted or not.
- xk_id 3y ago> Bash/sh is good for when you need to combine some commands and what needs to be done can be accomplished mostly by CLI commands with a little glue to tie them together. Once I’ve learned bash, I realised how much more problems i could solve, in addition to a majority of old ones. It’s an entirely new level of “computer literacy”; and a more genuine one.
- jph 3y agoGood info. You can improve your debugging by using exit codes like this: # die: print error message to stderr, then exit with error code. # example: die 69 "Service unavailable." die() { n="$1" ; shift ; >&2 printf %s\\n "$*" ; exit "$n" } Many more shell script exit codes and helper functions: https://github.com/SixArm/unix-shell-script-kit/blob/main/unix-shell-script-kit https://github.com/SixArm/unix-shell-script-kit/blob/main/un...
- ykonstant 3y agoThat's a nice list; I guess every experienced user has their helper functions. However, I have a small criticism for the philosophy of that `die`: `die` functions should pass by default the exit code of the failed command, and not silence its error output. If I want to give my own meaning to the command failure in a large script for instance, I will use a different, more specialized `die`. My own die is roughly as follows: __errex() { printf 'Fatal error [%s] on line %s in '"'"'%s'"'"': %s\n' \ "${1:-"?"}" \ "${2:-"?"}" \ "${3:-"unknown script"}" \ "${4:-"unknown error"}" >&2 ; exit "${1:-1}" } alias die='__errex "$?" "${LINENO}" "$0"'
- bewuethr 3y agoThere's also a fairly powerful gdb style actual debugger: https://bashdb.sourceforge.net/ https://bashdb.sourceforge.net/
- E39M5S62 3y agoWe use a couple nice home-grown functions in ZFSBootMenu to help debug things. We have a zdebug logging function that's peppered liberally throughout the code base - https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootmenu/lib/kmsg-log-lib.sh#L16-L38 https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootme... Hitting ctrl-t on our main menu will, when booting with debug logging enabled, show a screen like this: https://i.imgur.com/Ge75zkP.png https://i.imgur.com/Ge75zkP.png We also have a flamegraph profiling mechanism that can be enabled with https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootmenu/profiling/profiling-lib.sh https://github.com/zbm-dev/zfsbootmenu/blob/master/zfsbootme... . That will dump data to a serial port, which when re-assembled, can be used to produce a graph like https://raw.githubusercontent.com/zbm-dev/zfsbootmenu/master/media/flamechart-v1.12.svg https://raw.githubusercontent.com/zbm-dev/zfsbootmenu/master... Bash is suprisingly flexible.
- thaumaturgy 3y agoThe `die()` trick is good, but bash has an annoying quirk: if you try to `exit` while you're inside a subshell, then the subshell exits but the rest of the script continues. Example: #!/bin/bash die() { echo "$1" >&2; exit 1; } cat myfile | while read line; do if [[ "$line" =~ "information" ]]; then die "Found match" fi done echo "I don't want this line" ..."I don't want this line" will be printed. You can often avoid subshells (and in this specific example, shellcheck is absolutely right to complain about UUOC, and fixing that will also fix the die-from-a-subshell problem). But, sometimes you can't, or avoiding a subshell really complicates the script. For those occasions, you can grab the script's PID at the top of the script and then use that to kill it dead: #!/bin/bash MYPID=$$ die() { echo "$1" >&2; kill -9 $MYPID; exit 1; } cat myfile | while read line; do if [[ "$line" =~ "information" ]]; then die "Found match" fi done echo "I don't want this line" ...but, of course, there are tradeoffs here too; killing it this way is a little bit brutal, and I've found that (for reasons I don't understand) it's not entirely reliable either.
- whatindaheck 3y agoCould killing the PID like that create zombies?
- ykonstant 3y agoPerhaps, but in any case I would never write code like this. First of all, sending sigkill is literally overkill and perpetuates a bad practice. Send `TERM`. If it doesn't work, figure out why. Secondly, subshells should be made as clear as possible and not hidden in pipes. Related, looping over `read` is essentially never the right thing to do. If you really need to do that, don't use pipes; use heredocs or herestrings. Fourth, if you cannot avoid subshells and you want to terminate the full script on some condition, exit with a specific exit code from the subshell, check for it outside and terminate appropriately.
- gpvos 3y ago> looping over `read` is essentially never the right thing to do Why? I do it quite often, though admittedly usually in one-time scripts.
- mmmpetrichor 3y agoIf I ever have to debug anything in bash. I stop using bash hah.
- sureglymop 3y agoDoesn't make sense. What if you get a script someone else wrote? Printing every command and confirming as you run every command is a great idea. And, unfortunately shell has become the norm in CI/CD environments, pipelines etc. Can be convenient at times but can also be inconvenient and confusing as these scripts don't run in interactive shells.
- bigstrat2003 3y ago> And, unfortunately shell has become the norm in CI/CD environments, pipelines etc. A pipeline which relies on shell is not worth using, tbh. That's how much shell sucks.
- ndsipa_pomu 3y agoWhat language would you base a pipeline on? I agree that bash sucks, but have yet to find anything to replace it that doesn't increase complexity and version problems.
- ykonstant 3y agoI find such reasoning backwards. Indeed, shell scripting is not friendly to debugging. But ensuring correctness of shell scripts is essential: usually, they touch part of your "$HOME" or system folders and do tons of I/O, some of it destructive. I find it baffling to see people write careless scripts; sometimes using `rm` for cleanup with unquoted parameters, or much worse, dangerous uses of `mv`.
- bigstrat2003 3y agoI believe OP's point was that any shell script complex enough to require debugging should not be a shell script any more.
- ykonstant 3y agoThe `trap DEBUG` thing is pretty interesting; I almost always write POSIX code, so I don't get to play with such tricks. Does anybody know of some wizardry that could mimic this in arbitrary POSIX compliant shells?
- c0l0 3y agoI shared https://johannes.truschnigg.info/writing/2021-12_colodebug/ https://johannes.truschnigg.info/writing/2021-12_colodebug/ a while ago here - you might appreciate it for another means to make your bash scripts easier to understand while they execute.
- ketanmaheshwari 3y agoPlug and slightly related. I once created a bash pipeline debugger that preserves the intermediate outputs. Has a few limitations but maybe generally useful: https://github.com/ketancmaheshwari/pd https://github.com/ketancmaheshwari/pd
- chlorion 3y agoGentoo has a script that you can source at /lib/gentoo/functions.sh that provide various helper methods, mostly for printing messages, and it provides nice little green and red starts to indicate whether something has succeeded of failed. I use functions.sh in all of my scripts that are known to be running on Gentoo only and it makes them feel Gentoo-y and is useful in general.
- hn_acker 3y agoOff-topic: The alt text for the image says > Image of a comic. To read the full HTML alt text, click "read the transcript". but I can't find any button relating to a transcript.
- schneems 3y agoI recommend shellcheck as well. It might not catch your problem, but it will point out possible problems. Also I recommend: rewriting scripts in another language. At work we are converting bash scripts to rust and while it’s a high ramp-up time, the resulting code is much easier to maintain and I have a much higher level of confidence in them. Bash is still good for quick scripts but once you hit 100 lines or so you really deserve a language with stronger guarantees.
- riperoni 3y agoI agree, but we gotta tell that to CI/CD engineering and yaml-pipelines
- sestep 3y agoWhen I was at Facebook, I wrote a Python script to extract shell scripts from GitHub Actions workflows, so we could run them all through ShellCheck: https://github.com/pytorch/pytorch/blob/69e0bda9996865e319db6afa318c8e27ee38d002/tools/extract_scripts.py https://github.com/pytorch/pytorch/blob/69e0bda9996865e319db...
- gkfasdfasdf 3y agoYou can use the bash 'caller' builtin to print a stack trace, e.g.: https://github.com/theimpostor/templates/blob/main/bash%2Ftemplate.sh https://github.com/theimpostor/templates/blob/main/bash%2Fte...
- graton 3y agoVery useful is: PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}: ' When using `set -x` this makes it so that it shows the filename, function name, and line number. Which in larger Bash scripts can be quite handy in debugging.
- e40 3y agoWOW! Thanks! Power user of BASH for a long time and I did not know this.
- egberts1 3y agoPyCharm (of JetBrain IDEA) with the Bash Debugger add-on is an awesome Bash/Profile debugger. The ability to see all variable names and its value is priceless. https://www.jetbrains.com/help/idea/shell-scripts.html https://www.jetbrains.com/help/idea/shell-scripts.html
- Scrutiny6707 3y agoThe trap command with printing the lines didn't work. Had to change it to drop the parantheses, then it worked, like this trap 'read -p "[${BASH_SOURCE:-}:${LINENO:-}] ${BASH_COMMAND:-}"' DEBUG