12 ms·
“Exit traps” can make your Bash scripts more robust and reliable (2013)
- DavideNL 3y agoFyi, related article...: "Minimal safe Bash script template" - https://news.ycombinator.com/item?id=25428621 https://news.ycombinator.com/item?id=25428621
- smcleod 3y agoYep, I use these all the time, they’re very useful indeed.
- thangalin 3y agohttps://github.com/DaveJarvis/keenwrite/blob/main/scripts/build-template https://github.com/DaveJarvis/keenwrite/blob/main/scripts/bu... My template script provides a way to make user-friendly shell scripts. In a script that uses the template, you define the dependencies and their sources as comma-separated values: DEPENDENCIES=( "gradle,https://gradle.org" "warp-packer,https://github.com/Reisz/warp/releases" "tar,https://www.gnu.org/software/tar" "wine,https://www.winehq.org" "unzip,http://infozip.sourceforge.net" ) You define the command-line arguments: ARGUMENTS+=( "a,arch,Target operating system architecture (amd64)" "o,os,Target operating system (linux, windows, mac)" "u,update,Java update version number (${ARG_JAVA_UPDATE})" "v,version,Full Java version (${ARG_JAVA_VERSION})" ) You define the "execute()" method that is called after the arguments are parsed: execute() { // Make the computer do the work. return 1 } If the script takes arguments, handle each one individually: argument() { local consume=2 case "$1" in -a|--arch) ARG_JAVA_ARCH="$2" ;; -o|--os) ARG_JAVA_OS="$2" ;; esac return ${consume} } Then call the template's main to start the script rolling: main "$@" For 99% of the scripts I write, this provides: * Built-in software dependencies verification. * Instructions to the user when requirements are missing. * Simple command-line argument parsing. * Help and logging using ANSI colour. Here's a complete script that builds the Windows, Linux, and Mac installers for my Markdown editor: https://github.com/DaveJarvis/KeenWrite/blob/main/installer.sh https://github.com/DaveJarvis/KeenWrite/blob/main/installer.... There's a write-up about creating the script that has a lot more details about how the template works: https://dave.autonoma.ca/blog/2019/05/22/typesetting-markdown-part-1/ https://dave.autonoma.ca/blog/2019/05/22/typesetting-markdow... Note that it is technically possible to improve the scripts such that handling individual arguments can be done in the template itself. This would require a slightly different argument definition semantics: ARGUMENTS+=( "ARG_JAVA_ARCH,a,arch,Target operating system architecture (amd64)" "ARG_JAVA_OS,o,os,Target operating system (linux, windows, mac)" "usage=utile_usage,h,help,Show this help message then exit" ) By detecting an `=` symbol for the first item in the lists, it's possible to know whether a command-line argument is assigning a variable value, or whether it means to perform additional functionality. (PR welcome!)
- snapcaster 3y agoVery cool! didn't know about these
- sigg3 3y agoYes. I use them for cleanup in every non-trivial script I write.
- phh 3y agoThis 100%. I'll complete with patterns I'm using for exit traps: - for temporary files I have a global array that lists files to remove (and for my use case umount them beforehand) - in the EC2 example, I add a line with just "bash", so I have an env with the container still running to debug what happened and I just need to close that shell to clear the allocated resources
- tommica 3y agoThis is very useful to know - thanks for sharing!
- arp242 3y agoAn annoying thing about bash is that EXIT will also run on SIGINT (^C), which most other shells won't (in my reading it's also not POSIX compliant, although the document is a bit vague). Some might argue this is a feature, but IMHO it's a bug – sometimes you really don't want cleanup to happen so people can inspect the contents of temporary files for debugging. Because trap doesn't pass the signal information to the handler it's hard to not do cleanup on SIGINT, so it's certainly less flexible, and it's an annoying incompatibility between bash and any other shell. Also, zsh has a much nicer mechanism for the common case: { echo lol } always { # Ensure *all* temporary files are cleaned up. nohup rm -rf / & }
- c5c3c9 3y ago[flagged]
- jcotton42 3y ago> Because trap doesn't pass the signal information to the handler it's not hard not to do cleanup on SIGINT Did you mean "it's hard" instead of "it's not hard"?
- arp242 3y agoOops, yes, thanks; seems a "not" got duplicated in editing – still within edit window.
- ggm 3y agoSome temporary file remover. Lol indeed
- arp242 3y ago"Keep non-temporary files intact" was not part of the design document.
- wkat4242 3y ago// Thinks about that type I typed rm -rf /<space>something by mistake. It took a few seconds before I thought... "Why does it take that long for only a handful of files?" I never did that again. Had my DOS filesystem mounted under Linux too (yes that long ago), and I spent a few days guessing the first letter of each deleted file with norton disk doctor or undeleter or something. That was fun (FAT16 filesystems overwrote the first letter of each filename to delete it) At least it wasn't a mistake I made at work on some production thing. Though there is a reason I make all the desktops on windows production servers bright red. One time I was tired and shut down "my laptop" forgetting I was still logged into a remote server 200km away..... :/ Of course the iLO wasn't hooked up but I was extremely happy to find that HP servers listen to wake on LAN even when they're off. Another one for the never again books :P
- cvalka 3y ago[flagged]
- anaganisk 3y agoA good read before dismissing http://n-gate.com/software/2017/ http://n-gate.com/software/2017/
- arsome 3y agoWhat's the gripe with Let's Encrypt? Certificate transparency?
- BenjiWiebe 3y agoI disagree with his disagreement. I'm not able to overthrow my government to make it illegal for my only ISP to stop intercepting my traffic. HTTPS simply makes it impossible for my ISP to add stuff to the page in transit.
- cvalka 3y agoSome of his arguments are wrong, some are not even wrong, some are absurd. Plus he seems to be an asshole. Being an asshole and wrong at the same time is not a great combination.
- anaganisk 3y agoI think you would be very interested to read their takes on HN articles and Fossdem :p
- mttjj 3y agoCan you expand on your first sentence with some reasons or justifications for stating this?
- cvalka 3y agoFor most use cases a guy like him writes a shell script, there's already some well written software. F.e. in his case, he'd used something like Dagu. Don't write shell scripts. If you want to do some programming, pick a proper programming language.
- chasil 3y agoI used an exit trap to kill an SSH agent that I am running, and I noticed that dash did not kill if the script was interrupted, but only if it ran to successful completion. I asked on the mailing list if this was expected behavior, and it turns out that POSIX only requires EXIT to run on a clean shutdown; to catch interruptions, add more signals. trap 'eval $(ssh-agent -k)' EXIT INT ABRT KILL TERM
- js2 3y agoI think you want: trap 'ssh-agent -k' EXIT INT TERM I don't see any reason for the eval as "ssh-agent -k" doesn't return anything useful you want the shell to evaluate.
- chasil 3y agoThat's not what the eval is for. The "ssh-agent -k" command will emit shell commands that the shell must then execute which will kill the agent daemon and unset the socket environment variable.
- leodag 3y ago> The "ssh-agent -k" command will emit shell commands Does it really? I've executed it here and it just runs kill, doesn't emit any bash. Running just ssh-agent (without any args) does that though, which is what's probably causing the confusion.
- chasil 3y agoI am on OpenBSD 7.2, and I see: $ eval $(ssh-agent) Agent pid 56785 $ ssh-agent -k unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 56785 killed; The correct processing of that output requires an eval. Did you have any other questions?
- pmarreck 3y agoWhy do you need to eval it? $(ssh-agent) won’t substitute that with the stdout and run that?
- ipnon 3y agoI just learned about these through “pair” programming with ChatGPT. It is the quintessential ML-enhanced programming trick: Using some old, robust language feature I’m skilled enough to grok but never had the time to learn about through endless documentation spelunking. My opinion is that LLM pair programming is most or maybe only beneficial to already skilled programmers. ChatGPT can open the door for you, but it can’t show you where the door is. I needed the experience to ask it for a Bash script that handles exit codes gracefully, which is not a question all junior programmers would be able to ask.
- abathur 3y agoI like combining this with a bash implementation of an event API (https://github.com/bashup/events https://github.com/bashup/events). This makes it easy/idiomatic, for example, to conditionally add cleanup as you go. Glossing over some complexity, but roughly: add_cleanup(){ event on cleanup "$@" } trap "event emit 'cleanup'" HUP EXIT start_postgres(){ add_cleanup stop_postgres # actually start pg } start_apache(){ add_cleanup stop_apache # actually start apache } I wrote a little about some other places where I've used it in https://www.t-ravis.com/post/shell/neighborly_shell_with_bashup.events/#a-quick-look-at-bashupevents https://www.t-ravis.com/post/shell/neighborly_shell_with_bas... and https://t-ravis.com/post/nix/avoid_trap_clobbering_in_nix-shell/ https://t-ravis.com/post/nix/avoid_trap_clobbering_in_nix-sh... (though I make the best use of it in my private bootstrap and backup scripts...)
- e12e 3y agoThank you for sharing - if i understand the code, the queue is serialized into bash variable(s) (arrays)? I must admit I find the code somewhat painfully terse and hard to read. Still, interesting idea. I wonder if using a temporary SQLite/Berkeley DB/etc for queue might generalize the idea to a "Unix" event system - allowing other programs and scripts to use it for coordinating? (Like logger(1) does for logging)?
- abathur 3y agoYep. Definitely something you can do manually, but the API makes it easier to reason about and coordinate across otherwise disconnected/unrelated code.
- e12e 3y agoWas thinking more as a "global" queue (like how psql/libpq will go look for a socket to local postgres in "the right place") - and a binary/program "event" could "magically" store ("on") and process ("emit") events in a db file /tmp/event.<namespace><random>.sqlite3 - creating/initializing or re-using db file as needed... So keep the api, but support cross process queues, more or less.
- telotortium 3y ago@redsymbol your site has a TLS certificate error. On Chrome I get NET::ERR_CERT_COMMON_NAME_INVALID because your certificate is from mobilewebup.com Otherwise a good article. I use the following code to enable passing the signal name to the trap handler, so that I can kill the Bash process with the correct signal name, which is best practice for Unix signal handling (EXIT would have to be handled specially in `sig_rekill`): # Set trap for several signals and pass signal name to trap function. # https://stackoverflow.com/a/2183063/207384 trap_with_arg() { func="$1" ; shift for sig ; do trap "$func $sig" "$sig" done } sig_rekill() { # Kill whole process group. trap "$1"; kill -"$1" -$$ } # Catch signal and kill whole process group. trap_with_arg sig_rekill HUP INT QUIT PIPE TERM
- jeron 3y agoI thought exit traps were just SPACs
- JohnMakin 3y agoI like to use these in combination with set -e and report the error that happened to whatever is capturing stdout for logging. You can report the error code with $? at the start of your trap, IIRC.
- deleted 3y ago[deleted]
- waselighis 3y agoI wish there was a nicer shell scripting language that simply transpiled to Bash and would generate all this boilerplate code for me. There is https://batsh.org/ https://batsh.org/ which has a nice syntax but it doesn't even support pipes or redirection, making it pretty worthless for shell scripting. I haven't found any other such scripting languages.
- paulddraper 3y agoWhat's the difference between that and Go?
- burnished 3y agoGo doesnt seem related at all?
- franknord23 3y agoI wish Bash had 'defer' like Go.
- cedws 3y agohttps://cedwards.xyz/defer-for-shell/ https://cedwards.xyz/defer-for-shell/ Enjoy. (blog post is mine)
- gscho 3y agoOff topic but I really enjoy the lofi website design!
- filereaper 3y ago>The secret sauce is a pseudo-signal provided by bash, called EXIT, that you can trap; commands or functions trapped on it will execute when the script exits for any reason. "Secret Sauce", why is this secret at all. Nothing against the author who's helping the ecosystem here, but is there an authoritative guide on Bash that anyone can recommend? Hopefully something that's portable between Mac & Linux. The web is full of contradictory guides and shellcheck seems to be the last line of defense. - https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- r3trohack3r 3y agoI don't know if it checks the right box as authoritative, but my goto guide has been tldp: https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/
- inimino 3y agoIt's secret enough to be well documented in the man page. The real question is, why do people look to random web pages prior to having digested everything in the manual? People used to say "rtfm" all the time, this would be regarded as shockingly rude in today's tech culture but it was a valuable public service to have it repeated, like being reminded to eat your vegetables.
- patrakov 3y ago> The real question is, why do people look to random web pages prior to having digested everything in the manual? Easy! It is a reference manual. It documents every feature, even those that you should not use, does not discuss pitfalls, and does not discuss the best practice. Even worse, there is sometimes a disagreement whether using a particular feature is a good practice. Often there is a factor of adjusting your code style to something that is not too advanced for other people to review.
- inimino 3y agoThe funny thing about this comment is that man pages do all of the things you mention. There seems to be a common sentiment that cargo culting random opinions from the internet is a "best practice" and that no code reviewer should ever have to learn anything new during the process. Both of these opinions are in my experience a fast track to a culture of mediocrity.
- rgrau 3y agoI couldn't find a way to have more than one callback per signal, and created a system to have an array of callbacks: https://github.com/kidd/scripting-field-guide/blob/master/book.org#array-of-callbacks-on_exit https://github.com/kidd/scripting-field-guide/blob/master/bo... A nice bonus is that it also keeps the return value of the last non-callback function, so your script behaves better when called from other scripts.
- sour-taste 3y agoShould go without saying, but don't rely on this for anything critical. It's not guaranteed this will run, even on successful completion of the script. Simple example: power is cut between the last line of the script and before the trap runs. Just a heads up
- hermannj314 3y agoIf you were building a critical system, what would do if power is cut after the last line of a script runs?
- munk-a 3y agoIdempotency is usually the best approach - have each step of a process examine the state of the disk and act only if it's an appropriate input then output something that isn't an appropriate input. Assuming you can ignore midstream corruption (which can safely be done by adding an atomically safe linking layer on top) then if power suddenly cuts out reboot and run the script until a clause finds an appropriate input and executes it continuing from there. The concepts are simple, the implementation is a pain and (honestly) if you need something truly resilient you're probably better off leaning on a system that can provide that guarantee for you (like using a database for state storage).
- dharmab 3y agoNot even a power cut- a SIGKILL (such as an OOMKill) is enough to cause the trap not to be run.
- asylteltine 3y agoThis is great and I have used it but the better advice is stop using bash scripts for anything other than your own personal scripts and use real languages for anything important.
- throwawaaarrgh 3y agoDon't use traps unless you have to. They are subtly complex and require a great deal more code to deal with edge cases. There is almost always a simpler way to accomplish what you want. If Bash has taught me anything, it's that many advanced features should seldom be used. Always resist the temptation to be fancy.
- thelastparadise 3y agoHow have exit traps come back to bite you?
- throwawaaarrgh 3y agoThe first is that the trap can come at any time. You can't assume at what point in the script it was running, so you have to test for different cases to find out what you now can/should do. Forget an edge case and now you've got an extra bug. Not using traps, it's clearer what happens at specific points in the execution of the rest of the code, so simpler to reason about how to deal with those cases as/where they happen. The second is different events can trigger an exit trap, and those may have different implications on what's going on. The third is there's parts of standards left out about what happens during/after a trap or when they get called, what data you have available, and different implementations can behave differently. Fourth is that sometimes people will use an exit trap to, say, report on a failure, but they may have lost context of what block they were in when it exited, and now the error reporting doesn't tell you everything you wanted to know. I can't remember more specifics atm because I stopped using them like a decade ago. I'll still use them to clean up temp files, but I also have to add the cleanup logic to the start of the script in case it didn't run.
- xmprt 3y agoAs with most things in programming, it sounds like if you use the wrong tool for the wrong job, then you're prone to writing bugs. Using traps is a great idea when used properly and dismissing it outright isn't doing anyone any favors.
- cabalamat 3y agoJust use Python or any other proper high-level language that has proper control structures.
- AHOHA 3y agoSometimes you can “just use python”, like openwrt script or similar.
- cabalamat 3y agoAccording to the openwrt website[1], packages exist for Erlang, Lua, node.js, Perl, PHP8, Python Ruby and Tcl. [1]: https://openwrt.org/packages/index/start https://openwrt.org/packages/index/start
- pwdisswordfishc 3y agoOpenWrt has both CPython and MicroPython packaged.
- michaelmior 3y ago> and may have security implications too While it's certainly true that leaving around files with sensitive data is a security problem, you probably don't want to put sensitive data in /tmp to begin with.
- vbezhenar 3y agoWhy not?
- armchairhacker 3y agoThe program could get paused mid-execution. Moreover, I’m pretty sure a malicious process can put file watchers in /tmp and read all written contents.
- ericbarrett 3y agoIf your script calls umask 077 ...before creating temp files then they won't be world-readable. Still lots of pitfalls. (What user are you running as, and who else is running as that user? What's the mount point file system, and does it have POSIX permissions? Why are you persisting secrets to disk in the first place? Etc.)
- dredmorbius 3y agoIt's possible to gather some information from a directory to which an attacker has write access, though I'd have to look up details. In general, this can usually be mitigated to some extent by creating a directory to which only the owner has access. There are a number of ... interesting ... other circumstances which you might want to consider: - /tmp is mounted as a ramdisk / memory-only filesystem. This is guaranteed not to persist over reboots, though there may be residual artefacts in memory even after a power-off. That last isn't a significant concern for many people, though it may turn up for others. - /tmp is a network share. This is uncommon, but NFS + sudo across shared systems means that a user on a remote system may be able to assume your credentials and access or modify your data. rootsquash means that root isn't available, but sudo means that any UID can be defined. - Various filesystem permissions or limitations may or may not apply to /tmp. I tend to prefer mounting /tmp as its own filesystem, with nodev and nosuid set. There might also be noexec, which can foul up a lot of temporary installation scripts. An alternative is for users to define their own preferred temporary directory. I usually include ~/tmp under $HOME.
- honkycat 3y agoWhat if instead of using a bunch of features bolted onto a shitty scripting language, we just used a real language like Python? I've read enough hacked-together bash BS to just despise the language.
- Nellyz 3y ago[flagged]
- collinvandyck76 3y agoI like this but the lazy part of me just treats anything i write into $(mktemp -d) as something that will be eventually GC'd by the operating system. I have no idea when it actually happens, or if it does at all, but that's how i roll.
- nikau 3y agoMore so now with containers
- jesse20 3y ago[dead]
- jesse20 3y ago[flagged]
- ranger207 3y agoRelated, but I use exit traps (or actually ERR traps) to make debugging bash scripts at runtime a little easier. This will print the number of the line of the script that failed along with any error messages from the line that failed. This is useful if for whatever reason your logging system or whatever doesn't capture stderr ``` failure() { local lineno="$1" local msg="$2" echo "Failed at ${lineno}: ${msg}" } trap 'failure "$LINENO" "BASH_COMMAND"' ERR ```
- LispSporks22 3y agoEmacs C-c C-t will insert that for ya in shell-script-mode
- parentheses 3y agoI'd like a way to do this for bash functions which I use quite extensively.
- Brenda900 3y ago[dead]
- worik 3y agoI have given up the unequal struggle to learn Bash. For me it is "read only". It is too arcane, even for me. I use Perl now. I tried to reform last year as I was building a system from lots of executable pieces, the perfect job for Bash After much pain and suffering I re-wrote it in Perl. What a (relative) breeze. Just. Don't. Do. Bash. Works for me!
- underdeserver 3y ago+1. Bash is riddled with absurd footguns. You want to set hack together three git commands and pipe to fzf? Fine. Anything more complex than that? Python, Perl, or any other proper programming language is there for you.
- abwizz 3y agoi'd argue that every language is somewhat imperfect, but there is something to be said in favor of pitfalls/footguns/inperfections that are somewhat well defined and understood for more than a decade.
- jdiff 3y agoA footgun doesn't stop being a footgun with time. New generations of programmers and shell scripters are constantly rolling through and shooting their toes off. Just because there was a "well, that sucks, but it's the least worst compromise" 4 decades ago doesn't make it a good reason today, doesn't make it intuitive today, and doesn't make it not still a horrible footgun today. A familiar footgun is still a footgun.
- ur-whale 3y ago> Bash is riddled with absurd footguns I agree, shell programming is ugly and very unsafe. > Anything more complex than that? Python, Perl, or any other proper programming language is there for you. I disagree. First, there are certain things, specifically when you want to process very large datasets that are easiest - by a very large margin - to build using shell scripts: a combination a ripgrep, sed, awk, grep, cut, tr, paste, jq, head, tail, etc ... is way easier and faster to put together in bash than anything else. Second, to get maximum flexibility you'd like to be able to switch from one (python, perl) to the other (shell script) transparently and both ways Do certain things that can be expressed cleanly in python, and then transparently switch to bash calling a horde of specialized shell commands when the task at hand is easier to express that way. Shell can transparently go down to Python or Perl very, very easily. Unfortunately, the converse is absolutely not true: while Perl can - to a certain extent - be used to construct complex pipelines of data processing external commands, it is nowhere near Shell in ease of use. And it is a giant PITA in Python which gives you fuck-all above exec/fork level subprocess manipulation: writing large external pipelines in Python is about as easy as it is in C.
- chrismarlow9 3y agoJust as an alternative suggestion, consider using a "down file" instead if you can and letting the code gracefully end. Plus you get to write "touch down" and do an end zone dance.
- tjoff 3y agoI think you need to be a little more specific about what the "down file" is for, especially since you can't for google it.
- chrismarlow9 3y agoApologies. It's for ending execution prematurely. Here's an example: while true; do echo hello test -f down && exit done Now to stop early you execute "touch down".
- 1letterunixname 3y agoMost people use Bash trap incorrectly, and it should be documented. # All normal and error exits trap 'e=$?; trap - EXIT; your cleanup here; exit $e' EXIT # Error only trap trap 'e=$?; trap - ERR; your error only cleanup here; exit $e' ERR Save the previous exit condition to preserve it, otherwise it will be destroyed. Untrapping is necessary to prevent multiple calls, especially if it can call exit or fail within the trap handler. You don't need an exhaustive list of signals, which is almost never correct in oft touted cargo culted examples.
- hawski 3y agoIt is sad that trap interface is not reversed: trap EXIT WHATEVER -- cmd args But as it is common with bash interfaces crystalized before good practices became apparent.
- 1letterunixname 3y agoStylistic bikeshedding. 3 keywords for normal, abnormal, and all exits would be semantically clear. The best practice for a problematic language is to use something else.
- xyproto 3y agoPoor man's defer