10 ms·
Why doesn't Bash’s ‘set -e’ do what I expected? (2021)
- gurjeet 4y agoAlso noteworthy, this post of mine with authoritative text from GNU docs. Read "(Un)Portable Shell Programming" https://news.ycombinator.com/item?id=31678176 https://news.ycombinator.com/item?id=31678176
- interroboink 4y agoOr, as Larry Wall put it, "It is easier to port a shell than a shell script."
- kazinator 4y agoThat was no idle musing; he had in mind a reversal of the situation. He put that into action by making a language so embroiled in its own implementation that cloning a new version would be a pointless exercise in failing to replicate the internal details of the original program. That nicely solved the problem of it being hard to port a script. Then he made it incredibly hard to build. Cross-compiling it wasn't even a thing, and probably still isn't; we just use distro builds that have Qemu.
- ohiovr 4y agoAll set -e does is halt further execution of your script if any line exits above 0. There is really nothing bash can do about this. It can't perform a psychological evaluation on why a program is not giving the expected output. And most of the time, when something fails, if it were made by a less than serious programmer (like myself) they don't bother to exit on error correctly with a code above zero. But if you are building a script collection, or python utility or even an binary, simply exit 1 if you encounter a general error. Then further up the chain in your shell scripts the error can be handled properly. There are more specific error codes you can use that might be helpful to your shell script. If you are wondering what in the world I'm talking about and I've missed your question entirely, sorry about that. My feet are tired.
- mdaniel 4y agoAlso, error messages to stderr, please, and bonus points for not trying to emit cute ansi color codes unconditionally
- hinkley 4y agoCI tools: telling you who the asshole is, since 2001.
- koala_man 4y ago> All set -e does is halt further execution of your script if any line exits above 0. This is the expectation, and the article is about why it's not that simple.
- stevebmark 4y agoThe `let ++` example is pretty damning. I find Bash to be a poor and outdated shell that we’re stuck with for now. Its design is the best argument against LSD I know, it came out of the hippie drug days, and Bash reflects that with its madness.
- somat 4y agoA lot of the sins of shell are from it's primary role as an interactive interface to the computer. it's use as a scripting language was a nice secondary goal. On the one hand it is nice to have your interactive interface and your scripts be the same language. on the other it is a bit horrifying the way all those convenient interactive features make your programs so prone to failure.
- justin_oaks 4y agoWe'd probably all be better off if our scripts weren't in the same language as used by our interactive interface. I do paste snippets of scripting (if, for, etc.) into my interactive terminal from time to time, but that's rare compared to just typing commands to run. I wouldn't mind it if all scripting in the shell had to be offloaded to a completely separate language that was less error prone and more consistent.
- kazinator 4y agoMainly, the reason why Bash's "set -e" doesn't do what you expected is this. set -e comes from the POSIX shell language, descended from the Bourne Shell. The examples you're using use obscure Bash features. For instance let turns an arithmetic result into a termination status, where 0 is fail (opposite to the POSIX convention and all). set -e works to the extent that your commands have a sane termination status, which they generally do if they are standard built-ins or well-behaved utilities. One Bash feature improves the effectiveness of set -e (or exit status testing in general). In a command pipe: a | b | .. | z the termination status is obtained from z. That's standard. If z indicates success, the pipeline is successful no matter how a through y terminate. Bash has a "pipefail" option to help with this.
- yread 4y agoTFA talks about pipefail and how it sometimes fails when command writes too much. Is read(1) an obscure feature now?
- strken 4y agoYou don't have to be hyper-aware of false positives with "set -euo pipefail". False positives bring themselves to your attention during testing, while false negatives don't announce themselves at all. It's almost always preferable for your code to incorrectly fail and force you to stick "|| true" on the end, than to incorrectly succeed and let you miss the bug.
- anderskaseorg 4y agoThere are false negatives, such as the one mentioned in the post after “A particularly dangerous pitfall with set -e is combining functions with conditionals.”
- jasonhansel 4y agoAnnoyingly, when a process is terminated by an unhandled signal (say, SIGTERM), it is treated as if it exited with a nonzero exit code. This can make it tricky to use non-builtin commands as conditions in "if" statements, since there's always the potential edge case where the "if" block is skipped because of a signal that the condition received.
- pcthrowaway 4y agoI'm not sure what this person did expect. That bash magically parsed the output and memory of running programs, and read the user's thoughts, to determine if the state of the program indicates a condition that the user would consider an error? set -e does exactly what you'd expect, arguably, with the exception of subshells and conditions. And those rules are extremely simple to learn too. If you understand when a statement which might be composed of other statements would have an error, you can predict what set -e will do
- wodenokoto 4y agoDid you look at the examples? The first examples has me completely puzzled. #!/usr/bin/env bash set -e i=0 let i++ echo "i is $i"
- thayne 4y agoWould you find it less surprising if the let command behaved differently than most other commands, and didn't stop the program if it returned a non-zero exit code? Because, this example works exactly the way I would expect. The next one (using the double parentheses syntax) is more questionable, and I could see it going either way, and would either look it up in the manual, or put an `|| true` after it to be safe. The fact that it changed in a minor release is pretty bad though.
- wodenokoto 4y agoCome again, why is `i++` obviously an error when `i` is zero? I’d expect an arithmetic operation to fail if I, say, divide by 0. Not if I add 1 to zero! And just because behavior is documented doesn’t mean it’s not surprising or complicated.
- powersnail 4y agoNot your parent commenter. I'd expect let command not to return a numeric value at all, since bash uses the returned numeric value to indicate success/failure. But that admittedly doesn't have anything to do with the design of "set -e".
- deleted 4y ago[deleted]
- TheAdamist 4y agoIf you modify your PS1 to include $? It makes writing shell scripts that rely on exit codes a lot easier. It should be the default in my opinion. Build scripts that fail and return 0 are my nemesis.
- yjftsjthsd-h 4y ago> Build scripts that fail and return 0 are my nemesis. I was also once party to an argument caused by a developer who thought that non-zero exit codes were an acceptable way to communicate success states. I suspect that he may have been trying to communicate different success conditions, but I couldn't tell you for certain.
- zgs 4y agoBecause you don't understand what it is meant to do.
- ziotom78 4y agoIn the 2000s, I was running extensive sets of simulations and data reduction scripts for a scientific experiment, and I was heavy relying on scripts to run the programs, collect the results, and distribute them over several servers. At first I developed those scripts using bash, but I needed to do math and complex iterations over file names and different parameter files, and I continuously stumbled upon weird behaviors and had to rely to hard-to-understand quirks like the ones explained in the article (which bit me more than once!). After a while I stumbled upon scsh [1], which at first didn't impress me because I ran it as an interactive shell, and from this point of view it was really ugly. But then I realized that scsh was primarily meant as a way to run shell scripts, and I immediately felt in love. I had the power of a Scheme interpreter, the ability to easily use mathematical expressions (the awesomeness of Scheme's numerical tower!) and macros, and a very effective way to redirect inputs and outputs and pipe commands that was embedded in the language [2]! In those years I used scsh a lot and developed quite complex scripts using it, it was really a godsend. Unfortunately the program got abandoned around 2006 because it was not trivial to add support for 64-bit architectures. However, while writing this post I've just discovered that somebody has revived the project and enabled 64-bit compilation [3]. I would love to see a revamp! Nowadays I use Python for complex scripts, but it's not the same as a language with native support for redirection and pipes! [1] https://scsh.net/ https://scsh.net/ [2] https://scsh.net/docu/html/man-Z-H-3.html#node_chap_2 https://scsh.net/docu/html/man-Z-H-3.html#node_chap_2 [3] https://github.com/scheme/scsh https://github.com/scheme/scsh
- adrian_b 4y agoI had a very similar experience with using scsh for shell scripts instead of bash and it was great and far more convenient than using Python or any other language with poor support for the shell style of command execution. I have also been forced to revert to bash later so it is good to know that there might be an update for it.
- olau 4y ago> Nowadays I use Python for complex scripts, but it's not the same as a language with native support for redirection and pipes! While this is no doubt right, I think you could build a little abstraction that simplified piping syntax considerable. The subprocess module is not terribly convenient, although it has gotten less confusing over the years. Python does have operator overloading, so you could even make it fancy.
- chubot 4y agoAll of these problems are fixed in OSH. It runs your bash scripts but you can also opt into correct error handling. The simple invariant is that it doesn't lose an exit code, and non-zero is fatal by default. See https://www.oilshell.org/release/latest/doc/error-handling.html https://www.oilshell.org/release/latest/doc/error-handling.h... Oil 0.10.0 - Can Unix Shell Error Handling Be Fixed Once and For All? https://www.oilshell.org/blog/2022/05/release-0.10.0.html https://www.oilshell.org/blog/2022/05/release-0.10.0.html This took quite awhile, and I became aware of more error handling gotchas than I knew about when starting the project: e.g. it's impossible in bash to even see the error status of a process sub like diff <(sort left.txt) <(sort OOPS) If you have bash scripts that you don't want to rewrite, try 1) run them with OSH 2) Add shopt --set oil:upgrade at the top to get the error handling fixes. Tell me what happens :) https://github.com/oilshell/oil https://github.com/oilshell/oil I spent a long time on that, but the post didn't get read much. I think it's because it takes a lot of bash experience to even understand what the problem is. (rehash of https://news.ycombinator.com/item?id=33075915 https://news.ycombinator.com/item?id=33075915 which came up the other day :) )
- maegul 4y agoNever used OilShell (OSH) before, but this opening line on their home page struck me: > … and it's a new language for Python and JavaScript users who avoid shell! Why not work towards using python/JavaScript as shell languages? There was a HN thread a few days ago on Xonsh, the main python attempt at this which looks nice to me, which naturally got a lot of anti-anti-bash energy. But if we want better shell ergonomics and are willing to ditch bash, why not take popular versatile and capable languages and get them shell-appropriate? It seems like a natural use case for these languages, and even the place where they could retire to peacefully in the future. I suppose OSH is aiming at bash compatibility, but how good or viable is that really (not a rhetorical question)? Seems to me like an all or nothing thing as someone who recently ran into some bash-zsh incompatibility. ~~~ HN thread on Xonsh: https://news.ycombinator.com/item?id=33044772 https://news.ycombinator.com/item?id=33044772 Xonsh homepage: https://xon.sh/ https://xon.sh/
- Dylan16807 4y ago
- NackerHughes 4y agoShort answer: because it's Bash. No syntax or language construct will do what you expected.
- deleted 4y ago[deleted]
- realharo 4y agoSometimes it also depends on the specific version, like when you need to deal with empty arrays - https://stackoverflow.com/questions/7577052/bash-empty-array-expansion-with-set-u/61551944 https://stackoverflow.com/questions/7577052/bash-empty-array... The answers there are just a few of the many great examples of why bash is best avoided in almost all cases.
- lopatin 4y agoI tend to think of Bash as a syntax-free language. There are no clear rules, you just have to kind of make it work every time. It's like the programming version of a freestyle rap or Parkour.
- shp0ngle 4y agoThe article is great, but I wholly disagree with the conclusion “don’t use -e”. It’s still much better to use it than not to use it.
- the_duke 4y agoI use "set -Eeuo pipefail" in pretty much all bash scripts. (and often -x as well) Makes things much saner. This post is a better introduction than the submission: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/ https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_p...
- hebrox 4y agoThanks! I'm not doing a lot of bash scripting, so there's always something weird happening. `set -Eeuxo pipefail` will be my new default.
- _dain_ 4y agoIt's still not enough though. https://www.oilshell.org/blog/2022/05/release-0.10.0.html https://www.oilshell.org/blog/2022/05/release-0.10.0.html
- trashtester 4y agoIn your bash shell, type: set -e let 1 let 0
- ilyash 4y agoOne of the reasons I created Next Generation Shell. It has exceptions. So "if $(grep ...)" works correctly as opposed to bash. grep exit codes: 0 - found, 1 - not found, 2 - error. bash can not handle this correctly in if. There are just two branches for 3 exit codes. NGS has two branches and exception that can be thrown. Yep, every single "if grep ..." in bash is a bomb.
- junon 4y agoset -euo pipefail This is how all of my shell scripts start.
- knorker 4y agoThese options are addressed in the post, and they are very far from fixing the problem. Yes, they are better than nothing. But as explained in the article it's still like wandering blindfolded into a minefield.
- woudsma 4y agoI can highly recommend using Shellcheck [0] when writing Bash, it also has extensions for VS Code and other IDE's. It makes writing Bash much easier. [0] https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- jakewins 4y agoAgree, this is the way.
- chubot 4y agoShellCheck is great, but since it works on syntax, it can't catch problems that can only be determined at runtime, like "if myfunc" with set -e on. Example here: https://news.ycombinator.com/item?id=33122256 https://news.ycombinator.com/item?id=33122256
- placatedmayhem 4y agoI moved over to shellharden a while ago. It can actually apply the suggestions it makes. Aside from that, my employer is somewhat disapproving for GPLv3 tools, but MPL that shellharden uses is essentially auto-approved. https://github.com/anordal/shellharden https://github.com/anordal/shellharden
- scbrg 4y ago> my employer is somewhat disapproving for GPLv3 tools Let's hope it's not bash scripts you're writing :-) http://git.savannah.gnu.org/cgit/bash.git/tree/COPYING http://git.savannah.gnu.org/cgit/bash.git/tree/COPYING
- placatedmayhem 4y agoYeah, other shell flavors are fine.
- franga2000 4y agoHow is GPLv3 any different from MPL when it comes to use (as opposed to modification or distribution)?
- pizza234 4y agoShell scripts can be thought like C++. They can be sanely managed only if one adopts a strict subset of functionalities. The page is probably oriented at situations where one needs to support any version of bash and any obscure/inadvisable functionality (mind that there are still devs that mix sh and bash syntax in the same script), which is very inconvenient and error prone, so manual error handling may make sense. While there is always something insane behind the corner in Bash, with a restricted subdomain (e.g. bash 4.2+, strict shell options, and shellcheck) it's possible to progressively write reasonably solid shell scripts. The document conclusion is somewhat biased. "to handle errors properly" implies that `-e` in inherently unreliable, which not fair - strict shell options do remove certain classes of errors, which doesn't hurt.
- thomashabets2 4y agoI disagree. Bash basically cannot be used correctly. It's like gets() in C; a knife with a blade but no handle. Unsafe at any speed. https://blog.habets.se/2021/06/The-uselessness-of-bash.html https://blog.habets.se/2021/06/The-uselessness-of-bash.html C++ can be used correctly. It's easier since C++11, with smart pointers and stuff. I'm not going to argue that writing solid C++ is easy, but it IS possible. Bash basically can't be used correctly. Even a simple command pipeline of three commands, with actual error handling, is probably more code in bash than in any other language.
- pizza234 4y ago> I disagree. Bash basically cannot be used correctly. It's like gets() in C; a knife with a blade but no handle. Unsafe at any speed. > C++ can be used correctly. It's easier since C++11, with smart pointers and stuff. I'm not going to argue that writing solid C++ is easy, but it IS possible. It seems this conflates two different aspects of programming languages - complexity and (memory) safety. The analogy with Bash was in relation to the language complexity. By reducing the language features to a restricted, well-known (by the given team) scope, it's possible to use the language effectively. A very good example of this is the Doom 3 source code. In relation to memory safety, Google hasn't been able to develop a safe enough Bluetooth stack, which has been a dumpster fire. Due to the small support windows of Android, as of today, a huge amount of devices with older Android versions are either in the garbage, or are extremely unsafe. If Google, with their engineering practices, can't develop a safe Bluetooth stack in an unsafe language, realistically, nobody can. > https://blog.habets.se/2021/06/The-uselessness-of-bash.html https://blog.habets.se/2021/06/The-uselessness-of-bash.html The topic in the post is very interesting (pipes error handling are no doubt a rough area of Bash), but the post takes the choice of being a rant rather than an informative research. The central thesis that "functionality of language X is broken in a 1-line program, but it works in a 83-lines program in language Y" doesn't have any meaningful conclusions. It would have been interesting and informative to compare against the correct version in language X, which in the case of Bash, it does exist (but in this case, it's advanced Bash, and sure, it's ugly). There is a very wide gap been 1-, 10-, 100- and 1000- line programs, and very different requirements. Asserting that only the 1-line case is the only valid use case for shell scripting, it's inefficient system administration (assuming disciplined shell scripting).
- cassepipe 4y agoI am using zsh but only because it comes with a little more out the box and because zle's excellent vi mode (supports text objects for the win) But I am wondering, does zsh fare better when it comes to writing more correct scripts or is it plagued with the same issues ?
- planede 4y agothe `read -r foo < configfile` might be obscure, but every line in a POSIX text file ought to have a newline terminator. If this is worth erroring out on probably depends on context.
- thomashabets2 4y agoBecause bash error handling is a thousand blades and no handle. https://blog.habets.se/2021/06/The-uselessness-of-bash.html https://blog.habets.se/2021/06/The-uselessness-of-bash.html I've reviewed a lot of code, bash and otherwise. I have never, not once, reviewed bash code that didn't have subtle bugs. And this is code written by smart people.
- xani_ 4y ago> Because bash is thousand blades and no handle. FTFY. It really is absolutely terrible tool as programming language. If your script is more than "a bunch of pipes in a trench coat" just use something else. I'd unironically prefer Perl to it.
- thomashabets2 4y agoThe final straw for me was when I (described in my blog post) realized that not even "foo | bar" does the right thing.
- lupire 4y agoOf course it's not ironic. Perl's raison d'entre is to replace bash+grep+sed+awk
- justin_oaks 4y agoIs there any good alternative to bash for stringing together multiple commands with pipes and redirections? "Real" programming languages usually have painful-to-use APIs for calling commmands, redirecting output, piping one command to another, etc.
- agumonkey 4y agoWhat's the best and smallest choice to replace admin/ops bash scripts ? Python ? Python with some lib ? Ruby ? Lua ? Rust ?
- favadi 4y agoIMO, Perl is perfect for this purpose. A good thing about Perl is, it doesn't really changed any more. So no need to spend time keeping up with latest developments like Ruby or Rust.
- lloeki 4y ago> it doesn't really changed any more it does change, it just has outstanding backward compatibility and new feature (including parser-level) opt-in per module.
- drothlis 4y agoA lot of commenters suggesting "pipefail" aren't realising the full extent of the problem. Here's an example that might be clearer, from https://david.rothlis.net/shell-set-e https://david.rothlis.net/shell-set-e This prints “a”, as you’d expect: set -e myfun() { printf a; false; printf b; } myfun printf c ...because the “set -e” terminates the whole script immediately after running the “false” command. But this script prints “a-b-True”, where some people (myself included) might have expected it to print “a-False”: set -e myfun() { printf a-; false; printf b-; } if myfun; then printf True else printf False fi The “set -e” is ignored completely. Putting another “set -e” inside the definition of myfun doesn’t make any difference.
- deleted 4y ago[deleted]
- jlokier 4y agoIt's always been odd that Bash has hundreds of options for controlling its behaviour in non-standard ways, via "set", "shopt" and environment variables, but it's never had an option to make "-e" do the obviously natural behaviour,, I encountered the "you can't enable -e" problem years ago in a complex build system that used Bash to build embedded sysem components. The scripts were all designed on the assumption that "set -eu -o pipefail" would cause early exit on any failing commands as if doing the equivalent of "exit $?" or "return $?", and of course it is not reliable, as your example shows. The very surprising behaviour of "set -e" inside functions was not noticed for a long time while the system was being written and used. When it was noticed, "set -e" was added inside the functions because that seemed to work for the Bash version at the time, then later it stopped working.
- chubot 4y agoit's never had an option to make "-e" do the obviously natural behaviour,, This is harder than it sounds! :) There's arguably not a natural behavior for shell, because $? is overloaded in Unix. It can mean success/failure or it can mean true/false/error. https://www.oilshell.org/release/latest/doc/error-handling.html#what-does-mean https://www.oilshell.org/release/latest/doc/error-handling.h... Oil introduces some new idioms that led you distinguish between the two paradigms, so it's best to change the way you write scripts. It's not something bash can magically fix. https://www.oilshell.org/release/latest/doc/idioms.html#error-handling https://www.oilshell.org/release/latest/doc/idioms.html#erro... That said, just running your scripts with Oil will flag problems, acting like a ShellCheck at runtime, catching things that ShellCheck can't. See the sibling reply for a demo, and this comment: https://news.ycombinator.com/item?id=33117337 https://news.ycombinator.com/item?id=33117337
- oftheoaks 4y agoThis is really interesting. The case pointed out in Ex.3 is pretty hilarious: (from the bash manual) The ERR trap [same rules as set -e] is not executed if the failed command is part of the command list immediately following a while or until keyword, part of the test in an if statement, part of a command executed in a && or || list except the command following the final && or || [...]
- wkdneidbwf 4y agomountain out of a molehill imo. there’s a certain point where you’ll be fighting shell more than it’s helping—choosing another language is the better choice. i’m not a wizard but i feel like i’ve developed a decent intuition for what’s sane to do in shell and what needs something else. and even then, shelling out from another language when absolutely necessary can be a better option
- teo_zero 4y agoFew people understand that if...then in bash is actually a form of try...catch. The -e terminates the script at every failure not caught by a try...catch. With this in mind, it's easier to predict bash's behavior.
- Maursault 4y agoIf anyone reading these threads is over 70yo, please reply to this comment with the correct explanation.