8 ms·
Common shell script mistakes (2008)
- anotherhue 5y agoThis advice is largely automated by https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- blueflow 5y agoIt felt nice when i discovered shellcheck. I made the mistake of trying to get my scripts 100% compliant... and was then astounded to find i broke several of them. If you don't know what you are doing, and shellcheck knows more, it might be useful. But if you know what you are doing, shellcheck becomes annoying quickly. Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions.
- anotherhue 5y agoVery true, and I experienced something similar just yesterday. I've begun piping such untrustworthy commands to xargs -n1 to guarantee good behavior.
- account42 5y ago> Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions. It should still be `printf '%s\n' "$(command)"` for portability unless the intention is for the output of your command to possible be interpreted as a flag for echo.
- q3k 5y agoAmazing example of why robust shell scripts are hard.
- blueflow 5y agoYou got me there! There are indeed values that start with a hyphen, somehow it didn't break previously.
- kergonath 5y ago> It should still be `printf '%s\n' "$(command)"` for portability Damn that’s ugly though. Not much better than Perl in terms of characters soup. And Perl has a massive advantage with its deep integration of regexes.
- hnlmorg 5y agoPerl's character soup is because it's an evolution of sed and awk so Perl adopted many of the UNIX shell idioms.
- hnlmorg 5y agoOne of my pet peeves with command line tools is where they take a list of values but also support flags. You then have no end of fun working with hyphen-prefixed file names and no end of hard to find bugs too (at least most shell script writers are aware that spaces in file names are a pain but these kinds of bugs are a lot more dangerous and easier to forget). I can forgive a lot of design flaws because the tech was new and people didn't know any better but I always struggle to apply that same reasoning with the design of having commands that support both free text and flags that change that command's operation. I'm surprised nobody at the time paused and said "this might be a problem" when they were writing their tools and only using the hyphen to distinguish between a flag or data. Maybe I'm being a little unfair though -- it is hard to put yourself in their shoes and unlearn 50 years of tech. My biggest peeve about it though is that I can write a shell that doesn't apply to POSIX and which fixes a lot of shortcomings of shell scripting. But I cannot fix this specific class of bugs without rewriting the entirety of coreutils.
- deleted 5y ago[deleted]
- dtgriscom 5y agoOh, yeah. I've broken multiple historic scripts in obscure ways "fixing" them under shellcheck's direction. It's a general lesson for linters: if it points out code where the results may be surprising, then be aware that the "surprising" result may be necessary for the code's function.
- arglebargle123 5y agoShellcheck is still incredibly handy to re-check your work for simple mistakes, but man do I ever end up using a lot of shellcheck ignore statements.
- neves 5y agoI'm a big fan of shellcheck, but use it for your new scripts. Don't try to change working code that depends on shell idiosyncrasies.
- gkfasdfasdf 5y agoAs well, having a bash language server enabled (https://github.com/bash-lsp/bash-language-server https://github.com/bash-lsp/bash-language-server) helps immensely.
- gorgoiler 5y agoAdvice at a higher level: don’t try too hard to handle error cases in your scripts. The best shell scripts are a list of imperative instructions to get something done. If an error occurs, bail as early as possible and with a useful message. Don’t try any harder than that. For example: if ! complex_function then handle_error fi …doesn’t work as you’d expect because the “if” context changes the rules of set -e in complex_function. It’s much better to have complex_function crash your script with a helpful error message that the operator can ameliorate. You may have a clever way of solving this problem, but the cleverer your solution gets, the further you diverge from the core language of shell scripting. You will suffer the lisp/Ruby problem of every script being in its own unique language that’s based on but not identical to the language with which your colleagues are familiar. One shouldn't dismiss shell scripts completely – the command line is the primary interface to Unix and with more and more idempotent commands showing up every year ("ip route replace") it only gets easier to write simple, imperative lists of commands.
- neves 5y agoI agree. You must fail fast. My best shell advice is to put these commands at the beginning of every script: set -e # stop scripts on errors set -u # stop script if undefined var set -o pipefail # stop script if there's a pipe failure Use them to have a saner life.
- misnome 5y agoAnd you can compact them to set -euo pipefail
- l0b0 5y agoMy favourite set of pragmas: set -o errexit -o noclobber -o nounset -o pipefail shopt -s failglob inherit_errexit These are all easy to find in the Bash manual.
- creamytaco 5y agoFor most uses, I consider all of these unwanted at best and potential disasters at worst. If you use them, make sure you understand exactly what each of these do. If you just follow a dictum you found on HN blindly and just copy-paste them, you're in for a world of hurt. Example for set -e: https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105
- rob74 5y agoThis may be controversial, but sometimes I feel the biggest mistake is using a shell script in the first place. For example, on servers where you have PHP installed anyway, a PHP CLI script is often an alternative. Say what you will about PHP, but PHP code is much more readable than shell scripts - especially if everyone working on the server is fluent in PHP anyway. Shell scripts do have their niche, but often you find that the scope of a script keeps growing, then soon it becomes cumbersome and makes you wish you would have chosen another alternative from the start...
- frankwiles 5y agoAgreed! If it’s more than a few lines I switch to something else usually Python these days. Couple decades ago even wrote some init scripts in Perl because they needed to be complicated.
- MonaroVXR 5y agoWhy not use Ansible?
- hnlmorg 5y agoAnsible didn't exist 20 years ago.
- BossingAround 5y agoThe question is "why not switch to Ansible instead of Python" rather than "why didn't OP use Ansible 20 years ago instead of Perl". Seeing that Ansible is basically a set of Python libraries that enable users to write declarative (and hopefully idempotent) scripts, it isn't a stupid question.
- sumtechguy 5y agohmm, I can take a stab at that. I have extensively used both. Ansible is pretty good for 'get something or set of machines into a particular state'. Python is pretty good for 'run these bits and give a nice error if something goes weird'. The 'error path' is where ansible falls down for me. It can generate some very strange ones. Also its yml syntax can sometimes create odd issues with people who have not used it before. Ansible is also terrible for any sort of data processing. It is very good at iterate over a list and do this action to those items though. I recently did an audit script using ansible. Because I thought hey I have this cool tool to do it. It was a mistake. It probably would have taken 1/3rd the amount of time to do in python. Also if someone (probably me) has to extend that script getting the yml right will be tricky. I also recently used ansible to write a bunch of upgrade scripts. It was a breeze and would have taken much longer to do in python. In my case many times you have to write to the lowest common denominator. So bash it is, with some sort of central ansible setup controller.
- maccard 5y agoThe minute I need anything other than 4-5 commands executed sequentially, I switch to python. It's not worth dealing with the hassle of string substitution, error handling, list handling, filtering, etc in bash.
- dspillett 5y agoA very common one I've hit once or twice is people assuming bash is the system shell (which is common with Linux, but far from ubiquitous as Debian for example uses dash) and using bashisms while leaving the hash-bang as #!/bin/sh. I always go with #!/bin/bash as I do often use bashisms, and only #!/bin/sh when I know I've been careful (because at the start I know I'm writing something intended to be as portable as practical, or I've gone through with a fine-tooth comb to test compatibility at a later time).
- mdaniel 5y agoOr its superior(?) friend "#!/usr/bin/env bash" which will select the bash that the user has on their PATH, which is especially considerate to Homebrew/Linuxbrew folks who have a modern version in $HOMEBREW_PREFIX/bin I believe there are some esoteric systems that relocate "env" as "/bin/env" or such, but the good ole "90/10" rule applies here
- n8henrie 5y agoI used to use `#!/usr/bin/env bash` for this reason (Linux / MacOS + Homebrew user) but have been bitten a few times by bash version differences when I write a script that is later called by a system utility like launchd (which doesn't inherit my PATH) that goes with /bin/bash instead. Because the backwards compatibility tends to be good, if I'm writing a script that will be run non-interactively, I will usually write a fixed `#!/bin/bash` shebang just so I can be sure that it run with the expected bash version on Linux or Mac.
- strzibny 5y agoNice tips, although doesn't look comprehensive. It's certainly hard learning all the gotchas of shell scripting. In Deployment from Scratch I teach minimum amount of Bash to get servers up and running. I avoid teaching anything I don't have to. For example, most people are fine with just "set -euo pipefail" and understanding simple functions and pipes. If your script is small enough, you will be fine. If your script is getting big, it might be a time to switch to smth else.
- mikotodomo 5y agoJust so you guys know, this website does not have HTTPS. Be aware.
- ilyash 5y agoComparing shamelessly to my own Next Generation Shell, puting aside the completely insane syntax matters that nobody today would do or accept (such as $x expanding to zero or more arguments). cleaning up temp files Use TmpFile. It is automatically removed when the script exits. stopping automatically on error As in many other modern languages, NGS has exceptions and prints stack trace when they are not handled. Nothing special is required from the programmer. echoing errors Use built in error("my message") function. Alternatively, use exit("my message") to print the error and exit, the exit code defaults to 1 but can provided. Tired of seeing bash scripts starting with definition of warn(), debug(), error(), etc functions. These are part of standard library in NGS. You are welcome to check out the Next Generation Shell project. https://github.com/ngs-lang/ngs https://github.com/ngs-lang/ngs
- deleted 5y ago[deleted]