8 ms·
Shellcheck finds bugs in your shell scripts
- RMPR 3y agoEverytime I see Shellcheck coming up, I have to mention shellharden[0] written by a colleague of mine. It is basically shellcheck but it applies the suggested changes automatically. 0: https://github.com/anordal/shellharden https://github.com/anordal/shellharden
- DamonHD 3y agoNice: I have learnt some things from this on the very first production /bin/sh script that I pointed it at, and I've been hacking such scripts since the 80s!
- Dries007 3y agoShameless self-plug: I very much live by the "fix all warnings before you commit" (or at least before you merge), so I have Shellcheck and a bunch of other linters set up in my pre-commit configurations. But the majority of the shell in most of my projects ends up embedded in .gitlab-ci.yml files, where it's hard to check. So I made a wrapper that does that automatically: https://pypi.org/project/glscpc/ https://pypi.org/project/glscpc/. It uses the Shellcheck project and some magic to give you the Shellcheck remarks with (mostly) accurate line-numbers.
- brirec 3y agoI’d love to see a project that would do this, but more generally. I don’t use GitLab CI, but I do use a good handful of other file types that essentially inline shell scripts. Dockerfiles, GitHub Actions, and Justfiles, just to name a couple. Usually, and almost exclusively for the sake of ShellCheck, I make a point of putting anything more complex than a couple of commands into their own shell scripts that I call from the inlined script in my Dockerfile. (This pattern also helps me keep my CI from being too locked into GitHub Actions)
- stouset 3y agoThis is the way
- iamjackg 3y agoAbsolutely agree. The main downside of that pattern is that it doesn't work with jobs included from other projects in GitLab CI, since the job runs in the context of the project that imported it and therefore can't find the script in its original repo. Huge bummer.
- vinnymac 3y agoBundle up configuration and scripts, and now we have containerized CI infrastructure.
- vinnymac 3y agoI’d rather see CI services add a mode that would enforce all scripts must live in separate files, rather than inline. It’s really not necessary to support inline except for single lines that are very short (under 30 chars).
- Piraty 3y agoso much this. it will help CI not become a holy, super-hard-to-debug, unreproducible mammoth. write scripts, call them from CI
- Dries007 3y agoI would say: stick to straight command sequences with variables, avoid if/for/while/functions etc. Subshells and pipes only for trivial things. Setting something like a 30 char limit will inevitably lead to code-golfing to fit a mess into the 30 char limit. But that becomes harder to automatically check, that's why you should still have good peer reviews based on written standards somewhere.
- plorkyeran 3y agoI am not generally a fan of Xcode Cloud's design, but this is one thing I think it gets very right. Rather than let you specify the build actions in the CI settings, it invokes scripts with fixed names in the `ci_scripts` directory of your repo if they exist. There just isn't a mechanism for creating jobs which aren't something that you can build locally and test independently of Xcode Cloud.
- iamjackg 3y agoOh, that's really cool. I was trying to solve this from a different perspective a while ago: I wanted to add some pre-processing that would take a "normal" shell script and render it to the `script` part of the corresponding job at build time, the advantage being that you still have everything self-contained in the gitlab-ci job. I stopped working on it because dealing with shell shenanigans in the GitLab CI runner environment is such a miserable experience that we're in the process of moving all our jobs to python scripts.
- Dries007 3y agoYea, Pythonifying the scripts is also generally my preferences the moment they become somewhat complex. But even then it's nice that you can be reasonably sure you're not forgetting quotes around variables or using bash constructs where you only have sh.
- deleted 3y ago[deleted]
- l0b0 3y agoHigh five - I did something similar very recently[1]; usage[2]: - repo: https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook rev: 174fda8f384db229aca5b40380e713d0c25a1cb9 # frozen: v1 hooks: - id: shellcheck-gitlab-ci-scripts files: ^\.gitlab-ci\.yml$ [1] https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook [2] https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5dd03d756eeedf801fc0/.pre-commit-config.yaml#L104-108 https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5...
- seb1204 3y agoThe page is thanking Mercedes Benz? That came unexpected.
- popcalc 3y agohttps://github.com/orgs/mercedes-benz/sponsoring https://github.com/orgs/mercedes-benz/sponsoring They're sponsoring quite a few devs. Caddy, curl, and SeaweedFS notably.
- wittekm 3y agoShellcheck is great, but dealing with source/imports is suuuch a pain. Not their fault sh is a nightmare.
- johnchristopher 3y agoWell, it's possible to do this: # shellcheck source=./deployment/deployment-example.env . "${1}" But I see how it's a pain point when you have multiple subshell scripts and files to source.
- ulrischa 3y agoSaves a lot of time for tons of legacy shell scripts
- hapulala89 3y agoI have a colleague that writes alot of shellscripts and there is an ongoing discussion if shellcripts or scripting languages like python is better.
- agumonkey 3y agoi remember trying to rewrite so fragile scraping bash script in python, and even with some effort to use nice libs and create some cool helpers it ended up as long and not much more solid bash is infectious in the bad sense :D
- pram 3y agoIt’s perfectly fine for glue type stuff in a CI pipeline imo. There’s frankly no easier way to work with files.
- sneed_chucker 3y agoPython's subprocess/shell-out story is just bad enough that I still find myself writing a lot of shell scripts if the task at hand warrants more than 2-3 subprocess calls. Realistically, Perl or Ruby would fill this role fine, but I hate adding another language to a project just for that purpose.
- pletnes 3y agoAgree, and also you have to write a few lines of shell to get your python going (and same for node or ruby or whatever).
- ovex 3y agoFrom a security point of view, Python is better because it is less of a footgun. So if you expose an interface to untrusted users, you should use Python because its behavior is more intuitive. An arithmetic expansion or missing quotes do not easily become a vulnerability in Python.
- eddtries 3y agoI also recommend https://github.com/bach-sh/bach https://github.com/bach-sh/bach when you have to use Bash for things long enough it probably shouldn't be!
- frizlab 3y agoBut not zsh scripts, sadly
- cglong 3y agozsh was originally supported, but unceremoniously removed: https://github.com/koalaman/shellcheck/issues/298 https://github.com/koalaman/shellcheck/issues/298 I've had great experiences with this tool, but, for some reason, this issue always makes me question taking too great a dependency on it.
- rascul 3y agoThere is also a bash language server. https://github.com/bash-lsp/bash-language-server/ https://github.com/bash-lsp/bash-language-server/
- lolc 3y agoMy take is that bugs in sh scripts are best avoided by not programming in sh. So my preferred tool for sh linting is git-rm. It's not always possible, but driving down the sh line count sure helps against bugs from this language and its weird expansion rules. Most people don't even know the language has expansion rules and write stuff that accidentally works after the fourth try. This lang wants to become obsolete.
- dimitar 3y agoI agree, but some bash can be unavoidable. I've found that even trivial looking bash can be helped with shell-check; this is a testament to the issues in bash more than anything.
- jzwinck 3y agoWhat bash is unavoidable? The aliases and functions you define in your personal shell, sure. But what else?
- auselen 3y agoPiping stuff, job control?
- dimitar 3y agoSome examples of possible exceptions: * Installing or starting your general-purpose runtime, passing the right configuration options * Project glue where introducing general purpose language makes things more complicated because of tooling or startup. * Simple cloud-init scripts * A command line tool is simply the best library for the job and you don't want to get a general purpose language just to immediately shell out.
- jjgreen 3y agoEasy to avoid, use POSIX sh. It is a truth universally acknowledged that 95% of bash scripts don't use any non-Bourne features (or can trivially be modified to be so).
- goombacloud 3y agoTo spot more common problems I recommend: alias shellcheck='shellcheck -o all -e SC2292 -e SC2250'
- throw0101a 3y agoSC2292: Prefer [[ ]] over [ ] for tests in Bash/Ksh. * https://www.shellcheck.net/wiki/SC2292 https://www.shellcheck.net/wiki/SC2292 SC2250: Prefer putting braces around variable references (${my_var}) even when not strictly required. * https://www.shellcheck.net/wiki/SC2250 https://www.shellcheck.net/wiki/SC2250
- mgdlbp 3y agoOpportunistic interjection that unnecessary ${} is the most bothersome style choice in any language I know of: - It obscures actual uses of modifiers, particularly ${foo-} when set -u is in effect, - It's obvious when a name runs into subsequent text, even if one has somehow avoided syntax highlighting, - And expansions followed by identifier chars don't actually occur in practice. Cases where the quotes cannot be moved to surround the variable are often interpolation of an argument to echo, whose behaviour is such a mess not even portable between bash and dash that shellcheck ought to be demanding printf at all times instead!
- arp242 3y agoRelated pet peeve: always writing variables as $UPPER_CASE in shell scripts. Useful: $UPPER_CASE for exported variables ("super globals"), $lower_case for anything else. Can also use $lower_case for function locals and $UPPER_CASE for exported and script global variables (stylistic preference; both are reasonable). Not useful or reasonable: $ALWAYS_UPPER_CASE_NO_MATTER_WHAT. I suppose people started doing it because they saw $EXPORTED_VARIABLE and thought "oh, I need to always upper case it", not realizing what that meant. And then after that more copy-copy of this "style".
- ovex 3y agoRecently, I found a privilege escalation vulnerability in a shell script as a result of arithmetic expansion (similar to the one described at https://research.nccgroup.com/2020/05/12/shell-arithmetic-expansion-and-evaluation-abuse/ https://research.nccgroup.com/2020/05/12/shell-arithmetic-ex...). For example, $((1 + ENV_VAR)) allows you to inject code if you can control $ENV_VAR. Unfortunately, shellcheck did not catch that. At least not with the default settings. But if you are implementing anything remotely security-critical, you should not be using shell anyway.
- reidjs 3y agoWhat should we use for more security?
- l0b0 3y agoBasically anything where it's difficult to treat variable values as code. Python, Ruby, Java, and even PHP are much better at this.
- LelouBil 3y agoI have flashbacks of when my PHP teacher showed us how to turn query parameters into their own variables by using PHP's dynamic variables feature. He waited a bit, and promptly said to never do that and started to explain the security risks.
- stevekemp 3y agoPerl is probably worthy of a mention there, with it's taint-mode you're explicitly forced to test externally-influenced variables before using them.
- zimmund 3y agoI've seen `eval()` in production code of several applications. The biggest vulnerability is more often than not the programmer :)
- 3y ago
- mmsc 3y agoShellcheck is great. Unfortunately, its checks pale at the idiosyncrasies of per-version bashism. For example: set u ignored_users=() for i in "${ignored_users[@]}"; do echo "$i" done passes shellcheck's checks, however bash <= 4.3 will crash with "bash: ignored_users[@]: unbound variable". Therefore, set -u isn't available to use in this (valid) use-case. Shellcheck also doesn't catch the expansion of variables as key names in testing assoc arrays: declare -A my_array un='$anything' [[ -v my_array["$un"] ]] && return 1 will will fail as "my_array: bad array subscript" because "$un" gets expanded to "$anything", which on a second pass, gets expanded to "", making the check [[ -v my_array[] ]]. Even worse, a value of un='$(huh)' actually gets executed: [[ -v my_array["$un"] ]] && return 1 -bash: huh: command not found Here's another one: in versions older than 4.3 (maybe?) these -v checks don't even work: $ declare -A my_array $ my_array["key"]=1 $ [[ -v 'my_array["key"]' ]] && echo exists $ [[ -v my_array["key"] ]] && echo exists $ [[ -v $my_array["key"] ]] && echo exists $ [[ -v "$my_array["key"]" ]] && echo exists $ bash --version GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu) I've recently been documenting some of this on my website: https://joshua.hu/more-fun-with-bash-ssh-and-ssh-keygen-version-differences https://joshua.hu/more-fun-with-bash-ssh-and-ssh-keygen-vers...
- throw0101a 3y agoLots of mentions of this: * https://news.ycombinator.com/from?site=shellcheck.net https://news.ycombinator.com/from?site=shellcheck.net with the last large-scale discussion (301 points; 54 comments) being in 2021: * https://news.ycombinator.com/item?id=27030504 https://news.ycombinator.com/item?id=27030504
- pvg 3y agoIt's actually 'follow-up dupe' of this https://news.ycombinator.com/item?id=38387464 https://news.ycombinator.com/item?id=38387464 where it comes up repeatedly.
- hyllos 3y agoI’ve turned some time ago a build and deploy script (single production server) some bash scripts into Haskell using Turtle [1]. What I enjoyed was the ability to reduce redundancies significantly. It was significantly shorter code afterwards. [1] https://hackage.haskell.org/package/turtle https://hackage.haskell.org/package/turtle
- mrkeen 3y agoI recently tried Turtle but ended up throwing it out in favour of typed-process. Afaik a Turtle program has a single current directory, which makes it hard when you want to run concurrent jobs that need to be executed from particular directories. I partially solved the problem by using locks/queues/workers. But it got too much for me when Turtle started failing due to its current directory being deleted. In contrast, typed-process lets you spawn separate processes, and execute within a working dir (rather than needing to cd there), so it works great for big, complicated workflows. And it also has good support for OverloadedStrings, which means you can generally copy & paste what you would have typed into bash, and it just works. I also use the interpolate package (with QuasiQuotes) to make the raw strings nicer in the source code, but it's not compatible with hlint, so I'm thinking of looking for a different package for string-handling.
- themk 3y agoAuthor of Shh [0] here. I've replaced a lot of Bash with Haskell (it's great!). I wrote Shh to help me out with that. I ended up liking PyF [1] for quasiquoting. A friend of mine worked with the author a while ago to strip down it's dependencies, so it's easy to justify. It's what I recommend in my cargo culting guide [2]. [0] https://hackage.haskell.org/package/shh https://hackage.haskell.org/package/shh [1] https://hackage.haskell.org/package/PyF https://hackage.haskell.org/package/PyF [2] https://github.com/luke-clifton/shh/blob/master/docs/porting.md#variable-substitution https://github.com/luke-clifton/shh/blob/master/docs/porting...
- mr-wendel 3y agoSome tips of my own: - It's almost always preferable to put `-u` (nounset) in your shebang to cause using undeclared variables to be an error. The only exception I typically run across is expansion of arrays using "${arr[@]}" syntax -- if the array is empty, this is considered unbound. - You can use `-n` (noexec) as a poor-man's attempt at a dry-run, as this will prevent execution of commands. - Also handy is `-e` (errexit), but you must take care to observe that essentially, this only causes "naked" commands that fail to cause an exit. Personally, I prefer to avoid this and append `|| fail "..."` to commands liberally.
- Calzifer 3y ago> to put `-u` (nounset) in your shebang Any particular reason why in the shebang instead of set -u? > The only exception I typically run across is expansion of arrays using "${arr[@]}" syntax In Bash? Works for me. Edit: another comment mentions it as well. Seem to behave better in newer versions of Bash and only problematic in <= 4.3 https://news.ycombinator.com/item?id=38397241 https://news.ycombinator.com/item?id=38397241 $ bash -uc 'unset x; echo "=> ${x[@]}"' => $ bash -uc 'x=(); echo "=> ${x[@]}"' => $ bash -uc 'x=(); echo "=> ${x[0]}"' bash: x[0]: unbound variable Zsh does not like the first example but both should support: $ bash -uc 'unset x; echo "=> ${x[@]:-null}"' => null > Also handy is `-e` (errexit), It is unfortunately very confusion with functions. Made me like it less over the years.
- mr-wendel 3y agoUsing the shebang just helps highlight the fact that the rule is in use globally, but otherwise has no advantage to using `set -u`. The clarifications on `-u` and arrays are useful. I'm definitely used to assuming newer (... non-ancient?) versions of Bash are what is available.
- Xophmeister 3y agoUsing `set -u` is more portable. If your shebang is `/usr/bin/env bash`, which it probably should be, then you can't add additional command line arguments in Linux with older coreutils. macOS supports additional arguments, regardless, and in Linux, coreutils 8.30 added the `-S` option to `env` to get around this problem.
- jamespwilliams 3y agoShellcheck is a godsend https://github.com/jamespwilliams/strictbash https://github.com/jamespwilliams/strictbash, I wrote this little wrapper a while back that you can use as a shebang for scripts. It runs shellcheck for you before the script executes, so it’s not possible to run the script at all if there are failures. It also sets all the bash “strict mode” [0] flags. [0] http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/
- w10-1 3y agoShellcheck is great, but requires some investment to tailor to your style Disable default checks or enable optional ones using directives: https://www.shellcheck.net/wiki/Directive https://www.shellcheck.net/wiki/Directive The error checks can be pretty arcane: https://github.com/koalaman/shellcheck/wiki/Checks https://github.com/koalaman/shellcheck/wiki/Checks I appreciate that the text for each check is brief and usually includes a suggestion. I end up disabling 26xx's a lot (for unquoted variables to be interpreted as multiple values). Python is probably the best alternative to bash, but Swift is getting surprisingly good. With shwift[1] you get NIO/async APIs, operator overloading for shell-like locutions, and trivial access to existing executables: /// Piping between two executables try await echo("Foo", "Bar") | sed("s/Bar/Baz/") /// Piping to a builtin try await echo("Foo", "Bar") | map { $0.replacingOccurrences(of: "Bar", with: "Baz") } Scripts can easily be configured with libraries and run pre-compiled by using clutch[2]. For cross-platform use, be sure only use libraries on all platforms (i.e., not Foundation). It's a pain, but at least the error shows up typically at compile-time instead of run-time. [1] - [shwift](https://github.com/GeorgeLyon/Shwift https://github.com/GeorgeLyon/Shwift) [2] - [clutch - any Swift scripts in a common nest](https://github.com/swift-nest/clutch https://github.com/swift-nest/clutch)
- 1vuio0pswjnm7 3y agoWhat about finding bugs in other peoples' shell scripts.
- Diti 3y agoBut it still doesn’t find bugs in my Zsh scripts. :<
- oriettaxx 3y agovery good the Mercedes Benz sponsor :)
- pseudo_meta 3y agozsh is not officially supported, but you can force shellcheck to check zsh regardless via `--shell=bash`. This makes shellcheck treat the file as bash script, so there are some false positives for zsh-specific syntax, but 99% of the rules work pretty much the same.
- arp242 3y agoCertainly for my own scripts it's a lot less than "99%". It won't even run on most of them as it will just exit on some fatal error: Special variables like $argv and $status. Most array things (e.g. ${(o)arr} for an ordered array) (( .. )) conditionals "short loops" without "do". repeat loops. You could rewrite these things, but what errors remain where shellcheck is helpful in zsh?
- pseudo_meta 3y agowell, for me it's 99%. But true, I don't use short loops, and I also don't use array a lot (when I feel the needs to use arrays, I switch to a different language anyway). "what errors remain" is hard for me to say – I get a wide variety of helpful errors, regularly.
- arp242 3y agoEven in simple scripts I found arrays to be very helpful once I let go of the "POSIX sh mindset" and was used to it. The way word-splitting and $IFS works is basically like an "implicit array", as is $@, $1, etc. (actually in zsh $@ is an array: $1, $@[1], $*[1], and $argv[1] are all the same), so everyone already uses arrays. It replaces a lot of uses of cut and "awk '{print $2}" for starters.