8 ms·

ShellCheck: Finds bugs in your shell scripts

301 points by superasn 5y ago

fiddlerwoaroof 5y ago
This is one of the absolutely essential tools for anyone who writes a non-trivial amount of shell scripts.
- nomel 5y ago
  Maybe it says something about the people I work with, but I've never seen a fully valid shell script, over 20 lines long, that handled spaces in paths and command errors properly, that didn't first go through ShellCheck. I've noticed that hardcore *nix people are usually some of the worst offenders, with their phobia of spaces, from years of broken shell scripts, completely preventing them from even thinking about the possibility of a space in a path, with some considering the space the bug, rather than its handling. I've found that the best way to introduce someone to ShellCheck is to run it on their script, right in front of them, with most saying something like "How did I not know about this?". Linters are great!
  - naniwaduni 5y ago
    The kind of issues shellcheck catches are mostly of the "didn't learn shell (and didn't believe that shell was a language that had to be learned)" kind. Or to put it another way, once you have a good hang of the shell language's peculiarities (which differ from most general-purpose languages, but are fairly self-consistent with a few high-frequency exceptions) ... well, less of your bugs are ones that shellcheck can catch. I consider it an example of the general trend against learning domain-specific languages.
  - ericbarrett 5y ago
    I can and do write shell scripts all the time which properly handle spaces, without ShellCheck. Not to say that it isn’t good to use, especially in a professional environment, but it’s not that hard. For parallel processing, xargs -0 and find -0 are your friends!
  - chasil 5y ago
    Oh good heavens, my friend. My boss and coworker just spent the weekend on seven million files on my script. find . -type f -print0 | xargs -0 (cygwin) sed 's/\ff//' They think me a genius. This is nothing.
    - pbhjpbhj 5y ago
      Did your regex get munged there? \f (form feed) is something I've never come across (well maybe back in BBC basic?) - what's the context of removing it/them (I'm guessing only pairs \f\f) from 7M files?
  - dwheeler 5y ago
    I have. There are a few simple rules that, once you learn them, eliminate those kinds of problems. The main one is always use double-quotes around variable dereferences. The second is learning a few patterns, like find ... -print0 | xargs -0 ... But even after you've learned those simple rules, shellcheck is still useful. It will find fewer problems, but that's a good thing, that means you learned something :-).
- cwingrav 5y ago
  Agreed. I have it built into my vim editor, running after saves. Not only does it help catch errors, it catches potential errors, poor formatting and has become an amazing learning tool. Bash isn’t my favorite language, but many of the issues I had with it are no longer relevant.
whateveracct 5y ago
probably one of the more widely-deployed Haskell programs
- deleted 5y ago
  [deleted]
- johnday 5y ago
  This and `pandoc` are solid proof that Haskell is a superb language for processing structured text (be it for compilation, or translation, or checking).
  - cle 5y ago
    Or that people who are good at processing structured text tend to use/like Haskell. (Not disagreeing with you, but the causal direction isn't immediately obvious to me.)
- chasil 5y ago
  I do admire it, for it's grieovus errors. It does not understand "nameref" amongst other flaws.
Tsiklon 5y ago
I love ShellCheck. It’s good at providing hints at improving your style too.
mey 5y ago
There is a convenient VS Code plugin for ShellCheck. https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck https://marketplace.visualstudio.com/items?itemName=timonwon...
Kenji 5y ago
I can highly recommend this tool. A few years ago, I ran it over our shell scripts and it gave us many good recommendations and even found some bugs. I learned a lot about shell scripting during that time. Really, I cannot recommend it enough, what a fantastic tool.
teeray 5y ago
The best best part of Shellcheck is that the errors it produces have a short code that can be looked up in their wiki. The wiki explains what you did, why it’s bad, and how to fix it. Using Shellcheck teaches you to write better Bash, rather than just yelling at you with inscrutable linter violations.
- TheLocehiliosan 5y ago
  If you use iTerm2 (or I think this feature is available in other terminal software) you can use "Smart Selections" to transform any ShellCheck errors into links to the wiki. 1. Create a Smart Selection with the regular expression "SC\d+" 2. Add an action to "Open URL...", with the URL "https://github.com/koalaman/shellcheck/wiki/\0 https://github.com/koalaman/shellcheck/wiki/\0" (\0 will be replaced with the matched regex) 3. Now holding down the command key will change any ShellCheck errors to be active http links.
- chasil 5y ago
  BASH is not an appropriate standard to use in writing a script. ShellCheck recognizes three hashbangs, POSIX, BASH, and the Korn shell (as is explained in the manual page). I would argue and assert that BASH is the least relevant of the three. I have occasionally needed to write to the POSIX standard, and remove advanced features from my scripts. This is most commonly needed for BusyBox (including Windows) and Ubuntu/Debian DASH. It is important to understand what is not in POSIX, for the time when an advanced shell isn't available. Please browse the POSIX definition, reported by a google of "POSIX shell": https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... The Korn shell lived and diverged for many years, beginning with ksh88 (issued before the first BASH release in 1989). David Korn retired, and AT&T was not pleased with ksh93 changes implemented by outsiders who impacted both compatibility and performance, and thus ksh93 was rewound. Cygwin, notibly, only implements mksh for Windows. The mksh binary on all platforms is a fraction of the size of BASH (check the output of the "size" command if an "ls" does not persuade you). However, the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020. It is also on every Android phone (chosen as the system shell by Google), so its deployment is massive. When I am purposely writing above the POSIX shell, I choose mksh from MirBSD as my standard. I do not think the BASH innovations to be well-thought.
LanternLight83 5y ago
Some related resources, for intermediate shell users already familiar with shellcheck and looking to dig deeper into the history underlying obscure corner cases, include the developer Vidar's blog (@ https://www.vidarholen.net/contents/blog/ https://www.vidarholen.net/contents/blog/), the blog of the Oil-Shell developer (@ https://www.oilshell.org/blog/ https://www.oilshell.org/blog/), and Greg's Wiki (@ https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls).
pabs3 5y ago
Some other shell script check and formatting tools: All shells take the -n option to perform a basic syntax check. Debian has a script for checking scripts for bashisms: https://manpages.debian.org/checkbashisms https://manpages.debian.org/checkbashisms bashate is a automated style checker for bash (similar to pep8 for python): https://opendev.org/openstack/bashate https://opendev.org/openstack/bashate lintshell is an early prototype of a shell linter based on the Morbig trustworthy static parser for POSIX shell, based on the Why3 platform for deductive program verification. https://www.irif.fr/~treinen/colis/ https://www.irif.fr/~treinen/colis/ https://github.com/colis-anr/morbig https://github.com/colis-anr/morbig https://github.com/colis-anr/lintshell https://github.com/colis-anr/lintshell Shellharden helps rewrite scripts for ShellCheck compliance: https://github.com/anordal/shellharden/ https://github.com/anordal/shellharden/ shfmt is gofmt for shell: https://github.com/mvdan/sh https://github.com/mvdan/sh ShellScriptFormatter is another formatter: https://github.com/osalvador/ShellScriptFormatter https://github.com/osalvador/ShellScriptFormatter Some of these tools and other tools can be automatically run by check-all-the-things: https://github.com/collab-qa/check-all-the-things/ https://github.com/collab-qa/check-all-the-things/
yamellasmallela 5y ago
ShellCheck is absolutely essential. Make sure to get the vscode plugin as well. I love it. The error code can be googled and it the first result will be exactly what you need. One of the best projects to happen to open source.
devenblake 5y ago
> E044: Use [[ for =~,<,> comparisions Is there any reason to prefer bashisms over POSIX?
- yamellasmallela 5y ago
  because its 2021 and we should get over compatibility with archaic tech?
- koala_man 5y ago
  That is bashate, not ShellCheck, but there is good reason to use [[ since these operators are not POSIX to begin with and don't work in [.
- chasil 5y ago
  In Debian/Ubuntu /bin/sh, most certainly.
- hashhar 5y ago
  Because the =~ and > and < operators aren't defined for [ in POSIX. Also ShellCheck suggests similar improvement only when your shebang says that bash is the expected interpreter instead of sh (or other ways of denoting POSIX compat).
sanketsaurav 5y ago
ShellCheck is a great tool! If you're looking for an easy way to run static analysis on .sh files continuously, you might want to look at DeepSource. ShellCheck is supported. https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/ https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/
aVx1uyD5pYWW 5y ago
Unfortunately, last I checked, shellcheck is not compatible with M1 macs yet. I love the tool so much that I while I wait for compatibility, I manually check my shell scripts at the web UI.
- theobeers 5y ago
  Yeah, there are still issues with Haskell and the new Mac architecture. Pandoc and ShellCheck are two of the essential tools affected. In the case of ShellCheck, I just downloaded an x86 binary from the GitHub Releases page and put it in /usr/local/bin. Good enough for the time being. https://github.com/koalaman/shellcheck/releases https://github.com/koalaman/shellcheck/releases
nxpnsv 5y ago
It found several things in my script that I’ve done wrong for over 20 years...
dejj 5y ago
ShellCheck and templates from the Shell Field Guide[1] is what keeps my bank running. Rust cmd_lib[2] is tempting with safety, though tougher maintainability. [1] https://raimonster.com/scripting-field-guide/ https://raimonster.com/scripting-field-guide/ [2] https://github.com/rust-shell-script/rust_cmd_lib https://github.com/rust-shell-script/rust_cmd_lib
Aissen 5y ago
One of the biggest value of shellcheck is that all of its errors are documented on their wiki, and easily searchable. If you don't understand why there is an error, the wiki page will tell you why an idiom is bad, and in some cases point you to examples of how it might fail, and you'll learn something new. I wish the wiki was packaged into the tool for offline consumption.
tempodox 5y ago
This is a very helpful tool, I can only recommend it. Works perfectly well with vim and Syntastic.
exdsq 5y ago
I’ve written some pretty horrible Bash applications such as a load testing tool for a blockchain, probably hitting > 2kloc. ShellCheck and ShUnit made that possible.
dang 5y ago
Discussed a few weeks ago: ShellCheck: A static analysis tool for shell scripts - https://news.ycombinator.com/item?id=26504661 https://news.ycombinator.com/item?id=26504661 - March 2021 (73 comments) (We mark stories as dupes if they've had significant attention in the last year or so). Previous threads, for those who are curious: Lessons learned from writing ShellCheck - https://news.ycombinator.com/item?id=22279585 https://news.ycombinator.com/item?id=22279585 - Feb 2020 (46 comments) Shellcheck: a static analysis tool for shell scripts - https://news.ycombinator.com/item?id=9001931 https://news.ycombinator.com/item?id=9001931 - Feb 2015 (46 comments) ShellCheck: a static analysis and linting tool for sh/bash scripts - https://news.ycombinator.com/item?id=8777705 https://news.ycombinator.com/item?id=8777705 - Dec 2014 (20 comments) ShellCheck – Online shell script analyzer - https://news.ycombinator.com/item?id=8182745 https://news.ycombinator.com/item?id=8182745 - Aug 2014 (14 comments)
- superasn 5y ago
  Oops sorry about that. Just found out about this wonderful program from this HN comment here(1). Should have been more careful to check it through algolia before posting though. (1) https://news.ycombinator.com/item?id=16085914 https://news.ycombinator.com/item?id=16085914
  - dang 5y ago
    Not to worry! It happens.
scns 5y ago
The first plugin i install in Intellij alongside IdeaVim.

301 points by superasn 5y ago

fiddlerwoaroof 5y ago

This is one of the absolutely essential tools for anyone who writes a non-trivial amount of shell scripts.

nomel 5y ago
Maybe it says something about the people I work with, but I've never seen a fully valid shell script, over 20 lines long, that handled spaces in paths and command errors properly, that didn't first go through ShellCheck. I've noticed that hardcore *nix people are usually some of the worst offenders, with their phobia of spaces, from years of broken shell scripts, completely preventing them from even thinking about the possibility of a space in a path, with some considering the space the bug, rather than its handling. I've found that the best way to introduce someone to ShellCheck is to run it on their script, right in front of them, with most saying something like "How did I not know about this?". Linters are great!
- naniwaduni 5y ago
  The kind of issues shellcheck catches are mostly of the "didn't learn shell (and didn't believe that shell was a language that had to be learned)" kind. Or to put it another way, once you have a good hang of the shell language's peculiarities (which differ from most general-purpose languages, but are fairly self-consistent with a few high-frequency exceptions) ... well, less of your bugs are ones that shellcheck can catch. I consider it an example of the general trend against learning domain-specific languages.
- ericbarrett 5y ago
  I can and do write shell scripts all the time which properly handle spaces, without ShellCheck. Not to say that it isn’t good to use, especially in a professional environment, but it’s not that hard. For parallel processing, xargs -0 and find -0 are your friends!
- chasil 5y ago
  Oh good heavens, my friend. My boss and coworker just spent the weekend on seven million files on my script. find . -type f -print0 | xargs -0 (cygwin) sed 's/\ff//' They think me a genius. This is nothing.
  - pbhjpbhj 5y ago
    Did your regex get munged there? \f (form feed) is something I've never come across (well maybe back in BBC basic?) - what's the context of removing it/them (I'm guessing only pairs \f\f) from 7M files?
- dwheeler 5y ago
  I have. There are a few simple rules that, once you learn them, eliminate those kinds of problems. The main one is always use double-quotes around variable dereferences. The second is learning a few patterns, like find ... -print0 | xargs -0 ... But even after you've learned those simple rules, shellcheck is still useful. It will find fewer problems, but that's a good thing, that means you learned something :-).
cwingrav 5y ago
Agreed. I have it built into my vim editor, running after saves. Not only does it help catch errors, it catches potential errors, poor formatting and has become an amazing learning tool. Bash isn’t my favorite language, but many of the issues I had with it are no longer relevant.

whateveracct 5y ago

probably one of the more widely-deployed Haskell programs

deleted 5y ago
[deleted]
johnday 5y ago
This and `pandoc` are solid proof that Haskell is a superb language for processing structured text (be it for compilation, or translation, or checking).
- cle 5y ago
  Or that people who are good at processing structured text tend to use/like Haskell. (Not disagreeing with you, but the causal direction isn't immediately obvious to me.)
chasil 5y ago
I do admire it, for it's grieovus errors. It does not understand "nameref" amongst other flaws.

Tsiklon 5y ago

I love ShellCheck. It’s good at providing hints at improving your style too.

mey 5y ago

There is a convenient VS Code plugin for ShellCheck. https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck https://marketplace.visualstudio.com/items?itemName=timonwon...

Kenji 5y ago

I can highly recommend this tool. A few years ago, I ran it over our shell scripts and it gave us many good recommendations and even found some bugs. I learned a lot about shell scripting during that time. Really, I cannot recommend it enough, what a fantastic tool.

teeray 5y ago

The best best part of Shellcheck is that the errors it produces have a short code that can be looked up in their wiki. The wiki explains what you did, why it’s bad, and how to fix it. Using Shellcheck teaches you to write better Bash, rather than just yelling at you with inscrutable linter violations.

TheLocehiliosan 5y ago
If you use iTerm2 (or I think this feature is available in other terminal software) you can use "Smart Selections" to transform any ShellCheck errors into links to the wiki. 1. Create a Smart Selection with the regular expression "SC\d+" 2. Add an action to "Open URL...", with the URL "https://github.com/koalaman/shellcheck/wiki/\0 https://github.com/koalaman/shellcheck/wiki/\0" (\0 will be replaced with the matched regex) 3. Now holding down the command key will change any ShellCheck errors to be active http links.
chasil 5y ago
BASH is not an appropriate standard to use in writing a script. ShellCheck recognizes three hashbangs, POSIX, BASH, and the Korn shell (as is explained in the manual page). I would argue and assert that BASH is the least relevant of the three. I have occasionally needed to write to the POSIX standard, and remove advanced features from my scripts. This is most commonly needed for BusyBox (including Windows) and Ubuntu/Debian DASH. It is important to understand what is not in POSIX, for the time when an advanced shell isn't available. Please browse the POSIX definition, reported by a google of "POSIX shell": https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... The Korn shell lived and diverged for many years, beginning with ksh88 (issued before the first BASH release in 1989). David Korn retired, and AT&T was not pleased with ksh93 changes implemented by outsiders who impacted both compatibility and performance, and thus ksh93 was rewound. Cygwin, notibly, only implements mksh for Windows. The mksh binary on all platforms is a fraction of the size of BASH (check the output of the "size" command if an "ls" does not persuade you). However, the MirBSD Korn shell is one of the most lively shells for development activity, the latest release being in October of 2020. It is also on every Android phone (chosen as the system shell by Google), so its deployment is massive. When I am purposely writing above the POSIX shell, I choose mksh from MirBSD as my standard. I do not think the BASH innovations to be well-thought.

LanternLight83 5y ago

Some related resources, for intermediate shell users already familiar with shellcheck and looking to dig deeper into the history underlying obscure corner cases, include the developer Vidar's blog (@ https://www.vidarholen.net/contents/blog/ https://www.vidarholen.net/contents/blog/), the blog of the Oil-Shell developer (@ https://www.oilshell.org/blog/ https://www.oilshell.org/blog/), and Greg's Wiki (@ https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls).

pabs3 5y ago

Some other shell script check and formatting tools: All shells take the -n option to perform a basic syntax check. Debian has a script for checking scripts for bashisms: https://manpages.debian.org/checkbashisms https://manpages.debian.org/checkbashisms bashate is a automated style checker for bash (similar to pep8 for python): https://opendev.org/openstack/bashate https://opendev.org/openstack/bashate lintshell is an early prototype of a shell linter based on the Morbig trustworthy static parser for POSIX shell, based on the Why3 platform for deductive program verification. https://www.irif.fr/~treinen/colis/ https://www.irif.fr/~treinen/colis/ https://github.com/colis-anr/morbig https://github.com/colis-anr/morbig https://github.com/colis-anr/lintshell https://github.com/colis-anr/lintshell Shellharden helps rewrite scripts for ShellCheck compliance: https://github.com/anordal/shellharden/ https://github.com/anordal/shellharden/ shfmt is gofmt for shell: https://github.com/mvdan/sh https://github.com/mvdan/sh ShellScriptFormatter is another formatter: https://github.com/osalvador/ShellScriptFormatter https://github.com/osalvador/ShellScriptFormatter Some of these tools and other tools can be automatically run by check-all-the-things: https://github.com/collab-qa/check-all-the-things/ https://github.com/collab-qa/check-all-the-things/

yamellasmallela 5y ago

ShellCheck is absolutely essential. Make sure to get the vscode plugin as well. I love it. The error code can be googled and it the first result will be exactly what you need. One of the best projects to happen to open source.

devenblake 5y ago

> E044: Use [[ for =~,<,> comparisions Is there any reason to prefer bashisms over POSIX?

yamellasmallela 5y ago
because its 2021 and we should get over compatibility with archaic tech?
koala_man 5y ago
That is bashate, not ShellCheck, but there is good reason to use [[ since these operators are not POSIX to begin with and don't work in [.
chasil 5y ago
In Debian/Ubuntu /bin/sh, most certainly.
hashhar 5y ago
Because the =~ and > and < operators aren't defined for [ in POSIX. Also ShellCheck suggests similar improvement only when your shebang says that bash is the expected interpreter instead of sh (or other ways of denoting POSIX compat).

sanketsaurav 5y ago

ShellCheck is a great tool! If you're looking for an easy way to run static analysis on .sh files continuously, you might want to look at DeepSource. ShellCheck is supported. https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/ https://deepsource.io/gh/deepsourcelabs/demo-shell/issues/

aVx1uyD5pYWW 5y ago

Unfortunately, last I checked, shellcheck is not compatible with M1 macs yet. I love the tool so much that I while I wait for compatibility, I manually check my shell scripts at the web UI.

theobeers 5y ago
Yeah, there are still issues with Haskell and the new Mac architecture. Pandoc and ShellCheck are two of the essential tools affected. In the case of ShellCheck, I just downloaded an x86 binary from the GitHub Releases page and put it in /usr/local/bin. Good enough for the time being. https://github.com/koalaman/shellcheck/releases https://github.com/koalaman/shellcheck/releases

nxpnsv 5y ago

It found several things in my script that I’ve done wrong for over 20 years...

dejj 5y ago

ShellCheck and templates from the Shell Field Guide[1] is what keeps my bank running. Rust cmd_lib[2] is tempting with safety, though tougher maintainability. [1] https://raimonster.com/scripting-field-guide/ https://raimonster.com/scripting-field-guide/ [2] https://github.com/rust-shell-script/rust_cmd_lib https://github.com/rust-shell-script/rust_cmd_lib

Aissen 5y ago

One of the biggest value of shellcheck is that all of its errors are documented on their wiki, and easily searchable. If you don't understand why there is an error, the wiki page will tell you why an idiom is bad, and in some cases point you to examples of how it might fail, and you'll learn something new. I wish the wiki was packaged into the tool for offline consumption.

tempodox 5y ago

This is a very helpful tool, I can only recommend it. Works perfectly well with vim and Syntastic.

exdsq 5y ago

I’ve written some pretty horrible Bash applications such as a load testing tool for a blockchain, probably hitting > 2kloc. ShellCheck and ShUnit made that possible.

dang 5y ago

Discussed a few weeks ago: ShellCheck: A static analysis tool for shell scripts - https://news.ycombinator.com/item?id=26504661 https://news.ycombinator.com/item?id=26504661 - March 2021 (73 comments) (We mark stories as dupes if they've had significant attention in the last year or so). Previous threads, for those who are curious: Lessons learned from writing ShellCheck - https://news.ycombinator.com/item?id=22279585 https://news.ycombinator.com/item?id=22279585 - Feb 2020 (46 comments) Shellcheck: a static analysis tool for shell scripts - https://news.ycombinator.com/item?id=9001931 https://news.ycombinator.com/item?id=9001931 - Feb 2015 (46 comments) ShellCheck: a static analysis and linting tool for sh/bash scripts - https://news.ycombinator.com/item?id=8777705 https://news.ycombinator.com/item?id=8777705 - Dec 2014 (20 comments) ShellCheck – Online shell script analyzer - https://news.ycombinator.com/item?id=8182745 https://news.ycombinator.com/item?id=8182745 - Aug 2014 (14 comments)

superasn 5y ago
Oops sorry about that. Just found out about this wonderful program from this HN comment here(1). Should have been more careful to check it through algolia before posting though. (1) https://news.ycombinator.com/item?id=16085914 https://news.ycombinator.com/item?id=16085914
- dang 5y ago
  Not to worry! It happens.

scns 5y ago

The first plugin i install in Intellij alongside IdeaVim.