10 ms·

Shellcheck: a static analysis tool for shell scripts

102 points by pyotrgalois 12y ago

ygra 12y ago
As someone who regularly answers batch file questions on Stack Overflow, I think this would be invaluable for all the mistakes people make there too.
rtpg 12y ago
Isn't this a pretty good argument against shell scripts? I feel like we've advanced far enough in PL research to think of something a bit safer
- icot 12y ago
  Well, there's plenty of legacy code laying around that may be easier to improve and fix than completely rewrite. In fact, I just presented this tool to my co-workers as a suggestion to clean over 150KLOC of shell scripts we have laying around.
  - igravious 12y ago
    Be interesting to run this checker on the default scripts that come with major UNIX OSes like MacOSX and Ubuntu and Fedora and the like - sounds like a great janitorial project...
- pjmlp 12y ago
  The Xerox PARC and ETHZ answer to that would be REPL instead of shell.
  - xiaq 12y ago
    A shell is a REPL. The problem is that the shell is sloppy in that it only deals with bytes and cannot process any complex data structure. Again, advertisement for my side project https://github.com/elves/elvish https://github.com/elves/elvish, a Unix shell with true data structures. Still a WIP though.
    - omaranto 12y ago
      There is already a well-developed shell with rich data structures and a fairly reasonable programming language: Microsoft's PowerShell. Sadly it is not a Unix shell. You're probably aware of it, but if not, check it out for design inspiration.
      - xiaq 12y ago
        Of course! PowerShell definitely has a lot of brilliant ideas. Sadly it is overenginnered and has quite some design mistakes. Nevertheless it has served as a great source of inspiration for me - I have actually gone through several PowerShell manuals before I started elvish.
        ygra 12y ago
        As someone who loves PowerShell and uses it daily, may I ask for specifics for design mistakes and overengineering? You may also answer per mail if you want. Don't get me wrong, I realise it has its flaws and warts, but for me, and comparing to cmd or bash I still think it's very, very much an improvement. Off the top of my head actual mistakes (the sort that tends to bite many people) include handling of [ and ] in -Path arguments (necessitating -LiteralPath arguments in later versions), and the constant wondering whether something returns a scalar or an array (and an array of one element being unwrapped into a scalar automatically). During my time working on Pash I also noted a few weirdnesses on source code side, most recently and notably LanguagePrimitives.Convert which has a dependency on the currently-executing runspace (which is stored in a thread-local field).
    - PantaloonFlames 12y ago
      > it only deals with bytes and cannot process any complex data structure. Just FYI. Microsoft tried to address that problem in Windows a long time ago when it introduced Powershell.
      - xiaq 12y ago
        See my reply to omaranto.
- xiaq 12y ago
  Time for advertisement! You might like elvish https://github.com/elves/elvish https://github.com/elves/elvish which proudly has optional typing and much more well-defined semantics than say, bash. This is work in progress though. Advertisement aside, there is some inherit unsafety in shell scripts that cannot be easily resolved, namely the unsafety involved in interacting with external commands. Compared to other scripting languages, the greatest advantage of shell languages is the convenience of interacting with external programs. However, at least in Unix, there are few static constraints you can apply to them. Everything we know is that the program will (probably) parse something in argv which are just bytes, (probably) take something from stdin which are just bytes, and (probably) put something to stdout which are again just bytes; there is no universal method to check that the commands arguments are well-formed, or the input format is correct, or the output format conforms to a certain schema without running the actual program. A solution is to define some kind of static protocols for external programs so that their invocations can be statically checked, but it's already too late.
  - hyperpape 12y ago
    Interesting! On first glance, this might be the most appealing attempt to improve on the shell I've seen yet. It's a really hard design space. Some questions: Why is set necessary? Once you have declaration with var, can't mutation be done without set? What's your thinking behind making var mandatory for declarations? Safety is obvious, but it seems like terseness is a really big goal for shell programming, especially interactive use. Also, documentation wise, I don't see how/if you do variable expansion in strings. Same as sh?
    - xiaq 12y ago
      var is for declaration, set for assignment. This is an important contrast that some dynamic languages miss; ironically JavaScript got it right. Contrast this var $x = "foo"; if $true { set $x = "bar" }; echo $x # outputs "bar" with var $x = "foo"; if $true { var $x = "bar" }; echo $x # outputs "foo" The declaration/assignment contrast is very important when it comes to closures (and there are closures in elvish). In python 2, for instance, there is no way (!) to assign to outer variables in closures since `=` declares and assigns at the same time in a `def` block. There are no variable expansions, but strings are concatenated implicitly when they run together. In sh: echo "hello $name, welcome!" In elvish: echo "hello "$name", welcome!" Implicit concatenation can read a bit weird at first, but it's actually conceptually much simpler and only slightly more cumbersome than string interpolation. It also makes the syntax much simpler.
- jcurbo 12y ago
  How about a Haskell DSL? http://www.haskellforall.com/2015/01/use-haskell-for-shell-scripting.html http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...
  - codygman 12y ago
    I've been using this and it's pretty nice. See a real world example in this pull request: https://github.com/Gabriel439/Haskell-Turtle-Library/commit/53273ab902ec414a6f70a1fc721665850ca4d9ff https://github.com/Gabriel439/Haskell-Turtle-Library/commit/...
dpina 12y ago
Just spent some time sending my scripts to this site for it to analyse and see what it does. I can see that while it wasn't be able to tell me of more efficient code to achieve my goal (wasn't really hopping for that), it did spot 1) one liners where some commands are not needed, 2) variables which are not used, 3) where I should use double quotes to prevent word splitting and 4) lines where my ssh was eating up my stdin. What a great sanity check for the days when I'm writing something on my own without a second pair of eyes to proof-read it.
xrstf 12y ago
This is awesome. As a beginner when it comes to writing shellscripts, this is my new jshint equivalent.
Munksgaard 12y ago
This looks very helpful: bash scripts are notoriously difficult to get right. I wish it'd suggest best practices like `set -e` and the like though.
- koala_man 12y ago
  The jury's still out on whether `set -e` is worth it. On paper it sounds like it's equivalent to `on error goto 0`, making a script fail-fast -- which would have been awesome. Instead, it makes a script fail sometimes for things that are sometimes errors. The rules for how and when are unexpected and unintuitive, several weird cases are described on http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105 If enabling it just granted a free 50% chance of stopping on any given error, it would have been worth it, but it triggers on false positives as well.
  - vetrom 12y ago
    Well, dealing with horrible exception handling is still better than just silently running off the edge on errors. Much like any language, you need to read and understand it to be able to truly write, I think. bash is deceptive in this regard IMO, due to how low its barrier to entry is. Now if you want to see some truly horrible code, implement a shell script that runs in bash and zsh and does exception printing in both.
nimrody 12y ago
The real problem with shell scripts is that they usually tie together a few external commands and tend to pass information around using the filesystem. This, together with poor error handling is a recipe for disaster: Problems with permissions, insufficient disk space, etc. Instead of stopping when encountering an error, most shell scripts will happily continue break at some other point in time (or worse - destroy valuable data).
reedlaw 12y ago
I ran it against a deploy script generated by Mina [1]. Only a few deprecation warnings and notes about using find instead of ls to better handle non-alphanumeric filenames. I've learned a lot about error handling by reading Mina-generated scripts. 1. http://nadarei.co/mina/ http://nadarei.co/mina/
Chico75 12y ago
Combine it with the sublime text plugin (https://github.com/SublimeLinter/SublimeLinter-shellcheck https://github.com/SublimeLinter/SublimeLinter-shellcheck) and you got real time static analysis while without your shell scripts !
grymoire1 12y ago
The emacs interface is very nice as well, using flycheck
eridius 12y ago
Shellcheck is great. The Vim Syntastic plugin already knows about Shellcheck so if you use Syntastic and install Shellcheck you'll automatically start getting warnings on your code. BTW, it can be a little hard to figure this out, but if Shellcheck gives you a warning that you want to ignore (because you intended to trigger that behavior), you can put the following comment above the offending line: # shellcheck disable=SC1234 where "SC1234" is replaced with the actual error code that Shellcheck gives.

102 points by pyotrgalois 12y ago

ygra 12y ago

As someone who regularly answers batch file questions on Stack Overflow, I think this would be invaluable for all the mistakes people make there too.

rtpg 12y ago

Isn't this a pretty good argument against shell scripts? I feel like we've advanced far enough in PL research to think of something a bit safer

icot 12y ago
Well, there's plenty of legacy code laying around that may be easier to improve and fix than completely rewrite. In fact, I just presented this tool to my co-workers as a suggestion to clean over 150KLOC of shell scripts we have laying around.
- igravious 12y ago
  Be interesting to run this checker on the default scripts that come with major UNIX OSes like MacOSX and Ubuntu and Fedora and the like - sounds like a great janitorial project...
pjmlp 12y ago
The Xerox PARC and ETHZ answer to that would be REPL instead of shell.
- xiaq 12y ago
  A shell is a REPL. The problem is that the shell is sloppy in that it only deals with bytes and cannot process any complex data structure. Again, advertisement for my side project https://github.com/elves/elvish https://github.com/elves/elvish, a Unix shell with true data structures. Still a WIP though.
  - omaranto 12y ago
    There is already a well-developed shell with rich data structures and a fairly reasonable programming language: Microsoft's PowerShell. Sadly it is not a Unix shell. You're probably aware of it, but if not, check it out for design inspiration.
    - xiaq 12y ago
      Of course! PowerShell definitely has a lot of brilliant ideas. Sadly it is overenginnered and has quite some design mistakes. Nevertheless it has served as a great source of inspiration for me - I have actually gone through several PowerShell manuals before I started elvish.
      - ygra 12y ago
        As someone who loves PowerShell and uses it daily, may I ask for specifics for design mistakes and overengineering? You may also answer per mail if you want. Don't get me wrong, I realise it has its flaws and warts, but for me, and comparing to cmd or bash I still think it's very, very much an improvement. Off the top of my head actual mistakes (the sort that tends to bite many people) include handling of [ and ] in -Path arguments (necessitating -LiteralPath arguments in later versions), and the constant wondering whether something returns a scalar or an array (and an array of one element being unwrapped into a scalar automatically). During my time working on Pash I also noted a few weirdnesses on source code side, most recently and notably LanguagePrimitives.Convert which has a dependency on the currently-executing runspace (which is stored in a thread-local field).
  - PantaloonFlames 12y ago
    > it only deals with bytes and cannot process any complex data structure. Just FYI. Microsoft tried to address that problem in Windows a long time ago when it introduced Powershell.
    - xiaq 12y ago
      See my reply to omaranto.
xiaq 12y ago
Time for advertisement! You might like elvish https://github.com/elves/elvish https://github.com/elves/elvish which proudly has optional typing and much more well-defined semantics than say, bash. This is work in progress though. Advertisement aside, there is some inherit unsafety in shell scripts that cannot be easily resolved, namely the unsafety involved in interacting with external commands. Compared to other scripting languages, the greatest advantage of shell languages is the convenience of interacting with external programs. However, at least in Unix, there are few static constraints you can apply to them. Everything we know is that the program will (probably) parse something in argv which are just bytes, (probably) take something from stdin which are just bytes, and (probably) put something to stdout which are again just bytes; there is no universal method to check that the commands arguments are well-formed, or the input format is correct, or the output format conforms to a certain schema without running the actual program. A solution is to define some kind of static protocols for external programs so that their invocations can be statically checked, but it's already too late.
- hyperpape 12y ago
  Interesting! On first glance, this might be the most appealing attempt to improve on the shell I've seen yet. It's a really hard design space. Some questions: Why is set necessary? Once you have declaration with var, can't mutation be done without set? What's your thinking behind making var mandatory for declarations? Safety is obvious, but it seems like terseness is a really big goal for shell programming, especially interactive use. Also, documentation wise, I don't see how/if you do variable expansion in strings. Same as sh?
  - xiaq 12y ago
    var is for declaration, set for assignment. This is an important contrast that some dynamic languages miss; ironically JavaScript got it right. Contrast this var $x = "foo"; if $true { set $x = "bar" }; echo $x # outputs "bar" with var $x = "foo"; if $true { var $x = "bar" }; echo $x # outputs "foo" The declaration/assignment contrast is very important when it comes to closures (and there are closures in elvish). In python 2, for instance, there is no way (!) to assign to outer variables in closures since `=` declares and assigns at the same time in a `def` block. There are no variable expansions, but strings are concatenated implicitly when they run together. In sh: echo "hello $name, welcome!" In elvish: echo "hello "$name", welcome!" Implicit concatenation can read a bit weird at first, but it's actually conceptually much simpler and only slightly more cumbersome than string interpolation. It also makes the syntax much simpler.
jcurbo 12y ago
How about a Haskell DSL? http://www.haskellforall.com/2015/01/use-haskell-for-shell-scripting.html http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...
- codygman 12y ago
  I've been using this and it's pretty nice. See a real world example in this pull request: https://github.com/Gabriel439/Haskell-Turtle-Library/commit/53273ab902ec414a6f70a1fc721665850ca4d9ff https://github.com/Gabriel439/Haskell-Turtle-Library/commit/...

dpina 12y ago

Just spent some time sending my scripts to this site for it to analyse and see what it does. I can see that while it wasn't be able to tell me of more efficient code to achieve my goal (wasn't really hopping for that), it did spot 1) one liners where some commands are not needed, 2) variables which are not used, 3) where I should use double quotes to prevent word splitting and 4) lines where my ssh was eating up my stdin. What a great sanity check for the days when I'm writing something on my own without a second pair of eyes to proof-read it.

xrstf 12y ago

This is awesome. As a beginner when it comes to writing shellscripts, this is my new jshint equivalent.

Munksgaard 12y ago

This looks very helpful: bash scripts are notoriously difficult to get right. I wish it'd suggest best practices like `set -e` and the like though.

koala_man 12y ago
The jury's still out on whether `set -e` is worth it. On paper it sounds like it's equivalent to `on error goto 0`, making a script fail-fast -- which would have been awesome. Instead, it makes a script fail sometimes for things that are sometimes errors. The rules for how and when are unexpected and unintuitive, several weird cases are described on http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105 If enabling it just granted a free 50% chance of stopping on any given error, it would have been worth it, but it triggers on false positives as well.
- vetrom 12y ago
  Well, dealing with horrible exception handling is still better than just silently running off the edge on errors. Much like any language, you need to read and understand it to be able to truly write, I think. bash is deceptive in this regard IMO, due to how low its barrier to entry is. Now if you want to see some truly horrible code, implement a shell script that runs in bash and zsh and does exception printing in both.

nimrody 12y ago

The real problem with shell scripts is that they usually tie together a few external commands and tend to pass information around using the filesystem. This, together with poor error handling is a recipe for disaster: Problems with permissions, insufficient disk space, etc. Instead of stopping when encountering an error, most shell scripts will happily continue break at some other point in time (or worse - destroy valuable data).

reedlaw 12y ago

I ran it against a deploy script generated by Mina [1]. Only a few deprecation warnings and notes about using find instead of ls to better handle non-alphanumeric filenames. I've learned a lot about error handling by reading Mina-generated scripts. 1. http://nadarei.co/mina/ http://nadarei.co/mina/

Chico75 12y ago

Combine it with the sublime text plugin (https://github.com/SublimeLinter/SublimeLinter-shellcheck https://github.com/SublimeLinter/SublimeLinter-shellcheck) and you got real time static analysis while without your shell scripts !

grymoire1 12y ago

The emacs interface is very nice as well, using flycheck

eridius 12y ago

Shellcheck is great. The Vim Syntastic plugin already knows about Shellcheck so if you use Syntastic and install Shellcheck you'll automatically start getting warnings on your code. BTW, it can be a little hard to figure this out, but if Shellcheck gives you a warning that you want to ignore (because you intended to trigger that behavior), you can put the following comment above the offending line: # shellcheck disable=SC1234 where "SC1234" is replaced with the actual error code that Shellcheck gives.