5 ms·

Testing Bash with BATS

51 points by samebreath 8y ago

orestes910 8y ago
Outside of the scope of elaborate CI pipelines, I wonder how useful this can really be. Big CI pipelines are one of the few instances where I can think of Bash being both an appropriate choice AND the resulting product being large, elaborate, and sensitive to failure - which would benefit from being tested. Most other applications of Bash are generally just so simple that fundamentally altering how you write scripts ("Bash scripts must also be broken down into multiple functions, which the main part of the script should call when the script is executed.") for the sake of testing them seems like it could easily fall into the category of over engineering. Beyond the CI pipeline use case, wouldn't the tools in which this would actually be properly useful be better off written in a proper programming language?
- npongratz 8y ago
  I've found shell scripting to be superior to other programming languages when the job calls for performing some combination of the following: 1) calling lots of commands 2) using lots of pipelines 3) handling exit codes from 1 and 2 for error handling and conditional execution control 4) doing job control for processes and process groups (backgrounding, pausing, foregrounding, sending signals, etc.)
  - jasonpeacock 8y ago
    Have you looked at Plumbumm for Python? It handles everything you just mentioned: https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ There's no reason not to use Python to achieve the same thing, and you get all the benefits of using Python instead of Bash. I'm sure other languages have similar libraries.
- LukeShu 8y ago
  Whenever someone says "most people don't write large Bash scripts", I have to chime in and say that I would consider most GNU/Linux distros to be giant piles of shell scripts. A year or two ago, Arch Linux migrated the tests for dbscripts (the server-side of how package releases happen) from shUnit2 to BATS. https://git.archlinux.org/dbscripts.git/ https://git.archlinux.org/dbscripts.git/
  - orestes910 8y ago
    That is actually a really solid point. I still don't know that it's necessarily the right way, but I'd certainly rather those scripts be tested!
  - chubot 8y ago
    I would consider most GNU/Linux distros to be giant piles of shell scripts. Yes! That was one of the primary motivations for my Oil project [1]. I was building containers from scratch with shell scripts (in 2012 or so, pre-Docker), and I was horrified when I discovered how Debian actually works. Why Create a New Unix Shell? http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html And of course it's not just Debian. Red Hat, Fedora, Alpine, etc. are all big packages of shell scripts, often mixed with other ad hoc macro processing or Makefiles. Alpine does this funny hack where their metadata is in APKBUILD shell scripts, which is limiting when you want to read metadata without executing shell. I also point out in that post that Kubernetes is pretty new (2014) and it has 48,000 lines of shell in its repo. That's true of most cloud infrastructure. If you deal with Heroku, OpenStack, Cloud Foundry, etc. there is a ton of shell all over the place. With buildpacks, Travis CI, etc. And that's obviously not because the authors of those projects don't know what they're doing. Shell is still the best tool for that job (bringing up Unix systems), despite all its flaws. The world now runs on clusters of Unix machines, and in turn big piles of shell scripts :) ----- New release here if anyone wants to help me test: OSH 0.6.pre15 http://www.oilshell.org/blog/2019/02/18.html http://www.oilshell.org/blog/2019/02/18.html Caveat: it's still too slow; I'm mainly looking for people to help test it. Running bats tests with it would be great. I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax. OSH also opens a lot of new opportunities for people interested in testing / static analysis and shell. There was some discussion here, even related to bats, but I'm not sure where it went: https://github.com/oilshell/oil/issues/200 https://github.com/oilshell/oil/issues/200 [1] The other being how hard it is to write an autocompletion script!
    - tomeon 8y ago
      > I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax. Bats runs the test files through a preprocessor[1] that, among other things, converts each `@test` case into a shell function. E.g. `@test "the doohickey should frob" { ... }` becomes `test_the_doohickey_should_frob() { ... }`. [1] https://github.com/bats-core/bats-core/blob/master/libexec/bats-core/bats-preprocess https://github.com/bats-core/bats-core/blob/master/libexec/b...
- jasonpeacock 8y ago
  Bash is proper programming language, though not necessarily modern nor ergonomic. The problem is: 1. Bash is assumed to be everywhere, so people use it for maximum portability or bootstrapping. 2. It started as a 10-100 line "quick" script, but then grew into a monster and nobody wanted to take the hit to rewrite it in a modern programming language. I've found when you enforce the same software best practice requirements regardless of language, people start choosing not-Bash since "they have to do it right anyway". Many devs see Bash as a shortcut to avoiding the extra work.
jasonpeacock 8y ago
Yes! So much yes. Many developers write crappy Bash scripts because they don't see it as a real language, but it's an interpreted language just like Ruby, Python, Perl, etc. If you're going to design modular libraries and scripts with best practices, unit tests, etc, for those languages, then you should be doing the same with Bash. If you're writing production scripts using Bash, you should be following software best practices - don't give me any "it's just Bash" excuses. Mocking In addition to bats-mock (part of the bats library in the article), there is another mocking approach: https://pbrisbin.com/posts/mocking_bash/ https://pbrisbin.com/posts/mocking_bash/ They both have their uses. Documentation NaturalDocs ( https://www.naturaldocs.org/ https://www.naturaldocs.org/ ) can happily parse & extract doc comments from your bash scripts & libraries: https://www.naturaldocs.org/ https://www.naturaldocs.org/ Just define your own language: https://www.naturaldocs.org/reference/languages.txt/#adding_languages https://www.naturaldocs.org/reference/languages.txt/#adding_... Format: 1.4 Language: bash Shebang Strings: bash Extensions: bash sh Ignore Extensions: bats Line Comments: # Linting Use shellcheck, it's amazing: https://www.shellcheck.net/ https://www.shellcheck.net/ If you use Vim, w0rp/ale knows how to integrate with it for real-time feedback: https://github.com/w0rp/ale/tree/master/ale_linters/sh https://github.com/w0rp/ale/tree/master/ale_linters/sh Argument Parsing Use Docopt ( http://docopt.org/ http://docopt.org/ ) to get modern arg-parsing in your Bash scripts. Don't waste your time trying to roll your own or use the built-in anemic arg-parsing. Shells implementation: https://github.com/docopt/docopts https://github.com/docopt/docopts Use Bash built-ins Stop invoking external processes for features already available in Bash: https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible More good resources: * http://www.tldp.org/LDP/abs/html/ http://www.tldp.org/LDP/abs/html/ * https://www.gnu.org/software/bash/manual/bashref.html https://www.gnu.org/software/bash/manual/bashref.html * http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide BUT why are you using Bash? Go use a modern language and don't suffer the idiosyncrasies of Bash where you have to re-invent many common libraries already available like JSON parsers, etc. My favorite Bash replacement is Python + Plumbum: https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ All the convenience of Bash when invoking commands (no more subprocess()!) with the joy of Python.
- nomel 8y ago
fxfan 8y ago
Im probably in minority but I use powershell on my arch and love it. Finding duplicates has never been so easy- `Get-childitem -Recurse |% {[System.Tuple]::Create( $_.FullName, (md5sum $_.FullName).split( " " )[ 0 ] ) }) | Group-Object Item2 | Where-Object { $_.Count -ge 1}`
- samebreath 8y ago
  You win for enthusiasm when it comes to that syntax. I appreciate I pipe output to uniq (MacOS) to get uniques, proving that easy is an extremely relative value :)
  - fxfan 8y ago
    Use powershell ;) It's wonderful.
zenlikethat 8y ago
My experience doing integration testing with BATS was that it was complete misery (not the BATS project’s fault, I think it’s just inherent in the nature of the thing). You REALLY want a real programming language for any kind of non trivial integration testing. Things like set up and tear down, checking outputs, and testing many-similar-but-not-identical cases gets hairy fast. Sure you CAN do it in bash but just because you can doesn't mean you should. When we rewrote our BATS tests in Go was a huge step forward for the project.
- dnautics 8y ago
  One time I had to write a Json marshaller in bash (don't ask), and bats was invaluable. BATS is great when you need to unit test bash. And for the most basic integration tests on something that is meant to be a CLI.
faitswulff 8y ago
Tangential question, but are there any efforts to treat shell scripts as a compilation target? It seems like such an insane language, I don't know why another nicer language hasn't emerged on top of it.
- codetrotter 8y ago
  > It seems like such an insane language I wouldn’t say insane, just limited. But the main point of the shell and of scripting the shell is to execute programs and creating pipelines. If you need to do complex things, or you need special data structures or you need to process the output directly yourself in any significant way, then shell scripting is simply not the right tool for the job. And in the case where you want to use another language why do you want to transpile to shell script? Better to just install Python 3 or something on the server. (It might even be installed there by default already.)
- freedomben 8y ago
  > It seems like such an insane language I can understand that because I thought that way too before I really learned it. Bash is a language like any other: it takes reading and practice to get good at it. Also most of the WTFs in Bash actually make decent sense when you consider that whitespace is the delimiter in bash (for example setting a variable is VAR='value' instead of VAR = 'value'. That trips newer people up pretty badly at first). Most people also abandon good software practices when they write bash for some reason. For example, you'd never write 100 lines of ruby or javascript without declaring functions, yet people do that all the time in bash (they shouldn't). If people followed good practice, I think bash would seem a lot less insane.
  - nicoburns 8y ago
    I think lack of proper arrays and structured data types is a big issue with bash. Basically everything is a string, with all the issues around escaping and corner cases that that implies
    - freedomben 8y ago

51 points by samebreath 8y ago

Outside of the scope of elaborate CI pipelines, I wonder how useful this can really be. Big CI pipelines are one of the few instances where I can think of Bash being both an appropriate choice AND the resulting product being large, elaborate, and sensitive to failure - which would benefit from being tested. Most other applications of Bash are generally just so simple that fundamentally altering how you write scripts ("Bash scripts must also be broken down into multiple functions, which the main part of the script should call when the script is executed.") for the sake of testing them seems like it could easily fall into the category of over engineering. Beyond the CI pipeline use case, wouldn't the tools in which this would actually be properly useful be better off written in a proper programming language?

npongratz 8y ago
I've found shell scripting to be superior to other programming languages when the job calls for performing some combination of the following: 1) calling lots of commands 2) using lots of pipelines 3) handling exit codes from 1 and 2 for error handling and conditional execution control 4) doing job control for processes and process groups (backgrounding, pausing, foregrounding, sending signals, etc.)
- jasonpeacock 8y ago
  Have you looked at Plumbumm for Python? It handles everything you just mentioned: https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ There's no reason not to use Python to achieve the same thing, and you get all the benefits of using Python instead of Bash. I'm sure other languages have similar libraries.
LukeShu 8y ago
Whenever someone says "most people don't write large Bash scripts", I have to chime in and say that I would consider most GNU/Linux distros to be giant piles of shell scripts. A year or two ago, Arch Linux migrated the tests for dbscripts (the server-side of how package releases happen) from shUnit2 to BATS. https://git.archlinux.org/dbscripts.git/ https://git.archlinux.org/dbscripts.git/
- orestes910 8y ago
  That is actually a really solid point. I still don't know that it's necessarily the right way, but I'd certainly rather those scripts be tested!
- chubot 8y ago
  I would consider most GNU/Linux distros to be giant piles of shell scripts. Yes! That was one of the primary motivations for my Oil project [1]. I was building containers from scratch with shell scripts (in 2012 or so, pre-Docker), and I was horrified when I discovered how Debian actually works. Why Create a New Unix Shell? http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html And of course it's not just Debian. Red Hat, Fedora, Alpine, etc. are all big packages of shell scripts, often mixed with other ad hoc macro processing or Makefiles. Alpine does this funny hack where their metadata is in APKBUILD shell scripts, which is limiting when you want to read metadata without executing shell. I also point out in that post that Kubernetes is pretty new (2014) and it has 48,000 lines of shell in its repo. That's true of most cloud infrastructure. If you deal with Heroku, OpenStack, Cloud Foundry, etc. there is a ton of shell all over the place. With buildpacks, Travis CI, etc. And that's obviously not because the authors of those projects don't know what they're doing. Shell is still the best tool for that job (bringing up Unix systems), despite all its flaws. The world now runs on clusters of Unix machines, and in turn big piles of shell scripts :) ----- New release here if anyone wants to help me test: OSH 0.6.pre15 http://www.oilshell.org/blog/2019/02/18.html http://www.oilshell.org/blog/2019/02/18.html Caveat: it's still too slow; I'm mainly looking for people to help test it. Running bats tests with it would be great. I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax. OSH also opens a lot of new opportunities for people interested in testing / static analysis and shell. There was some discussion here, even related to bats, but I'm not sure where it went: https://github.com/oilshell/oil/issues/200 https://github.com/oilshell/oil/issues/200 [1] The other being how hard it is to write an autocompletion script!
  - tomeon 8y ago
    > I don't know how bats works with the @test annotation? That doesn't look like valid shell syntax. Bats runs the test files through a preprocessor[1] that, among other things, converts each `@test` case into a shell function. E.g. `@test "the doohickey should frob" { ... }` becomes `test_the_doohickey_should_frob() { ... }`. [1] https://github.com/bats-core/bats-core/blob/master/libexec/bats-core/bats-preprocess https://github.com/bats-core/bats-core/blob/master/libexec/b...
jasonpeacock 8y ago
Bash is proper programming language, though not necessarily modern nor ergonomic. The problem is: 1. Bash is assumed to be everywhere, so people use it for maximum portability or bootstrapping. 2. It started as a 10-100 line "quick" script, but then grew into a monster and nobody wanted to take the hit to rewrite it in a modern programming language. I've found when you enforce the same software best practice requirements regardless of language, people start choosing not-Bash since "they have to do it right anyway". Many devs see Bash as a shortcut to avoiding the extra work.

jasonpeacock 8y ago

Yes! So much yes. Many developers write crappy Bash scripts because they don't see it as a real language, but it's an interpreted language just like Ruby, Python, Perl, etc. If you're going to design modular libraries and scripts with best practices, unit tests, etc, for those languages, then you should be doing the same with Bash. If you're writing production scripts using Bash, you should be following software best practices - don't give me any "it's just Bash" excuses. Mocking In addition to bats-mock (part of the bats library in the article), there is another mocking approach: https://pbrisbin.com/posts/mocking_bash/ https://pbrisbin.com/posts/mocking_bash/ They both have their uses. Documentation NaturalDocs ( https://www.naturaldocs.org/ https://www.naturaldocs.org/ ) can happily parse & extract doc comments from your bash scripts & libraries: https://www.naturaldocs.org/ https://www.naturaldocs.org/ Just define your own language: https://www.naturaldocs.org/reference/languages.txt/#adding_languages https://www.naturaldocs.org/reference/languages.txt/#adding_... Format: 1.4 Language: bash Shebang Strings: bash Extensions: bash sh Ignore Extensions: bats Line Comments: # Linting Use shellcheck, it's amazing: https://www.shellcheck.net/ https://www.shellcheck.net/ If you use Vim, w0rp/ale knows how to integrate with it for real-time feedback: https://github.com/w0rp/ale/tree/master/ale_linters/sh https://github.com/w0rp/ale/tree/master/ale_linters/sh Argument Parsing Use Docopt ( http://docopt.org/ http://docopt.org/ ) to get modern arg-parsing in your Bash scripts. Don't waste your time trying to roll your own or use the built-in anemic arg-parsing. Shells implementation: https://github.com/docopt/docopts https://github.com/docopt/docopts Use Bash built-ins Stop invoking external processes for features already available in Bash: https://github.com/dylanaraps/pure-bash-bible https://github.com/dylanaraps/pure-bash-bible More good resources: * http://www.tldp.org/LDP/abs/html/ http://www.tldp.org/LDP/abs/html/ * https://www.gnu.org/software/bash/manual/bashref.html https://www.gnu.org/software/bash/manual/bashref.html * http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide BUT why are you using Bash? Go use a modern language and don't suffer the idiosyncrasies of Bash where you have to re-invent many common libraries already available like JSON parsers, etc. My favorite Bash replacement is Python + Plumbum: https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ All the convenience of Bash when invoking commands (no more subprocess()!) with the joy of Python.

nomel 8y ago

fxfan 8y ago

Im probably in minority but I use powershell on my arch and love it. Finding duplicates has never been so easy- `Get-childitem -Recurse |% {[System.Tuple]::Create( $_.FullName, (md5sum $_.FullName).split( " " )[ 0 ] ) }) | Group-Object Item2 | Where-Object { $_.Count -ge 1}`

samebreath 8y ago
You win for enthusiasm when it comes to that syntax. I appreciate I pipe output to uniq (MacOS) to get uniques, proving that easy is an extremely relative value :)
- fxfan 8y ago
  Use powershell ;) It's wonderful.

zenlikethat 8y ago

My experience doing integration testing with BATS was that it was complete misery (not the BATS project’s fault, I think it’s just inherent in the nature of the thing). You REALLY want a real programming language for any kind of non trivial integration testing. Things like set up and tear down, checking outputs, and testing many-similar-but-not-identical cases gets hairy fast. Sure you CAN do it in bash but just because you can doesn't mean you should. When we rewrote our BATS tests in Go was a huge step forward for the project.

dnautics 8y ago
One time I had to write a Json marshaller in bash (don't ask), and bats was invaluable. BATS is great when you need to unit test bash. And for the most basic integration tests on something that is meant to be a CLI.

faitswulff 8y ago

Tangential question, but are there any efforts to treat shell scripts as a compilation target? It seems like such an insane language, I don't know why another nicer language hasn't emerged on top of it.

codetrotter 8y ago
> It seems like such an insane language I wouldn’t say insane, just limited. But the main point of the shell and of scripting the shell is to execute programs and creating pipelines. If you need to do complex things, or you need special data structures or you need to process the output directly yourself in any significant way, then shell scripting is simply not the right tool for the job. And in the case where you want to use another language why do you want to transpile to shell script? Better to just install Python 3 or something on the server. (It might even be installed there by default already.)
freedomben 8y ago
> It seems like such an insane language I can understand that because I thought that way too before I really learned it. Bash is a language like any other: it takes reading and practice to get good at it. Also most of the WTFs in Bash actually make decent sense when you consider that whitespace is the delimiter in bash (for example setting a variable is VAR='value' instead of VAR = 'value'. That trips newer people up pretty badly at first). Most people also abandon good software practices when they write bash for some reason. For example, you'd never write 100 lines of ruby or javascript without declaring functions, yet people do that all the time in bash (they shouldn't). If people followed good practice, I think bash would seem a lot less insane.
- nicoburns 8y ago
  I think lack of proper arrays and structured data types is a big issue with bash. Basically everything is a string, with all the issues around escaping and corner cases that that implies
  - freedomben 8y ago