116 ms·

Shell scripting gets a lot of hate, but after spending time to grok the language (and copious use of shellcheck), I've grown quite fond of it. This also has the

Shell scripting gets a lot of hate, but after spending time to grok the language (and copious use of shellcheck), I've grown quite fond of it. This also has the nice side effect of emphasizing use of old unix tools. I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays in POSIX shell for me. Stdin/stdout are your friend. Certainly, shell scripting isn't the answer for everything, but with shellcheck (and bats!), I feel like it's a really reasonable default for systemy things. Heck, the status bar I use in my window manager is essentially just a 700 line bash script, with asyncronous update and everything. It's even quite readable and orthogonally organized, if I might boast. Have been pleasantly surprised over the years every time I go back and read it.

thundergolfer 8y ago
Haven't tried BATS before, but it sounds like the missing part of our bash codebase, which I am currently mostly skeptical of because it isn't unit tested. I'd fear though that having BATS would just increase the tendency to let a bash script grow from 50 lines through to 200, when it should've just been ported to Python at 100.
- mstade 8y ago
  Actually, my experience with bats is that it forces you to modularize and keep things small. Easier to test things that way. Bats is truly an indispensable tool when doing shell scripting – I even use it to test CLIs built in other languages too!
yakshaving_jgt 8y ago
If that was the big realisation, you may enjoy FP.
hyperpallium 8y ago
> [...] the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays [...] Could you elaborate please? I see that data - especially sequential, array-like data - is conveniently passed via pipes (and e.g. jq https://stedolan.github.io/jq/ https://stedolan.github.io/jq/ uses this concept), but I feel you're saying something deeper or a different kind of use-case... [perhaps because you say "state" not "data")
- glic3rinu 8y ago
  By "state" I think OP is referring to an style of programming heavily based on global variables (global state). Function calls set global variables that the caller can read after. Functions and imports (sourcing) are all riddle with "side-effects", making the execution of the program very difficult to follow (spaghetti code). Because "return values" in bash are implemented by echoing (printing to stdout): return=$(my_function arg1) I believe OP has the point that echoing friendly-parsable output is a way of implementing "data-structures" in shells. result=$(my_function arg1) result_a=$(echo "$result" | grep A) result_b=$(echo "$result" | grep B)
- xelxebar 8y ago
  The sibling comment by @glic3rinu pretty much says it, but I'll elaborate a little bit. Can't make any claims about being anything deep, but in a sense, my shift in perspective came from treating program state as "data in pipes" and doing that as extensively as possible. A big part of this was figuring out I could the `read` builtin to parse stuff into local variables: if read -r foo bar baz _; do # do stuff with $foo, $bar, $baz # $_ contains anything extra on the line done Also, once everything is on stdin, all the standard unix tools Just Work without having to echo and print everywhere, e.g. something like echo $some_local variable | unix_tool For me, this turned a lot ugly scripts into primarily a collection of functions that compose nicely via pipelines. Anyway, using `read` works especially nicely with tabular data too. For example, a naive CSV parser might look like while IFS=',' read -r field1 field2 field3 _; do # do stuff done
  - h1d 8y ago
    while IFS=',' read -r field1 field2 field3 _; do # do stuff done This is the reason we need something better than bash than stick with it until it's been used for 50 years. It's just too cryptic. Fish is better but still looks legacy.
  - gubbrora 8y ago
    > Fuck quoting/escaping, it will work 99% of the time. Encouraging this is why bash scripts scare me.
mstade 8y ago
Bats, shellcheck and jq has completely transformed the shell scripting experience for me. To the point where I’ll reach for bash much more often than other languages these days, for anything terminal based.
- SOLAR_FIELDS 8y ago
  jq also has some really nice quick interactive features. My favorite super simple one is that jq beautifies JSON by very virtue of just piping it into jq with no additional arguments.
arendtio 8y ago
I think one of the biggest problems is the inconsistent runtime environment. I mean, yes, you can just say fuck POSIX and code for Bash only, but then again, not even Bash is consistent across its versions. Just to give an example: The infamous `set -e` behaves quite differently across different shells https://www.in-ulm.de/~mascheck/various/set-e/ https://www.in-ulm.de/~mascheck/various/set-e/ And those are just POSIX compliant shells. I mean this doesn't matter if you just want to write a script for yourself but becomes frustrating when you want to level up you shell skills and write cross-platform scripts ;-) That said, I have to add that I love shell scripting... I am not exactly sure why, but I guess it is because it lets you build powerful things with very few lines of code.
- mekazu 8y ago
  I agree, and I like having those ‘oh wow, I can do that in bash? That’s dangerous, but awesome’ moments just by researching how to get stuff done. Really people don’t know what they’re getting themselves into when they start down the shell scripting path.
- zimbatm 8y ago
  and to make things worse, macOS ships with a 12 years old version of Bash.
  - h1d 8y ago
    Use Homebrew version. https://formulae.brew.sh/formula/bash https://formulae.brew.sh/formula/bash
  - arendtio 8y ago
    Indeed, I have been burned by that... Wrote a script which worked perfectly fine with Linux. Tried it with MacOS... Oh gosh, it was such a pain work around those old bugs... Does someone know why Apple doesn't update the bash?
    - jen20 8y ago
      I believe Apple ship the last version of bash released under GPLv2, and do not ship versions under GPLv3.
DyslexicAtheist 8y ago
> I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. ever since I started dabbling in racket I've grown fond of not dealing with state and look at evreything like a function. (I'm not a functional programmer). The old UNIX philosophy with pipes is very close to that as in every command in the pipeline is a function in similar way. I'm not saying hacking together a bash script is FP but it's still possible to apply the FP thinking in the UNIX pipeline better than building a monolith that handles the whole pipe. I can pass state around (env vars) but it's an extra effort and it forces me to think whether I can get it done without state first (via optimizing the input output formats).
oaiey 8y ago
Pipeline thinking: This is the pipe-and-filter architectural style. Well documented in the industry. For pipe-and-filter system management use case, I tend to think that Powershell is the peak of the shell evolution currently, but that is probably my limited insight. When I learnt shell scripting, zsh was the best available :) but I preferred the ksh.
mhd 8y ago
The problem is that "shell scripting" is "bash scripting", and the focus of that particular outgrowth of the original Bourne shell isn't really focused on that. It was mostly about interactive improvements, and while a few of the bits of added functionality help (like not needing to shell out for every bit of string manipulation), it doesn't contribute a lot to writing longer scripts. Zsh just doubled down on the interactive use. And let's never mention csh again. Now there was one offshoot that tried to get better at longer scripts: The Korn Shell. In it's later edition (ksh93) it actually had hashes and a pretty good way of expanding its functionality. Heck, dtksh would've given Tcl/Tk a run for its money in the 90s if both ksh and Motif (the library dtksh used for its widgets) weren't proprietary at the time. I worked on a "devops" system in the early '00s that was used to provision all kinds of Unix servers and workstations (including HP-UX, AIX, Solaris, Linux, BSD..), and the major parts of it was written in ksh93. It worked quite well. I still think that all of this is inferior to both Perl5 and Tcl/Tk, but ksh93 made some decent strides way back when.
- exelius 8y ago
  I have a special place in my heart for Perl5, but I would never use it for anything DevOps related... Perl’s motto is “there’s always another way to write it” — and that’s the exact opposite of what you want in a DevOps environment :) Oh yeah, and Motif can die in a fire.
  - mhd 8y ago
    Sorry to say, but I find the criticism in this form to be quite meaningless. What does a motto have to do with any kind of validity for a certain purpose? Python has the "Zen of Python", does this inherently make it a better solution? "There are many ways to skin a cat" is older than computers, and it applies to basically any programming language. Some languages make it harder to do it in a different manner, some allow it, but don't offer it without including some additional package. C has no hash tables, but it's not that hard to get them, for example. I'd go so far as to say, that plenty of current programming languages allow more ways to do a thing out of the box than Perl. What mythical language does offer a straight and completely obvious path from problem to solution? Haven't I ascended enough in the halls of programmer-dom to be told of this coding panacea? And why would this be particularly bad for DevOps? Are DevOps programmers worse and/or more easily confused? Just recently we had a thread here about different ways to remove duplicates from arrays, and that was done in Golang, devops Holy Grail du jour.
xorcist 8y ago
Exactly this. Every time to reach for an array in bash, try a pipe instead. Every time you desire multidimensional arrays, pipe through filters. (At least that's what I think you're saying.)
h1d 8y ago
After writing a decent backup shell script in fish, I converted it to bash as IntelliJ didn't have support for fish and also non strict Posix compatibility wasn't too comfortable and then realized it's quite a pain to write long script in bash knowing modern languages are far easier to write with, I may just go back to using perl for shell scripting. All of the later scripting languages aren't really meant for writing shell script as invoking commands is just tedious instead of just using back ticks in perl. Talk about going in circles of history.
marcosdumay 8y ago
Shell (Bash) scripting have a few huge problems, mostly with error handling and number arithmetic. The arithmetic one isn't a large problem for its niche, but the error handling is so large a bother that it's best avoided for anything complex, even if something else (AKA, Python) will lead to a much larger code. That said, there are niches where bad error handling isn't as much problematic. Ironically, Bash is much more fit for web programming than for system scripting.
neop1x 8y ago
It is very inconvenient, ineffective and slow. For example lists are slow as hell. And the syntax is kind of workaround - look at '[' executable for example. It definitely has its place in CS history and it made things possible in the past but using it to anything more complex nowadays is a bad, bad, ba idea! I am glad people prefer Python or Ruby for more complicated stuff and Go, Java or compiled langages like Rust, C/C++ for performance-intensive jobs. Perl is fast but the syntax is terribly non-intuitive and I would mark it 'do not use for new designs'.

thundergolfer 8y ago

Haven't tried BATS before, but it sounds like the missing part of our bash codebase, which I am currently mostly skeptical of because it isn't unit tested. I'd fear though that having BATS would just increase the tendency to let a bash script grow from 50 lines through to 200, when it should've just been ported to Python at 100.

mstade 8y ago
Actually, my experience with bats is that it forces you to modularize and keep things small. Easier to test things that way. Bats is truly an indispensable tool when doing shell scripting – I even use it to test CLIs built in other languages too!

yakshaving_jgt 8y ago

If that was the big realisation, you may enjoy FP.

hyperpallium 8y ago

> [...] the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. That completely removed the pain of no arrays [...] Could you elaborate please? I see that data - especially sequential, array-like data - is conveniently passed via pipes (and e.g. jq https://stedolan.github.io/jq/ https://stedolan.github.io/jq/ uses this concept), but I feel you're saying something deeper or a different kind of use-case... [perhaps because you say "state" not "data")

glic3rinu 8y ago
By "state" I think OP is referring to an style of programming heavily based on global variables (global state). Function calls set global variables that the caller can read after. Functions and imports (sourcing) are all riddle with "side-effects", making the execution of the program very difficult to follow (spaghetti code). Because "return values" in bash are implemented by echoing (printing to stdout): return=$(my_function arg1) I believe OP has the point that echoing friendly-parsable output is a way of implementing "data-structures" in shells. result=$(my_function arg1) result_a=$(echo "$result" | grep A) result_b=$(echo "$result" | grep B)
xelxebar 8y ago
The sibling comment by @glic3rinu pretty much says it, but I'll elaborate a little bit. Can't make any claims about being anything deep, but in a sense, my shift in perspective came from treating program state as "data in pipes" and doing that as extensively as possible. A big part of this was figuring out I could the `read` builtin to parse stuff into local variables: if read -r foo bar baz _; do # do stuff with $foo, $bar, $baz # $_ contains anything extra on the line done Also, once everything is on stdin, all the standard unix tools Just Work without having to echo and print everywhere, e.g. something like echo $some_local variable | unix_tool For me, this turned a lot ugly scripts into primarily a collection of functions that compose nicely via pipelines. Anyway, using `read` works especially nicely with tabular data too. For example, a naive CSV parser might look like while IFS=',' read -r field1 field2 field3 _; do # do stuff done
- h1d 8y ago
  while IFS=',' read -r field1 field2 field3 _; do # do stuff done This is the reason we need something better than bash than stick with it until it's been used for 50 years. It's just too cryptic. Fish is better but still looks legacy.
- gubbrora 8y ago
  > Fuck quoting/escaping, it will work 99% of the time. Encouraging this is why bash scripts scare me.

mstade 8y ago

Bats, shellcheck and jq has completely transformed the shell scripting experience for me. To the point where I’ll reach for bash much more often than other languages these days, for anything terminal based.

SOLAR_FIELDS 8y ago
jq also has some really nice quick interactive features. My favorite super simple one is that jq beautifies JSON by very virtue of just piping it into jq with no additional arguments.

arendtio 8y ago

I think one of the biggest problems is the inconsistent runtime environment. I mean, yes, you can just say fuck POSIX and code for Bash only, but then again, not even Bash is consistent across its versions. Just to give an example: The infamous `set -e` behaves quite differently across different shells https://www.in-ulm.de/~mascheck/various/set-e/ https://www.in-ulm.de/~mascheck/various/set-e/ And those are just POSIX compliant shells. I mean this doesn't matter if you just want to write a script for yourself but becomes frustrating when you want to level up you shell skills and write cross-platform scripts ;-) That said, I have to add that I love shell scripting... I am not exactly sure why, but I guess it is because it lets you build powerful things with very few lines of code.

mekazu 8y ago
I agree, and I like having those ‘oh wow, I can do that in bash? That’s dangerous, but awesome’ moments just by researching how to get stuff done. Really people don’t know what they’re getting themselves into when they start down the shell scripting path.
zimbatm 8y ago
and to make things worse, macOS ships with a 12 years old version of Bash.
- h1d 8y ago
  Use Homebrew version. https://formulae.brew.sh/formula/bash https://formulae.brew.sh/formula/bash
- arendtio 8y ago
  Indeed, I have been burned by that... Wrote a script which worked perfectly fine with Linux. Tried it with MacOS... Oh gosh, it was such a pain work around those old bugs... Does someone know why Apple doesn't update the bash?
  - jen20 8y ago
    I believe Apple ship the last version of bash released under GPLv2, and do not ship versions under GPLv3.

DyslexicAtheist 8y ago

> I think the "big realization" for me was shifting to think in pipelines instead of passing state around in variables. ever since I started dabbling in racket I've grown fond of not dealing with state and look at evreything like a function. (I'm not a functional programmer). The old UNIX philosophy with pipes is very close to that as in every command in the pipeline is a function in similar way. I'm not saying hacking together a bash script is FP but it's still possible to apply the FP thinking in the UNIX pipeline better than building a monolith that handles the whole pipe. I can pass state around (env vars) but it's an extra effort and it forces me to think whether I can get it done without state first (via optimizing the input output formats).

oaiey 8y ago

Pipeline thinking: This is the pipe-and-filter architectural style. Well documented in the industry. For pipe-and-filter system management use case, I tend to think that Powershell is the peak of the shell evolution currently, but that is probably my limited insight. When I learnt shell scripting, zsh was the best available :) but I preferred the ksh.

mhd 8y ago

The problem is that "shell scripting" is "bash scripting", and the focus of that particular outgrowth of the original Bourne shell isn't really focused on that. It was mostly about interactive improvements, and while a few of the bits of added functionality help (like not needing to shell out for every bit of string manipulation), it doesn't contribute a lot to writing longer scripts. Zsh just doubled down on the interactive use. And let's never mention csh again. Now there was one offshoot that tried to get better at longer scripts: The Korn Shell. In it's later edition (ksh93) it actually had hashes and a pretty good way of expanding its functionality. Heck, dtksh would've given Tcl/Tk a run for its money in the 90s if both ksh and Motif (the library dtksh used for its widgets) weren't proprietary at the time. I worked on a "devops" system in the early '00s that was used to provision all kinds of Unix servers and workstations (including HP-UX, AIX, Solaris, Linux, BSD..), and the major parts of it was written in ksh93. It worked quite well. I still think that all of this is inferior to both Perl5 and Tcl/Tk, but ksh93 made some decent strides way back when.

exelius 8y ago
I have a special place in my heart for Perl5, but I would never use it for anything DevOps related... Perl’s motto is “there’s always another way to write it” — and that’s the exact opposite of what you want in a DevOps environment :) Oh yeah, and Motif can die in a fire.
- mhd 8y ago
  Sorry to say, but I find the criticism in this form to be quite meaningless. What does a motto have to do with any kind of validity for a certain purpose? Python has the "Zen of Python", does this inherently make it a better solution? "There are many ways to skin a cat" is older than computers, and it applies to basically any programming language. Some languages make it harder to do it in a different manner, some allow it, but don't offer it without including some additional package. C has no hash tables, but it's not that hard to get them, for example. I'd go so far as to say, that plenty of current programming languages allow more ways to do a thing out of the box than Perl. What mythical language does offer a straight and completely obvious path from problem to solution? Haven't I ascended enough in the halls of programmer-dom to be told of this coding panacea? And why would this be particularly bad for DevOps? Are DevOps programmers worse and/or more easily confused? Just recently we had a thread here about different ways to remove duplicates from arrays, and that was done in Golang, devops Holy Grail du jour.

xorcist 8y ago

Exactly this. Every time to reach for an array in bash, try a pipe instead. Every time you desire multidimensional arrays, pipe through filters. (At least that's what I think you're saying.)

h1d 8y ago

After writing a decent backup shell script in fish, I converted it to bash as IntelliJ didn't have support for fish and also non strict Posix compatibility wasn't too comfortable and then realized it's quite a pain to write long script in bash knowing modern languages are far easier to write with, I may just go back to using perl for shell scripting. All of the later scripting languages aren't really meant for writing shell script as invoking commands is just tedious instead of just using back ticks in perl. Talk about going in circles of history.

marcosdumay 8y ago

Shell (Bash) scripting have a few huge problems, mostly with error handling and number arithmetic. The arithmetic one isn't a large problem for its niche, but the error handling is so large a bother that it's best avoided for anything complex, even if something else (AKA, Python) will lead to a much larger code. That said, there are niches where bad error handling isn't as much problematic. Ironically, Bash is much more fit for web programming than for system scripting.

neop1x 8y ago

It is very inconvenient, ineffective and slow. For example lists are slow as hell. And the syntax is kind of workaround - look at '[' executable for example. It definitely has its place in CS history and it made things possible in the past but using it to anything more complex nowadays is a bad, bad, ba idea! I am glad people prefer Python or Ruby for more complicated stuff and Go, Java or compiled langages like Rust, C/C++ for performance-intensive jobs. Perl is fast but the syntax is terribly non-intuitive and I would mark it 'do not use for new designs'.