6 ms·

My rule is that shell scripts with growing complexity must be converted to a language with clearer and more bulletproof treatment of data, like Python. This al

My rule is that shell scripts with growing complexity must be converted to a language with clearer and more bulletproof treatment of data, like Python. This also happens any time I want to support several command line arguments. At one point in history, options were more limited. Now, it is just not worth the time to read code that may or may not have obscure errors in its very expressions that you must understand before you can even begin to examine the real purpose of the program!

mbrock 9y ago
You still need to know how to write safe shell programs even if they're one-liners. And once you do, shell is really, really useful and convenient. For example, Linux distributions are full of shell scripts that would be frankly awful to write in any other language that I know of... like this, to take a random example: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/virtualization/docker/default.nix#L99 https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
- Jabbles 9y ago
  Shouldn't they be quoted?
- tedmiston 9y ago
  Most app developers aren't writing the scripts in Linux distros though.
- deleted 9y ago
  [deleted]
- masklinn 9y ago
  Eh. Looks OK to me, even with Python probably not being the tersest shell language: https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb88ecb5e https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb8...
  - mbrock 9y ago
    Hmm, I admit that's not too bad. I still disagree with the ideology that any non-trivial script needs to be rewritten in Python. The shell is the standard way of interacting with a *nix system, and for me that's the major benefit of using shell for "system"-ish tasks. I already know how to make a symlink, a pipeline, a chmod, etc, so I don't need to learn a new syntax for that.
  - Too 9y ago
    The shell script probably also behaves strange if any of $out ${docker-runc} $version, $man, etc contains a space, while the python-script most likely wont.
    - mbrock 9y ago
      Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables. Of course, as a developer you have to understand this. The semantics of shell string interpolation are generally not that hard to learn, compared to learning all the nuances of programming in general that can make your program incorrect, so I find it a bit strange how it's considered to make shell completely useless for scripting. Shell scripting has been working pretty well since, like 1975! It's not that bad!
      - zokier 9y ago
        > Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables Which works as long as attacker (or sleepy admin) doesn't happen to create suitably "bad" path > Shell scripting has been working pretty well since, like 1975! It's not that bad! Considering the amount of borked system because spaces/uninitialized variables -stories I've heard over years, I'm not so sure about how well it has been working...
        mbrock 9y ago
        Attackers who have commit access to your repository can do a lot of bad things quite easily.
- developer2 9y ago
  >> need to know how to write safe shell programs >> once you do, shell is really, really useful and convenient Part of the issue here is that a large number of people who think they know how to write safe shell scripts do not in fact have the ability to do so. There are so many gotchas to learn that it's all too easy to believe oneself competent, while being unaware of just how short one's knowledge falls. It's just too easy to write a script in the shell of your choice that works - until it encounters an edge case scenario that wasn't planned for, at which point the results can be catastrophic. This is true of any programming language, but the tasks for which people write shell scripts (such as massive system-wide filesystem manipulation) makes the consequences of a single-line bug very real.
  - ams6110 9y ago
    > This is true of any programming language Glad you mentioned that, because that was going to be my reply after reading the first half of your post. I'm not sure a person who is fairly well-versed in shell scripting is going to do a worse job than trying to use python, perl, lua, etc. if he doesn't also hsve a high level of experience and compentence in those languages. It has been literally 10 years since I touched Python. I use shell scripting every day. I know which one I'm going to be better at using to produce reliable code that does what I want it to do.
    - JoBrad 9y ago
      > It has been literally 10 years since I touched Python That's fine. Python 2.6 is still in wide use.
- TheAceOfHearts 9y ago
  Writing bug-free shell programs is hard because it doesn't provide safe defaults. By now I've gotten reasonably decent at shell scripts due to trial and error, and shellcheck... But if I'm doing anything fancy, I just opt for a node script using shelljs and any other third-party deps to cover my needs. This also has the benefit of typically working cross-platform. Although that's not as big of a concern for me, for some people it makes a big difference. For my day-to-day shell, I use fish. IMO, it has much better defaults than bash and zsh, and it's fast. I still have to use bash sometimes, but it's becoming increasingly rare. Your linked example doesn't really seem particularly tricky to me. I'd feel comfortable writing it in something like node or ruby, and I believe the result would probably be easier to consume for certain people.
  - mbrock 9y ago
    There are a lot of unsafe defaults in JavaScript too, most prominently async error propagation. Basically all languages have some tricky fatal flaws, even Haskell (lazy I/O can make very strange bugs, etc). I just don't think we should reject shell scripts entirely because some care is required to write them correctly. Shell is way too valuable.
    - TheAceOfHearts 9y ago
      You make a fair point. Although in more recent versions of JS the whole async error propagation isn't as big of an issue thanks to async functions and promises. I guess the biggest reason to use JS is that since I'm already using it every day, it's much easier to continue using a language I'm highly familiar with. Whenever I have to write a shell script I always have to pull up references and read through a couple Stack Overflow posts. I don't reject shell scripts, though. I agree that in some cases it can lead to much simpler code. The power and composability with pipes and immediate substitutions is incredibly powerful. I've had a few cases where I replaced far more complicated node scripts with simpler shell scripts. What I'd truly love is a reference guide that showed how to correctly do certain things with shell scripts. Here's a real-world example I struggled with: Writing end-to-end tests with selenium and a test runner. I wanted a script to start up a selenium server and wait for it to be ready. Then it should start the test runner. If the test runner exited successfully, kill the selenium server and browser, and exit with a 0. If the test runner or selenium server crashed or failed somehow, clean up and exit with a 1. Even after much trial and error, I still ended up with occasional Chrome copies staying open after failures.
arca_vorago 9y ago
Could you explain how writing, for example, text manipulation, would be safer in python than in shell? I understand formatting errors, shell escape failures, and the low hanging fruit, but in the end if I am just sed and awking my way around, I don't know how to do nearly the level of stuff I do in awk in python. I also will never forget the time I spent a day doing a perl script to parse some data... took some time, came back, and did it in a one line awk... Very powerful. I think somewhere in here is room for discussion of data, text, and the unix philosophy as everything as a file.
- makecheck 9y ago
  I said scripts with “growing complexity” should be expressed differently. I can and do use shell scripts for things that are relatively straightforward, and sed/awk-style transforms are a good example (although, even there, if too many quotes are involved you start to long for real escaping and things like triple-quotes).
- omaranto 9y ago
  A one-liner in Awk is almost certainly also a one-liner in Perl.
  - arca_vorago 9y ago
    I didn't say I wasn't doing the perl badly.
- highd 9y ago
  It's an issue of comprehensibility, not any current performance metric.
  - ams6110 9y ago
    Comprehensible to who? Other python programmers? Most sysadmins do not use python routinely. They work in the shell all day every day though.
    - 9y ago
mdekkers 9y ago
or write it in go and run it in a container!
lloeki 9y ago
Shellcheck is a boon to write more (by at least three orders of magnitude) reliable shell code. I wish there were a 'set -o strict' that would make bash consider all shellcheck warnings into errors as well as unilaterally set -euo pipefail. That said, bash is still my go to language for many many things because at the end of the day it's insanely efficient as that koan[0] is still valid when substituting C for mostly any other language. As an anecdote I've replaced ridiculously big and slow Rakefiles and unmaintainable ad hoc ruby code with much shorter, readable and reliable shell scripts. Using the shell doesn't mean you can't drop a call to python, perl, jq or whatever, and the composability and performance can be terrific when properly wielded. [0] http://www.catb.org/esr/writings/unix-koans/ten-thousand.html http://www.catb.org/esr/writings/unix-koans/ten-thousand.htm...
lobster_johnson 9y ago
I agree. Shell scripts are awesome until you reach a certain threshold of complexity, and that threshold is... low. Error handling is particularly problematic. If you know all the tricks (things like set -e, set -o pipefail, always capture stderr, etc.), it's not too bad, but there are still so many traps that the mental overhead of catching every single edge case is exhausting. One nasty issue with Bash is quoting, escaping and word splitting, because almost everything in Bash is about string manipulation. There are so many edge cases: $FOO is not the same as "$FOO", and sometimes you have no choice but to turn it into an array with eval FOO=($FOO) if you need mimic Bash's tokenization behaviour (e.g. respect backslashes and quoting). (Yes, Bash has arrays! A lot of scripts get simpler when you realize this.) These days I reach for Ruby very quickly even for small things.

mbrock 9y ago

You still need to know how to write safe shell programs even if they're one-liners. And once you do, shell is really, really useful and convenient. For example, Linux distributions are full of shell scripts that would be frankly awful to write in any other language that I know of... like this, to take a random example: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/virtualization/docker/default.nix#L99 https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...

Jabbles 9y ago
Shouldn't they be quoted?
tedmiston 9y ago
Most app developers aren't writing the scripts in Linux distros though.
deleted 9y ago
[deleted]
masklinn 9y ago
Eh. Looks OK to me, even with Python probably not being the tersest shell language: https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb88ecb5e https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb8...
- mbrock 9y ago
  Hmm, I admit that's not too bad. I still disagree with the ideology that any non-trivial script needs to be rewritten in Python. The shell is the standard way of interacting with a *nix system, and for me that's the major benefit of using shell for "system"-ish tasks. I already know how to make a symlink, a pipeline, a chmod, etc, so I don't need to learn a new syntax for that.
- Too 9y ago
  The shell script probably also behaves strange if any of $out ${docker-runc} $version, $man, etc contains a space, while the python-script most likely wont.
  - mbrock 9y ago
    Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables. Of course, as a developer you have to understand this. The semantics of shell string interpolation are generally not that hard to learn, compared to learning all the nuances of programming in general that can make your program incorrect, so I find it a bit strange how it's considered to make shell completely useless for scripting. Shell scripting has been working pretty well since, like 1975! It's not that bad!
    - zokier 9y ago
      > Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables Which works as long as attacker (or sleepy admin) doesn't happen to create suitably "bad" path > Shell scripting has been working pretty well since, like 1975! It's not that bad! Considering the amount of borked system because spaces/uninitialized variables -stories I've heard over years, I'm not so sure about how well it has been working...
      - mbrock 9y ago
        Attackers who have commit access to your repository can do a lot of bad things quite easily.
developer2 9y ago
>> need to know how to write safe shell programs >> once you do, shell is really, really useful and convenient Part of the issue here is that a large number of people who think they know how to write safe shell scripts do not in fact have the ability to do so. There are so many gotchas to learn that it's all too easy to believe oneself competent, while being unaware of just how short one's knowledge falls. It's just too easy to write a script in the shell of your choice that works - until it encounters an edge case scenario that wasn't planned for, at which point the results can be catastrophic. This is true of any programming language, but the tasks for which people write shell scripts (such as massive system-wide filesystem manipulation) makes the consequences of a single-line bug very real.
- ams6110 9y ago
  > This is true of any programming language Glad you mentioned that, because that was going to be my reply after reading the first half of your post. I'm not sure a person who is fairly well-versed in shell scripting is going to do a worse job than trying to use python, perl, lua, etc. if he doesn't also hsve a high level of experience and compentence in those languages. It has been literally 10 years since I touched Python. I use shell scripting every day. I know which one I'm going to be better at using to produce reliable code that does what I want it to do.
  - JoBrad 9y ago
    > It has been literally 10 years since I touched Python That's fine. Python 2.6 is still in wide use.
TheAceOfHearts 9y ago
Writing bug-free shell programs is hard because it doesn't provide safe defaults. By now I've gotten reasonably decent at shell scripts due to trial and error, and shellcheck... But if I'm doing anything fancy, I just opt for a node script using shelljs and any other third-party deps to cover my needs. This also has the benefit of typically working cross-platform. Although that's not as big of a concern for me, for some people it makes a big difference. For my day-to-day shell, I use fish. IMO, it has much better defaults than bash and zsh, and it's fast. I still have to use bash sometimes, but it's becoming increasingly rare. Your linked example doesn't really seem particularly tricky to me. I'd feel comfortable writing it in something like node or ruby, and I believe the result would probably be easier to consume for certain people.
- mbrock 9y ago
  There are a lot of unsafe defaults in JavaScript too, most prominently async error propagation. Basically all languages have some tricky fatal flaws, even Haskell (lazy I/O can make very strange bugs, etc). I just don't think we should reject shell scripts entirely because some care is required to write them correctly. Shell is way too valuable.
  - TheAceOfHearts 9y ago
    You make a fair point. Although in more recent versions of JS the whole async error propagation isn't as big of an issue thanks to async functions and promises. I guess the biggest reason to use JS is that since I'm already using it every day, it's much easier to continue using a language I'm highly familiar with. Whenever I have to write a shell script I always have to pull up references and read through a couple Stack Overflow posts. I don't reject shell scripts, though. I agree that in some cases it can lead to much simpler code. The power and composability with pipes and immediate substitutions is incredibly powerful. I've had a few cases where I replaced far more complicated node scripts with simpler shell scripts. What I'd truly love is a reference guide that showed how to correctly do certain things with shell scripts. Here's a real-world example I struggled with: Writing end-to-end tests with selenium and a test runner. I wanted a script to start up a selenium server and wait for it to be ready. Then it should start the test runner. If the test runner exited successfully, kill the selenium server and browser, and exit with a 0. If the test runner or selenium server crashed or failed somehow, clean up and exit with a 1. Even after much trial and error, I still ended up with occasional Chrome copies staying open after failures.

arca_vorago 9y ago

Could you explain how writing, for example, text manipulation, would be safer in python than in shell? I understand formatting errors, shell escape failures, and the low hanging fruit, but in the end if I am just sed and awking my way around, I don't know how to do nearly the level of stuff I do in awk in python. I also will never forget the time I spent a day doing a perl script to parse some data... took some time, came back, and did it in a one line awk... Very powerful. I think somewhere in here is room for discussion of data, text, and the unix philosophy as everything as a file.

makecheck 9y ago
I said scripts with “growing complexity” should be expressed differently. I can and do use shell scripts for things that are relatively straightforward, and sed/awk-style transforms are a good example (although, even there, if too many quotes are involved you start to long for real escaping and things like triple-quotes).
omaranto 9y ago
A one-liner in Awk is almost certainly also a one-liner in Perl.
- arca_vorago 9y ago
  I didn't say I wasn't doing the perl badly.
highd 9y ago
It's an issue of comprehensibility, not any current performance metric.
- ams6110 9y ago
  Comprehensible to who? Other python programmers? Most sysadmins do not use python routinely. They work in the shell all day every day though.
  - 9y ago

mdekkers 9y ago

or write it in go and run it in a container!

lloeki 9y ago

Shellcheck is a boon to write more (by at least three orders of magnitude) reliable shell code. I wish there were a 'set -o strict' that would make bash consider all shellcheck warnings into errors as well as unilaterally set -euo pipefail. That said, bash is still my go to language for many many things because at the end of the day it's insanely efficient as that koan[0] is still valid when substituting C for mostly any other language. As an anecdote I've replaced ridiculously big and slow Rakefiles and unmaintainable ad hoc ruby code with much shorter, readable and reliable shell scripts. Using the shell doesn't mean you can't drop a call to python, perl, jq or whatever, and the composability and performance can be terrific when properly wielded. [0] http://www.catb.org/esr/writings/unix-koans/ten-thousand.html http://www.catb.org/esr/writings/unix-koans/ten-thousand.htm...

lobster_johnson 9y ago

I agree. Shell scripts are awesome until you reach a certain threshold of complexity, and that threshold is... low. Error handling is particularly problematic. If you know all the tricks (things like set -e, set -o pipefail, always capture stderr, etc.), it's not too bad, but there are still so many traps that the mental overhead of catching every single edge case is exhausting. One nasty issue with Bash is quoting, escaping and word splitting, because almost everything in Bash is about string manipulation. There are so many edge cases: $FOO is not the same as "$FOO", and sometimes you have no choice but to turn it into an array with eval FOO=($FOO) if you need mimic Bash's tokenization behaviour (e.g. respect backslashes and quoting). (Yes, Bash has arrays! A lot of scripts get simpler when you realize this.) These days I reach for Ruby very quickly even for small things.