5 ms·

You still need to know how to write safe shell programs even if they're one-liners. And once you do, shell is really, really useful and convenient. For exampl

by mbrock 10y ago

You still need to know how to write safe shell programs even if they're one-liners. And once you do, shell is really, really useful and convenient. For example, Linux distributions are full of shell scripts that would be frankly awful to write in any other language that I know of... like this, to take a random example: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/virtualization/docker/default.nix#L99 https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...

Jabbles 10y ago
Shouldn't they be quoted?
tedmiston 10y ago
Most app developers aren't writing the scripts in Linux distros though.
deleted 10y ago
[deleted]
masklinn 10y ago
Eh. Looks OK to me, even with Python probably not being the tersest shell language: https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb88ecb5e https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb8...
- mbrock 10y ago
  Hmm, I admit that's not too bad. I still disagree with the ideology that any non-trivial script needs to be rewritten in Python. The shell is the standard way of interacting with a *nix system, and for me that's the major benefit of using shell for "system"-ish tasks. I already know how to make a symlink, a pipeline, a chmod, etc, so I don't need to learn a new syntax for that.
- Too 10y ago
  The shell script probably also behaves strange if any of $out ${docker-runc} $version, $man, etc contains a space, while the python-script most likely wont.
  - mbrock 10y ago
    Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables. Of course, as a developer you have to understand this. The semantics of shell string interpolation are generally not that hard to learn, compared to learning all the nuances of programming in general that can make your program incorrect, so I find it a bit strange how it's considered to make shell completely useless for scripting. Shell scripting has been working pretty well since, like 1975! It's not that bad!
    - zokier 10y ago
      > Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables Which works as long as attacker (or sleepy admin) doesn't happen to create suitably "bad" path > Shell scripting has been working pretty well since, like 1975! It's not that bad! Considering the amount of borked system because spaces/uninitialized variables -stories I've heard over years, I'm not so sure about how well it has been working...
      - mbrock 10y ago
        Attackers who have commit access to your repository can do a lot of bad things quite easily.
- CJefferson 10y ago
  Someone should try to automate this process, or at least make a complete bash -> python guide. This is much nicer than I expected, but the functions are scattered around a bit.
  - masklinn 10y ago
    Namespaces are a honking great idea, let's do more of those! Alternatively there's always the option of using something like sh[0] and just shelling out to the existing utilities, if you don't mind the portability loss. [0] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
- ReidZB 10y ago
  There are several strings in your gist that have the f misplaced, e.g. 'f{out}/libexec/docker/docker-proxy' instead of f'{out}/libexec/docker/docker-proxy' Not sure it is materially important to the argument, but I thought it was interesting
  - masklinn 10y ago
    > Not sure it is materially important to the argument Not really, I've also forgotten to convert a few dashes to underscores in f-strings and probably made a few other mistakes along the way. It doesn't detract from the main point: it's a rough estimate of how such a converted script would look. Though it makes me wonder if PyLint already has a lint for strings which start with F and have f-string placeholders.
developer2 10y ago
>> need to know how to write safe shell programs >> once you do, shell is really, really useful and convenient Part of the issue here is that a large number of people who think they know how to write safe shell scripts do not in fact have the ability to do so. There are so many gotchas to learn that it's all too easy to believe oneself competent, while being unaware of just how short one's knowledge falls. It's just too easy to write a script in the shell of your choice that works - until it encounters an edge case scenario that wasn't planned for, at which point the results can be catastrophic. This is true of any programming language, but the tasks for which people write shell scripts (such as massive system-wide filesystem manipulation) makes the consequences of a single-line bug very real.
- ams6110 10y ago
  > This is true of any programming language Glad you mentioned that, because that was going to be my reply after reading the first half of your post. I'm not sure a person who is fairly well-versed in shell scripting is going to do a worse job than trying to use python, perl, lua, etc. if he doesn't also hsve a high level of experience and compentence in those languages. It has been literally 10 years since I touched Python. I use shell scripting every day. I know which one I'm going to be better at using to produce reliable code that does what I want it to do.
  - JoBrad 10y ago
    > It has been literally 10 years since I touched Python That's fine. Python 2.6 is still in wide use.
TheAceOfHearts 10y ago
Writing bug-free shell programs is hard because it doesn't provide safe defaults. By now I've gotten reasonably decent at shell scripts due to trial and error, and shellcheck... But if I'm doing anything fancy, I just opt for a node script using shelljs and any other third-party deps to cover my needs. This also has the benefit of typically working cross-platform. Although that's not as big of a concern for me, for some people it makes a big difference. For my day-to-day shell, I use fish. IMO, it has much better defaults than bash and zsh, and it's fast. I still have to use bash sometimes, but it's becoming increasingly rare. Your linked example doesn't really seem particularly tricky to me. I'd feel comfortable writing it in something like node or ruby, and I believe the result would probably be easier to consume for certain people.
- mbrock 10y ago
  There are a lot of unsafe defaults in JavaScript too, most prominently async error propagation. Basically all languages have some tricky fatal flaws, even Haskell (lazy I/O can make very strange bugs, etc). I just don't think we should reject shell scripts entirely because some care is required to write them correctly. Shell is way too valuable.
  - TheAceOfHearts 10y ago
    You make a fair point. Although in more recent versions of JS the whole async error propagation isn't as big of an issue thanks to async functions and promises. I guess the biggest reason to use JS is that since I'm already using it every day, it's much easier to continue using a language I'm highly familiar with. Whenever I have to write a shell script I always have to pull up references and read through a couple Stack Overflow posts. I don't reject shell scripts, though. I agree that in some cases it can lead to much simpler code. The power and composability with pipes and immediate substitutions is incredibly powerful. I've had a few cases where I replaced far more complicated node scripts with simpler shell scripts. What I'd truly love is a reference guide that showed how to correctly do certain things with shell scripts. Here's a real-world example I struggled with: Writing end-to-end tests with selenium and a test runner. I wanted a script to start up a selenium server and wait for it to be ready. Then it should start the test runner. If the test runner exited successfully, kill the selenium server and browser, and exit with a 0. If the test runner or selenium server crashed or failed somehow, clean up and exit with a 1. Even after much trial and error, I still ended up with occasional Chrome copies staying open after failures.

by mbrock 10y ago

Jabbles 10y ago

Shouldn't they be quoted?

tedmiston 10y ago

Most app developers aren't writing the scripts in Linux distros though.

deleted 10y ago

[deleted]

masklinn 10y ago

Eh. Looks OK to me, even with Python probably not being the tersest shell language: https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb88ecb5e https://gist.github.com/anonymous/4203b7c6ffc9cb9ecf34d17cb8...

mbrock 10y ago
Hmm, I admit that's not too bad. I still disagree with the ideology that any non-trivial script needs to be rewritten in Python. The shell is the standard way of interacting with a *nix system, and for me that's the major benefit of using shell for "system"-ish tasks. I already know how to make a symlink, a pipeline, a chmod, etc, so I don't need to learn a new syntax for that.
Too 10y ago
The shell script probably also behaves strange if any of $out ${docker-runc} $version, $man, etc contains a space, while the python-script most likely wont.
- mbrock 10y ago
  Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables. Of course, as a developer you have to understand this. The semantics of shell string interpolation are generally not that hard to learn, compared to learning all the nuances of programming in general that can make your program incorrect, so I find it a bit strange how it's considered to make shell completely useless for scripting. Shell scripting has been working pretty well since, like 1975! It's not that bad!
  - zokier 10y ago
    > Nix-generated paths don't contain spaces, so it's unnecessary to quote those variables Which works as long as attacker (or sleepy admin) doesn't happen to create suitably "bad" path > Shell scripting has been working pretty well since, like 1975! It's not that bad! Considering the amount of borked system because spaces/uninitialized variables -stories I've heard over years, I'm not so sure about how well it has been working...
    - mbrock 10y ago
      Attackers who have commit access to your repository can do a lot of bad things quite easily.
CJefferson 10y ago
Someone should try to automate this process, or at least make a complete bash -> python guide. This is much nicer than I expected, but the functions are scattered around a bit.
- masklinn 10y ago
  Namespaces are a honking great idea, let's do more of those! Alternatively there's always the option of using something like sh[0] and just shelling out to the existing utilities, if you don't mind the portability loss. [0] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
ReidZB 10y ago
There are several strings in your gist that have the f misplaced, e.g. 'f{out}/libexec/docker/docker-proxy' instead of f'{out}/libexec/docker/docker-proxy' Not sure it is materially important to the argument, but I thought it was interesting
- masklinn 10y ago
  > Not sure it is materially important to the argument Not really, I've also forgotten to convert a few dashes to underscores in f-strings and probably made a few other mistakes along the way. It doesn't detract from the main point: it's a rough estimate of how such a converted script would look. Though it makes me wonder if PyLint already has a lint for strings which start with F and have f-string placeholders.

developer2 10y ago

>> need to know how to write safe shell programs >> once you do, shell is really, really useful and convenient Part of the issue here is that a large number of people who think they know how to write safe shell scripts do not in fact have the ability to do so. There are so many gotchas to learn that it's all too easy to believe oneself competent, while being unaware of just how short one's knowledge falls. It's just too easy to write a script in the shell of your choice that works - until it encounters an edge case scenario that wasn't planned for, at which point the results can be catastrophic. This is true of any programming language, but the tasks for which people write shell scripts (such as massive system-wide filesystem manipulation) makes the consequences of a single-line bug very real.

ams6110 10y ago
> This is true of any programming language Glad you mentioned that, because that was going to be my reply after reading the first half of your post. I'm not sure a person who is fairly well-versed in shell scripting is going to do a worse job than trying to use python, perl, lua, etc. if he doesn't also hsve a high level of experience and compentence in those languages. It has been literally 10 years since I touched Python. I use shell scripting every day. I know which one I'm going to be better at using to produce reliable code that does what I want it to do.
- JoBrad 10y ago
  > It has been literally 10 years since I touched Python That's fine. Python 2.6 is still in wide use.

TheAceOfHearts 10y ago

Writing bug-free shell programs is hard because it doesn't provide safe defaults. By now I've gotten reasonably decent at shell scripts due to trial and error, and shellcheck... But if I'm doing anything fancy, I just opt for a node script using shelljs and any other third-party deps to cover my needs. This also has the benefit of typically working cross-platform. Although that's not as big of a concern for me, for some people it makes a big difference. For my day-to-day shell, I use fish. IMO, it has much better defaults than bash and zsh, and it's fast. I still have to use bash sometimes, but it's becoming increasingly rare. Your linked example doesn't really seem particularly tricky to me. I'd feel comfortable writing it in something like node or ruby, and I believe the result would probably be easier to consume for certain people.

mbrock 10y ago
There are a lot of unsafe defaults in JavaScript too, most prominently async error propagation. Basically all languages have some tricky fatal flaws, even Haskell (lazy I/O can make very strange bugs, etc). I just don't think we should reject shell scripts entirely because some care is required to write them correctly. Shell is way too valuable.
- TheAceOfHearts 10y ago
  You make a fair point. Although in more recent versions of JS the whole async error propagation isn't as big of an issue thanks to async functions and promises. I guess the biggest reason to use JS is that since I'm already using it every day, it's much easier to continue using a language I'm highly familiar with. Whenever I have to write a shell script I always have to pull up references and read through a couple Stack Overflow posts. I don't reject shell scripts, though. I agree that in some cases it can lead to much simpler code. The power and composability with pipes and immediate substitutions is incredibly powerful. I've had a few cases where I replaced far more complicated node scripts with simpler shell scripts. What I'd truly love is a reference guide that showed how to correctly do certain things with shell scripts. Here's a real-world example I struggled with: Writing end-to-end tests with selenium and a test runner. I wanted a script to start up a selenium server and wait for it to be ready. Then it should start the test runner. If the test runner exited successfully, kill the selenium server and browser, and exit with a 0. If the test runner or selenium server crashed or failed somehow, clean up and exit with a 1. Even after much trial and error, I still ended up with occasional Chrome copies staying open after failures.