5 ms·

Maybe some controversial advice: Go ahead, fall in these pits. I write my fair share of shell scripts and I've hit practically every one of these snags in the

Maybe some controversial advice: Go ahead, fall in these pits. I write my fair share of shell scripts and I've hit practically every one of these snags in the past. However, for the majority of tasks I perform with bash, I genuinely don't care if I support spaces in filenames, or if I throw away a little efficiency with a few extra sub-shells, or if I can't test numbers vs strings or have a weird notion of booleans. Your scripts are going to have bugs. The important question is: What happens when they fail? Are your scripts idempotent? Are they audit-able? Interruptible? Do you have backups before performing destructive operations? How do you verify that they did the right job? For example, if your shell scripts operate only on files under version control, you can simply run a diff before committing. Rather than spent a bunch of time tracking down a word expansion bug, you can simply rename that one file that failed to not include a space in its name.

druiid 13y ago
I totally agree. Sometimes the strength of making a 'quick bash script', is that you are making a quick bash script. I have made some pretty strong, well tested projects in bash before including one that was a big part of an open-source qmail project. Sometimes though you just need to get stuff done with the least amount of fuss, without worrying about the extreme edge-cases which the majority of the webpage attached to this story talk about. Heck, it's probably the majority of bash work I'd say that ends up like that.
- clarry 13y ago
  On the other hand, if you examine the given examples, you'll see that very often the "correct" way isn't really longer or harder to type. If you make a point of sticking to the correct way, it will eventually become automatic, and there'll be no fuss and worry. This might end up saving some sorry ass later on. And having learned the correct way, you'll instantly see it when a script you review is doing something in a way that will eventually bite someone. There's no downsides to learning and doing things right. Of course it does take some extra time and effort at the start, as everything.. Sure, there are extreme examples that can be really hard to handle portably and safely if you're doing something more complicated (embedded newlines in filenames come to mind). So in the end some corner cutting is often inevitable :-)
  - brandonbloom 13y ago
    > There's no downsides to learning and doing things right. There is never "no downsides". For one example, there is "feedback fatigue". It's easy to pick on syntax or "small" semantics during a code review, but that's not nearly as useful as analyzing the big picture. I have been party to many code reviews that involved a dozen nit-picky stylistic comments. The reviewer feels like they did their job, the reviewee feels like they have appeased the reviewer, and so the now style-guide-correct code gets a half-hearted "LGTM" and is merged. That code looks great... And it handles filenames with spaces in them!.. But does the totally wrong thing.
brennen 13y ago
I live and die by the shell. I'm constantly composing little one-liners, and keep an absurdly long Bash / zsh history to draw from. There are places the obvious answer is almost always "how about you just write a shell script?" That said, I long ago reached a place where I realized that, while shell scripting is entertaining, I'd much rather write anything more than a handful of lines in a general purpose programming language. Perl, Python, Ruby, whatever - even PHP involves far less syntactic suffering and general impedance than Bash. It's not that I'm exceptionally worried about correctness in stuff that no one besides me is ever going to use, it's just that once you're past a certain very low threshold of complexity, the agony you spend for a piece of reusable code is so much less. Even just stitching together some standard utilities, there are plenty of times it'll take a tenth as long and a thousandth as much swearing to just write some Perl that uses backticks here and there or mangles STDIN as needed. > Are your scripts idempotent? Are they > audit-able? Interruptible? Do you have > backups before performing destructive > operations? How do you verify that they > did the right job? Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives.
- sigil 13y ago
  > Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives. I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?" Statefulness itself is a pain, and shell is in some sense the ultimate language for simply and flexibly dealing with external state. Simplicity: the filesystem is an extremly simple and powerful state representation. Show me a language that interacts with the fs more concisely than tr '[A-Z]' '[a-z]' < upper.txt > lower.txt Flexibility: if shell can't do it, just use another program in another language that can, like `tr` in the above example. What other language enables polyglot programming like this? Literally any program in any language can become a part of a shell program. > it's just that once you're past a certain very low threshold of complexity, the agony you spend for a piece of reusable code is so much less. Here's where I admit I was playing devil's advocate to an extent, because I fully agree with you here. I write lots of shell scripts. I never write big shell scripts. Above some length they just get targeted for replacement in a "real" language, or at the very least, portions of them get rewritten so they can remain small. Empirically, it also seems true that shell is harder for people to grasp, harder to read, and harder for people to get right. These are real costs that have to be figured in. PS. Speaking of shell brennen, we should be working on our weekend project. :)

druiid 13y ago

I totally agree. Sometimes the strength of making a 'quick bash script', is that you are making a quick bash script. I have made some pretty strong, well tested projects in bash before including one that was a big part of an open-source qmail project. Sometimes though you just need to get stuff done with the least amount of fuss, without worrying about the extreme edge-cases which the majority of the webpage attached to this story talk about. Heck, it's probably the majority of bash work I'd say that ends up like that.

clarry 13y ago
On the other hand, if you examine the given examples, you'll see that very often the "correct" way isn't really longer or harder to type. If you make a point of sticking to the correct way, it will eventually become automatic, and there'll be no fuss and worry. This might end up saving some sorry ass later on. And having learned the correct way, you'll instantly see it when a script you review is doing something in a way that will eventually bite someone. There's no downsides to learning and doing things right. Of course it does take some extra time and effort at the start, as everything.. Sure, there are extreme examples that can be really hard to handle portably and safely if you're doing something more complicated (embedded newlines in filenames come to mind). So in the end some corner cutting is often inevitable :-)
- brandonbloom 13y ago
  > There's no downsides to learning and doing things right. There is never "no downsides". For one example, there is "feedback fatigue". It's easy to pick on syntax or "small" semantics during a code review, but that's not nearly as useful as analyzing the big picture. I have been party to many code reviews that involved a dozen nit-picky stylistic comments. The reviewer feels like they did their job, the reviewee feels like they have appeased the reviewer, and so the now style-guide-correct code gets a half-hearted "LGTM" and is merged. That code looks great... And it handles filenames with spaces in them!.. But does the totally wrong thing.

brennen 13y ago

I live and die by the shell. I'm constantly composing little one-liners, and keep an absurdly long Bash / zsh history to draw from. There are places the obvious answer is almost always "how about you just write a shell script?" That said, I long ago reached a place where I realized that, while shell scripting is entertaining, I'd much rather write anything more than a handful of lines in a general purpose programming language. Perl, Python, Ruby, whatever - even PHP involves far less syntactic suffering and general impedance than Bash. It's not that I'm exceptionally worried about correctness in stuff that no one besides me is ever going to use, it's just that once you're past a certain very low threshold of complexity, the agony you spend for a piece of reusable code is so much less. Even just stitching together some standard utilities, there are plenty of times it'll take a tenth as long and a thousandth as much swearing to just write some Perl that uses backticks here and there or mangles STDIN as needed. > Are your scripts idempotent? Are they > audit-able? Interruptible? Do you have > backups before performing destructive > operations? How do you verify that they > did the right job? Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives.

sigil 13y ago
> Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives. I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?" Statefulness itself is a pain, and shell is in some sense the ultimate language for simply and flexibly dealing with external state. Simplicity: the filesystem is an extremly simple and powerful state representation. Show me a language that interacts with the fs more concisely than tr '[A-Z]' '[a-z]' < upper.txt > lower.txt Flexibility: if shell can't do it, just use another program in another language that can, like `tr` in the above example. What other language enables polyglot programming like this? Literally any program in any language can become a part of a shell program. > it's just that once you're past a certain very low threshold of complexity, the agony you spend for a piece of reusable code is so much less. Here's where I admit I was playing devil's advocate to an extent, because I fully agree with you here. I write lots of shell scripts. I never write big shell scripts. Above some length they just get targeted for replacement in a "real" language, or at the very least, portions of them get rewritten so they can remain small. Empirically, it also seems true that shell is harder for people to grasp, harder to read, and harder for people to get right. These are real costs that have to be figured in. PS. Speaking of shell brennen, we should be working on our weekend project. :)