5 ms·

Any time I have to do any serious shell scripting at work, I die a little inside. I do need to try out shellcheck thoroughly, it might catch some bugs in our pi

Any time I have to do any serious shell scripting at work, I die a little inside. I do need to try out shellcheck thoroughly, it might catch some bugs in our pipelines

dcminter 6y ago
I started using it about a year ago (I wasn't aware of it before that) and it's astonishing the number of things it can pick up on in even a five line script. Often they're things you will get away with most or even all of the time in a specific scenario, but there they lurk, waiting to blow your leg off when you start including spaces in a filename or somesuch!
dataflow 6y ago
Honestly, the only practical solution to this in my experience is to just buckle up and spend the hours/days it takes to learn the dirty corners of Bash properly. This doesn't mean you have to learn 100% of it, but it does mean you need to learn enough of it that will (a) serve your needs comfortably (instead of making you constantly have to hack around), and (b) give you insight as to what subset(s) to avoid unless absolutely necessary. Same goes with other tools, whether with warranted or unwarranted complexity (like git, Bazel, C++, etc.). Yes it's painful while you're still learning, but it's an investment that makes you more productive and just makes everything so much more tolerable afterward. If you never spend the time then you'll keep dying inside every time you have to deal with tools you don't like.
- doctoboggan 6y ago
  I feel like I’m overdue for learning she’ll scripting properly. I’ve hacked around for years google as I go. What resources would you suggest for decent training material?
  - dataflow 6y ago
    I wish I had a great answer, but I don't. Maybe these will help, though. What I've done is, every time I came across something that didn't behave the way I expected, I tried to Google it and tinker around with it until I could understand exactly what is going on, so that I "got" the underlying feel for the language and didn't have to look up every consequence every time. e.g., the difference between [ ] and [[ ]], which is confusing at first, becomes more intuitive once you realize [ is a portable POSIX command whereas [[ is special Bash syntax, and once you grasp this, you can understand what its implications are. e.g., that Bash has no way to detect '=' as a special token inside [ ], and consequently you should worry about ambiguities. (This does assume a little background knowledge of parsing ambiguities; if you haven't taken a course related to programming languages, going through one might also be a worthwhile investment.) Also, you have to be on the lookout for things that you would want to do in a different language, and learn how to do it in Bash, like creating functions & local variables. It's mostly a matter of being proactive about learning how to structure your code like you would in other languages. Often I see people try to keep things "simple" by just copy/pasting code so that they don't have to organize things at a higher level. You wouldn't do that in other languages; don't do that in Bash. Some pitfalls you really just do have to learn the hard way; there's no other way I can think of. The key here is to spend the extra minutes necessary to actually learn what broke when you come across it, instead of just tinkering with your code until it works. Like when you realize that (set -e; false && false; echo hi) prints 'hi' despite the fact that (set -e; false; echo hi) does not (this is absolutely bonkers), dig in and figure out what's going on so you don't shoot yourself in the foot the next time. You can't avoid it the first time, but you can at least make sure to avoid it in the future. Also, always try to think of unintended consequences when you learn how to do something in Bash. Like when you learn how `trap 'rm -f file.bak' EXIT` lets you clean up a file, ask yourself what happens if someone has already set a trap? Ask yourself how to dynamically generate and pass a file name to that command safely? Ask yourself in what scope/context the command is evaluated? Spend some time figuring these out either online or by tinkering manually, then remember your solution for later (maybe write it down). Possibly the most costly mistake you can make is to assume something will work in exactly the same way you expect in every other language, because chances are pretty darn good it won't. Even pipelines (which you probably already use) are tricky if you start thinking about them... e.g., what happens if the first command terminates prematurely? What happens if you assign to a variable? etc. Some sites I can recommend for some things: - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s... - https://tldp.org/LDP/abs/html/index.html https://tldp.org/LDP/abs/html/index.html - https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls The POSIX pages (OpenGroup, first link) are useful for understanding what you can rely on; I'd use that as a command reference whenever it's possible. In some cases they're overly pessimistic (e.g., you can probably rely on 'local' despite it not being POSIX), but generally they're good starting points. tldp is where I still look up some things, like the heredoc syntax. I've probably looked that up 20 times and I still don't remember it well enough for some reason. And it might be fun to just browse through the BashPitfalls page once you have some comfort with shell scripting just to get an idea of what can go wrong, but I wouldn't really recommend it as a learning guide, and you definitely shouldn't expect to memorize all of it.
  - yrro 6y ago
    For some reason people always overlook the Bash Reference Manual: https://www.gnu.org/software/bash/manual/bash.html https://www.gnu.org/software/bash/manual/bash.html (Usually available locally at 'info bash', I know you hate info don't @ me)
    - ufo 6y ago
      I also recommend the man pages for the dash shell. They are the first reference I look at when I need to remember how something works in POSIX shell, without bashisms.
  - deleted 6y ago
    [deleted]
  - wadkar 6y ago
    Personally, I learnt my bash by reading a lot of Gentoo EBUILD scripts and config files. I can recognize other Gentoo users just by how many curly braces and double quotes I see in a simple one liner. The one resource that I have bookmarked decades ago and would highly recommend to get the ball rolling are the TLDP pages on bash: https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/ Not certain if they’re up to date or error free but I really enjoy the style and tutorial/show by example nature of the TLDP
akvadrako 6y ago
On the other hand bash is one of my favorite languages to develop for. It took many years to get proficient and you shouldn't try to do anything too complex but it has good sides: 1. The REPL is top notch. 2. Good documentation. Most commands come with a complete manual and examples are easy to find. 3. It's portable and has lots of available APIs, probably even more than C.
- TazeTSchnitzel 6y ago
  It's portable until you realise that you've used several utilities that aren't available on all Unices by default.
ransom1538 6y ago
"Any time I have to do any serious shell scripting at work," Right when this happens, that very second, remember there are hundreds of other languages that can do anything bash can in a readable fashion. EG. os.system() in python. If you are writing more than 3 lines of bash you are doing it all wrong.
- TazeTSchnitzel 6y ago
  When I need a shell script, I write a PHP script instead. (PHP has its oddities, but it has many benefits over Bash, including an optionally strictly-enforced type system, binary-safe strings, clear C-like syntax… and like Bash it has backticks!)
JeremyNT 6y ago
Shellcheck is immensely helpful and I highly recommend incorporating it into your editing workflow (that is, lint as you go with a language server) before it hits the CI/CD pipeline. (ba)sh scripting has so many footguns it astounds me that so much complicated code has been written (more-or-less) successfully in it. Once I reach more than a few dozen lines I usually hit the escape hatch and reach for a more powerful scripting language (you can always still shell out to utilize handy CLI tools if you need them).
SavantIdiot 6y ago
Yeah, but I think it is good to do this kind of work because it reminds of of what can be done with an interpreter that is less than 1MB. Time and again I've heard folks on HN say, "Why aren't shell scripts just written in Python?" On the one hand, I think it is great to think in "bare metal", on the other hand do we really need to think in "bare metal"? Shouldn't we free our brains from solving (and resolving) 40 year old problems. I dunno, my point is: I feel that "little death" you describe too. (Non, francaise, pas cette petit mort!)
l0b0 6y ago
I wrote The newline Guide to Bash Scripting[1] (released today!) with users like you in mind: already familiar with programming, needs to use Bash, but don't want to invest hundreds of hours to understand how to list files, use redirects, handle arguments, and so on. [1] https://www.newline.co/courses/newline-guide-to-bash-scripting https://www.newline.co/courses/newline-guide-to-bash-scripti...

Any time I have to do any serious shell scripting at work, I die a little inside. I do need to try out shellcheck thoroughly, it might catch some bugs in our pipelines

dcminter 6y ago

I started using it about a year ago (I wasn't aware of it before that) and it's astonishing the number of things it can pick up on in even a five line script. Often they're things you will get away with most or even all of the time in a specific scenario, but there they lurk, waiting to blow your leg off when you start including spaces in a filename or somesuch!

dataflow 6y ago

Honestly, the only practical solution to this in my experience is to just buckle up and spend the hours/days it takes to learn the dirty corners of Bash properly. This doesn't mean you have to learn 100% of it, but it does mean you need to learn enough of it that will (a) serve your needs comfortably (instead of making you constantly have to hack around), and (b) give you insight as to what subset(s) to avoid unless absolutely necessary. Same goes with other tools, whether with warranted or unwarranted complexity (like git, Bazel, C++, etc.). Yes it's painful while you're still learning, but it's an investment that makes you more productive and just makes everything so much more tolerable afterward. If you never spend the time then you'll keep dying inside every time you have to deal with tools you don't like.

doctoboggan 6y ago
I feel like I’m overdue for learning she’ll scripting properly. I’ve hacked around for years google as I go. What resources would you suggest for decent training material?
- dataflow 6y ago
  I wish I had a great answer, but I don't. Maybe these will help, though. What I've done is, every time I came across something that didn't behave the way I expected, I tried to Google it and tinker around with it until I could understand exactly what is going on, so that I "got" the underlying feel for the language and didn't have to look up every consequence every time. e.g., the difference between [ ] and [[ ]], which is confusing at first, becomes more intuitive once you realize [ is a portable POSIX command whereas [[ is special Bash syntax, and once you grasp this, you can understand what its implications are. e.g., that Bash has no way to detect '=' as a special token inside [ ], and consequently you should worry about ambiguities. (This does assume a little background knowledge of parsing ambiguities; if you haven't taken a course related to programming languages, going through one might also be a worthwhile investment.) Also, you have to be on the lookout for things that you would want to do in a different language, and learn how to do it in Bash, like creating functions & local variables. It's mostly a matter of being proactive about learning how to structure your code like you would in other languages. Often I see people try to keep things "simple" by just copy/pasting code so that they don't have to organize things at a higher level. You wouldn't do that in other languages; don't do that in Bash. Some pitfalls you really just do have to learn the hard way; there's no other way I can think of. The key here is to spend the extra minutes necessary to actually learn what broke when you come across it, instead of just tinkering with your code until it works. Like when you realize that (set -e; false && false; echo hi) prints 'hi' despite the fact that (set -e; false; echo hi) does not (this is absolutely bonkers), dig in and figure out what's going on so you don't shoot yourself in the foot the next time. You can't avoid it the first time, but you can at least make sure to avoid it in the future. Also, always try to think of unintended consequences when you learn how to do something in Bash. Like when you learn how `trap 'rm -f file.bak' EXIT` lets you clean up a file, ask yourself what happens if someone has already set a trap? Ask yourself how to dynamically generate and pass a file name to that command safely? Ask yourself in what scope/context the command is evaluated? Spend some time figuring these out either online or by tinkering manually, then remember your solution for later (maybe write it down). Possibly the most costly mistake you can make is to assume something will work in exactly the same way you expect in every other language, because chances are pretty darn good it won't. Even pipelines (which you probably already use) are tricky if you start thinking about them... e.g., what happens if the first command terminates prematurely? What happens if you assign to a variable? etc. Some sites I can recommend for some things: - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s... - https://tldp.org/LDP/abs/html/index.html https://tldp.org/LDP/abs/html/index.html - https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls The POSIX pages (OpenGroup, first link) are useful for understanding what you can rely on; I'd use that as a command reference whenever it's possible. In some cases they're overly pessimistic (e.g., you can probably rely on 'local' despite it not being POSIX), but generally they're good starting points. tldp is where I still look up some things, like the heredoc syntax. I've probably looked that up 20 times and I still don't remember it well enough for some reason. And it might be fun to just browse through the BashPitfalls page once you have some comfort with shell scripting just to get an idea of what can go wrong, but I wouldn't really recommend it as a learning guide, and you definitely shouldn't expect to memorize all of it.
- yrro 6y ago
  For some reason people always overlook the Bash Reference Manual: https://www.gnu.org/software/bash/manual/bash.html https://www.gnu.org/software/bash/manual/bash.html (Usually available locally at 'info bash', I know you hate info don't @ me)
  - ufo 6y ago
    I also recommend the man pages for the dash shell. They are the first reference I look at when I need to remember how something works in POSIX shell, without bashisms.
- deleted 6y ago
  [deleted]
- wadkar 6y ago
  Personally, I learnt my bash by reading a lot of Gentoo EBUILD scripts and config files. I can recognize other Gentoo users just by how many curly braces and double quotes I see in a simple one liner. The one resource that I have bookmarked decades ago and would highly recommend to get the ball rolling are the TLDP pages on bash: https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/ Not certain if they’re up to date or error free but I really enjoy the style and tutorial/show by example nature of the TLDP

akvadrako 6y ago

On the other hand bash is one of my favorite languages to develop for. It took many years to get proficient and you shouldn't try to do anything too complex but it has good sides: 1. The REPL is top notch. 2. Good documentation. Most commands come with a complete manual and examples are easy to find. 3. It's portable and has lots of available APIs, probably even more than C.

TazeTSchnitzel 6y ago
It's portable until you realise that you've used several utilities that aren't available on all Unices by default.

ransom1538 6y ago

"Any time I have to do any serious shell scripting at work," Right when this happens, that very second, remember there are hundreds of other languages that can do anything bash can in a readable fashion. EG. os.system() in python. If you are writing more than 3 lines of bash you are doing it all wrong.

TazeTSchnitzel 6y ago
When I need a shell script, I write a PHP script instead. (PHP has its oddities, but it has many benefits over Bash, including an optionally strictly-enforced type system, binary-safe strings, clear C-like syntax… and like Bash it has backticks!)

JeremyNT 6y ago

Shellcheck is immensely helpful and I highly recommend incorporating it into your editing workflow (that is, lint as you go with a language server) before it hits the CI/CD pipeline. (ba)sh scripting has so many footguns it astounds me that so much complicated code has been written (more-or-less) successfully in it. Once I reach more than a few dozen lines I usually hit the escape hatch and reach for a more powerful scripting language (you can always still shell out to utilize handy CLI tools if you need them).

SavantIdiot 6y ago

Yeah, but I think it is good to do this kind of work because it reminds of of what can be done with an interpreter that is less than 1MB. Time and again I've heard folks on HN say, "Why aren't shell scripts just written in Python?" On the one hand, I think it is great to think in "bare metal", on the other hand do we really need to think in "bare metal"? Shouldn't we free our brains from solving (and resolving) 40 year old problems. I dunno, my point is: I feel that "little death" you describe too. (Non, francaise, pas cette petit mort!)

l0b0 6y ago

I wrote The newline Guide to Bash Scripting[1] (released today!) with users like you in mind: already familiar with programming, needs to use Bash, but don't want to invest hundreds of hours to understand how to list files, use redirects, handle arguments, and so on. [1] https://www.newline.co/courses/newline-guide-to-bash-scripting https://www.newline.co/courses/newline-guide-to-bash-scripti...