8 ms·

I wrote a book on Bash too. The most important thing for anyone to know about Bash is that it's intended as a command language, not a general purpose scripting

I wrote a book on Bash too. The most important thing for anyone to know about Bash is that it's intended as a command language, not a general purpose scripting language. If it's longer than 10 lines, or if it uses two or more variables, you should probably have written it in something other than Bash.

Ancalagon 9y ago
Amen to that. Now if only I could have gotten my Systems Programming professor to feel the same way... :)
chucknelson 9y ago
Would you consider non-trivial install scripts as an exception to this general rule? I mean, you wouldn't write some install script in ruby or python, right?
- ghettoimp 9y ago
  I don't know why not? At the least, rather than Bash, you might consider Perl as a default, lowest common denominator for scripts that need to run anywhere. - It's nearly as ubiquitous as bash. - It has approximately the same kinds of file/path operations built in. - It has reasonably good support for strings/regexes/etc. all built-in, so you don't have to call out to tools like sed/awk/grep all the time and hope that they are available and compatible across your target platforms. - It provides reasonably good arrays and hashes, which are horribly horrible in bash.[1] - You can use syscalls very easily if you really need to, but usually you don't. [1] Of course, no language can save you from the file system disaster (https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames....), but being able to know that "foo bar" is a string instead of two array elements is a good start. Mostly this all applies to Ruby or Python too, modulo perhaps the degree of ubiquity.
  - gbacon 9y ago
    Great point about ubiquity. Perl regexes are the best of breed that everyone else replicates — far better than “reasonably good.” The Perl erasure in this HN thread is startling.
    - pletnes 9y ago
      Perl erasure? After using the linux command line (or its many «relatives» like cygwin, mac os, unixes) for more than 10 years now, I’ve talked to exactly one person that used perl to accomplish anything at all. He used it to edit text files, so he could have done the same in awk/sed/vim in my opinion. I know more people who write fortran 77 than perl. People just don’t seem to use (or like) Perl very much.
    - grzm 9y ago
      > "The Perl erasure in this HN thread is startling." "Erasure" to me implies some active effort to remove Perl from discourse. I don't see anything like that here: indeed, there are a number of positive mentions, and no negative ones I see. Granted, Python and Ruby are both mentioned more often, but none of those is at Perl's expense. Am I misunderstanding what you mean by 'erasure'?
      - gbacon 9y ago
        This is what Perl is designed to be. Perl unlike the others is almost certain to be on any Unix or Linux installation. Several commenters leaving out Perl in discussions of the next step up from bash scripts is truly strange. I suppose being ignored beats the typical herp-derp anti-Perl bigotry, but I’d prefer all-around civility.
        grzm 9y ago
        > "Several commenters leaving out Perl in discussions of the next step up from bash scripts is truly odd." I'm having a hard time following you here. Do you think that they're doing so for any other reason that Perl is no longer their go-to tool? There are communities where Perl is still used: PostgreSQL for example uses Perl for some of its scripting, as well as its build farm tool, in particular because of its portability on older systems. That said, from what I've seen over the past 10 years or so, Perl hasn't had much of a presence in areas where a lot of computer work in tech is being done. For example, in cloud computing, or scientific computing, or machine learning, or web frameworks. Please don't read this to mean that Perl couldn't be or isn't being used in these cases or wouldn't be a better fit. (As an aside, I think Perl missed out a lot while a large portion of the community was focused on Perl 6: there's only so much energy in a community, and that absorbed on Perl 6 wasn't focusing on evangelism. But that's not something I'm interested in litigating here.) Or that there isn't something a bit frustrating in seeing the wheel reinvented time and time again. And so many examples on the web use bash as a common denominator. This puts Perl further out of mind if it's not already part of your everyday workflow. And how many developers today have come of age without seeing Perl in their everyday environments? Consider the current forum. What's the percentage of front-page posts that are about Perl or tools where Perl is a part of the tool chain? It would be understandable for the people who frequent HN to not view Perl as their go-to. I don't consider it uncivil for people to neglect to mention some other language when it's not something they'd actually think of reaching for. It seems the solution would be to share examples of where Perl provides advantages, both in the comments here and in submissions to HN.
- thomaslangston 9y ago
  Yes, you would write non-trivial install scripts in ruby. https://en.m.wikipedia.org/wiki/Chef_(software) https://en.m.wikipedia.org/wiki/Chef_(software)
  - vinceguidry 9y ago
    If any of my bash scripts get long enough to where I want a reusable class, I rewrite in Ruby.
- khedoros1 9y ago
  Bash has too many surprising edge cases. A lot of my install scripts have at least snippets of Perl in them. The installation/upgrade/maintenance scripts for my employer's main product is Ruby-based. We support a lot of different OSes, and there's usually less variance between the Perl deployed on them than there is in the shell.
- sethrin 9y ago
  I would not consider those an exception, and would typically prefer to see the use of some other language. Bash is excellent at dealing with semi-structured text, and as part of a command pipeline, and if you have no alternative than to write POSIX sh, well, it exists. Bash does not have niceties like typed variables or named function arguments, and arrays are best avoided. Even parsing command line arguments is more fun in other languages. The problem is that Bash has about the lowest barrier to entry which can be found: just dump the things you were going to type anyway into a text file and mark it executable. It's simple, and then you bloody your nose on one of Bash's many idiosyncrasies, and people will say, "Oh yes, ']' is just a required last argument to '['. You gotta watch for that." And the true Bash master knows that my rule is silly and that anything may be written in Bash. Just...please don't.
  - mbrock 9y ago
    I say please don't write any goddamned software at all.
- weaksauce 9y ago
  Isn't that what chef and Capistrano are at some level?
Goladus 9y ago
I disagree. Here's how I decide: Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash to its limits and I tend to set the bar fairly low here. Is the program oriented around commands? If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language. Argument passing is more natural and convenient and the built-in support for the standard i/o streams makes it easy for the different commands to pass data between them. The number of variables or lines of logic is usually less important than the number of different installed commands I need to combine. (Or the number of variations on a single command) Do I need to modify the environment in a significant way? In a bash script it's trivial to source in an environment script, which can be done conditionally or even interactively. This is usually more tedious to do in scripting languages.
- ilyash 9y ago
  [shameless] > Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash > If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language The two points above resonate with my view that there is a missing piece. On one hand we have bash which is optimized for being the glue (second point). On the other hand we have Ruby, Python, Perl, Go, etc which are good for first point. What I think is missing is newer, more powerful shell, which supports both use cases and more. I'm working on it: https://github.com/ilyash/ngs https://github.com/ilyash/ngs Please note that I'm not the only one that thinks there is room for more powerful shells. See the readme for links to other projects.
- falsedan 9y ago
  Also: do I need to run a producer & consumer (and maybe some intermediary filters) on a stream of data, and want to run them in parallel? Shell pipelines are trivially easy to set up & test.
- kinkrtyavimoodh 9y ago
  As someone who has to occasionally modify 100+ line bash scripts written by Coworkers from Christmas Past which matched your spec in terms of what they had to do, please please just use Python (or similar). Yes, you will have a few extra lines but it will be vastly more readable and maintainable. And yes, I know I will get the standard the person who wrote the script did a bad job but at some point it should be okay to blame the tools instead of the workman if workmen disproportionately create worse results with a set of tools.
JepZ 9y ago
I disagree too. Bash has many downsides, but there are very few languages out there which have the ability to 'connect' different programs so easily. Bash scripts are slow as hell (as most commands have to spawn new processes), it is hard to write "secure" code (if even possible), handling whitespaces can be a pain in the * and the amount of repetition is awful. If your kid has done something wrong, just tell it to write a bash script: it is the equivalent of writing a hundred times: x="$(...)" Nevertheless, I enjoy writing bash scripts. Many times it starts with a simple curl command and by the end of the day you have a new OS installer. Granted, there are tasks which other languages can do better, but thats the real power of bash, it doesn't care: Then off you go write your super complicated algorithm in Rust, Go, Python, R or whatever and just call the other program, that's what Bash is good at. It is the glue which keeps everything working together. Disclaimer: Please don't build a complete cathedral out of glue.
- dgsb 9y ago
  I'd rather use perl for these use cases. It's almost as concise as bash regarding subprocess management. It's ubiquitous as bash, and have the python like 2 vs 3 version issue.
- joncrane 9y ago
  "Bash is the glue which keeps everything working together. Please don't build a complete cathedral out of glue." --JepZ
- IshKebab 9y ago
  The reason it can connect programs so "easily" is because it basically ignores errors and robustness - it really relies on there being a user looking at the output and going "hmm that looked like it failed". Doing things properly in Python or Go may take a few more lines (not much more really) but it is 100 tones more robust, and you need that if you are writing anything more than a 10-line one-off hack.
  - mbrock 9y ago
    No, it doesn't. Unix error codes are extremely well understood. Tell the Git maintainers that their program ignores errors and robustness—they'll be surprised.
shrumm 9y ago
I think the 10 line limit seems a little harsh. Bash is great to prototype something, especially because of all the great commands at your disposal versus writing it yourself. e.g. grep, sed, cut are all commands I use frequently. Once you've got a working script and it proves useful or needs to scale - that's a good time to go the 'real language' route like Python.
- bch 9y ago
  > I think the 10 line limit seems a little harsh. I think it's a fine guide. If I'm writing something that starts approaching a program rather than a few lines of utility-throw-away, it's time to at least immediately start co-developing in something more sane. My exception is for Makefiles, if one considers them "shell".
joncrane 9y ago
How do you feel about the use of bash for AWS userdata and the like? You pretty much have to put all the init stuff in a bash script. I can't think of any real-world examples of userdata scripts being less than 10 lines. The alternative of course would be a three line script that downloads a file and executes it. But what language would that file be in? Probably bash. And it would make the infrastructure-as-code tracking and deployment process much more complicated.
- pas 9y ago
  Not OP, but ... You can use Go/Rust for static linked bootstrap instead of Bash. (See rustup.) I'm thinking about combining GitLab (private token access + repository files: https://docs.gitlab.com/ce/api/repository_files.html#repository-files-api https://docs.gitlab.com/ce/api/repository_files.html#reposit... ) and a bootstrap script in Bash, that then launches something better. Plus self registration back into a GitLab repo (hey, it has the access token already, so it can push). Yes, Ansible/SaltStack/Chef/Puppet reinvented, but you're now not bound to the horrible database, language, bootstrap, speed (slowness) and workflow of any of them. If you want more access control, then creating a HTTP microservice that has the real private token (or proper OAuth2 access) to GitLab plus handles the self-registration and token handout/revocation is easy compared to the other parts. (And it can also store everything in GitLab.)
- yeukhon 9y ago
  I believe in your userdata script you can write a line to download a Python script from S3, then proceed to execute. Don't quote me on this since I don't use userdata. However, note that userdata will only run once at launch time. If you ever need to re-run userdata, you have to stop the instance, remove the userdata from settings, and then add again. YMMV. I do the Python way when launching a beanstalk instance (basically cloud init).
_kst_ 9y ago
I'd make the limit a lot more than 10 lines. I certainly prefer Perl for large scripts (Python is good too but I happen to know Perl better), but I often have to write scripts for targets that have bash but don't have Perl or Python. Since I've been forced to use bash, I've found it to be a better scripting language than I expected it to be.
- sethrin 9y ago
  10 lines is a rule which is meant to be broken. Rules exist to guide the novice, and to inform the expert. Bash is an extremely useful tool, but not one which should be employed carelessly or casually.

Ancalagon 9y ago

Amen to that. Now if only I could have gotten my Systems Programming professor to feel the same way... :)

chucknelson 9y ago

Would you consider non-trivial install scripts as an exception to this general rule? I mean, you wouldn't write some install script in ruby or python, right?

ghettoimp 9y ago
I don't know why not? At the least, rather than Bash, you might consider Perl as a default, lowest common denominator for scripts that need to run anywhere. - It's nearly as ubiquitous as bash. - It has approximately the same kinds of file/path operations built in. - It has reasonably good support for strings/regexes/etc. all built-in, so you don't have to call out to tools like sed/awk/grep all the time and hope that they are available and compatible across your target platforms. - It provides reasonably good arrays and hashes, which are horribly horrible in bash.[1] - You can use syscalls very easily if you really need to, but usually you don't. [1] Of course, no language can save you from the file system disaster (https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames....), but being able to know that "foo bar" is a string instead of two array elements is a good start. Mostly this all applies to Ruby or Python too, modulo perhaps the degree of ubiquity.
- gbacon 9y ago
  Great point about ubiquity. Perl regexes are the best of breed that everyone else replicates — far better than “reasonably good.” The Perl erasure in this HN thread is startling.
  - pletnes 9y ago
    Perl erasure? After using the linux command line (or its many «relatives» like cygwin, mac os, unixes) for more than 10 years now, I’ve talked to exactly one person that used perl to accomplish anything at all. He used it to edit text files, so he could have done the same in awk/sed/vim in my opinion. I know more people who write fortran 77 than perl. People just don’t seem to use (or like) Perl very much.
  - grzm 9y ago
    > "The Perl erasure in this HN thread is startling." "Erasure" to me implies some active effort to remove Perl from discourse. I don't see anything like that here: indeed, there are a number of positive mentions, and no negative ones I see. Granted, Python and Ruby are both mentioned more often, but none of those is at Perl's expense. Am I misunderstanding what you mean by 'erasure'?
    - gbacon 9y ago
      This is what Perl is designed to be. Perl unlike the others is almost certain to be on any Unix or Linux installation. Several commenters leaving out Perl in discussions of the next step up from bash scripts is truly strange. I suppose being ignored beats the typical herp-derp anti-Perl bigotry, but I’d prefer all-around civility.
      - grzm 9y ago
        > "Several commenters leaving out Perl in discussions of the next step up from bash scripts is truly odd." I'm having a hard time following you here. Do you think that they're doing so for any other reason that Perl is no longer their go-to tool? There are communities where Perl is still used: PostgreSQL for example uses Perl for some of its scripting, as well as its build farm tool, in particular because of its portability on older systems. That said, from what I've seen over the past 10 years or so, Perl hasn't had much of a presence in areas where a lot of computer work in tech is being done. For example, in cloud computing, or scientific computing, or machine learning, or web frameworks. Please don't read this to mean that Perl couldn't be or isn't being used in these cases or wouldn't be a better fit. (As an aside, I think Perl missed out a lot while a large portion of the community was focused on Perl 6: there's only so much energy in a community, and that absorbed on Perl 6 wasn't focusing on evangelism. But that's not something I'm interested in litigating here.) Or that there isn't something a bit frustrating in seeing the wheel reinvented time and time again. And so many examples on the web use bash as a common denominator. This puts Perl further out of mind if it's not already part of your everyday workflow. And how many developers today have come of age without seeing Perl in their everyday environments? Consider the current forum. What's the percentage of front-page posts that are about Perl or tools where Perl is a part of the tool chain? It would be understandable for the people who frequent HN to not view Perl as their go-to. I don't consider it uncivil for people to neglect to mention some other language when it's not something they'd actually think of reaching for. It seems the solution would be to share examples of where Perl provides advantages, both in the comments here and in submissions to HN.
thomaslangston 9y ago
Yes, you would write non-trivial install scripts in ruby. https://en.m.wikipedia.org/wiki/Chef_(software) https://en.m.wikipedia.org/wiki/Chef_(software)
- vinceguidry 9y ago
  If any of my bash scripts get long enough to where I want a reusable class, I rewrite in Ruby.
khedoros1 9y ago
Bash has too many surprising edge cases. A lot of my install scripts have at least snippets of Perl in them. The installation/upgrade/maintenance scripts for my employer's main product is Ruby-based. We support a lot of different OSes, and there's usually less variance between the Perl deployed on them than there is in the shell.
sethrin 9y ago
I would not consider those an exception, and would typically prefer to see the use of some other language. Bash is excellent at dealing with semi-structured text, and as part of a command pipeline, and if you have no alternative than to write POSIX sh, well, it exists. Bash does not have niceties like typed variables or named function arguments, and arrays are best avoided. Even parsing command line arguments is more fun in other languages. The problem is that Bash has about the lowest barrier to entry which can be found: just dump the things you were going to type anyway into a text file and mark it executable. It's simple, and then you bloody your nose on one of Bash's many idiosyncrasies, and people will say, "Oh yes, ']' is just a required last argument to '['. You gotta watch for that." And the true Bash master knows that my rule is silly and that anything may be written in Bash. Just...please don't.
- mbrock 9y ago
  I say please don't write any goddamned software at all.
weaksauce 9y ago
Isn't that what chef and Capistrano are at some level?

Goladus 9y ago

I disagree. Here's how I decide: Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash to its limits and I tend to set the bar fairly low here. Is the program oriented around commands? If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language. Argument passing is more natural and convenient and the built-in support for the standard i/o streams makes it easy for the different commands to pass data between them. The number of variables or lines of logic is usually less important than the number of different installed commands I need to combine. (Or the number of variations on a single command) Do I need to modify the environment in a significant way? In a bash script it's trivial to source in an environment script, which can be done conditionally or even interactively. This is usually more tedious to do in scripting languages.

ilyash 9y ago
[shameless] > Do I need to manipulate rich data structures like hash-maps or nested lists? That sort of thing tends to stretch the capabilities of Bash > If I'm gluing executable scripts and binaries, using bash is often superior to a scripting language The two points above resonate with my view that there is a missing piece. On one hand we have bash which is optimized for being the glue (second point). On the other hand we have Ruby, Python, Perl, Go, etc which are good for first point. What I think is missing is newer, more powerful shell, which supports both use cases and more. I'm working on it: https://github.com/ilyash/ngs https://github.com/ilyash/ngs Please note that I'm not the only one that thinks there is room for more powerful shells. See the readme for links to other projects.
falsedan 9y ago
Also: do I need to run a producer & consumer (and maybe some intermediary filters) on a stream of data, and want to run them in parallel? Shell pipelines are trivially easy to set up & test.
kinkrtyavimoodh 9y ago
As someone who has to occasionally modify 100+ line bash scripts written by Coworkers from Christmas Past which matched your spec in terms of what they had to do, please please just use Python (or similar). Yes, you will have a few extra lines but it will be vastly more readable and maintainable. And yes, I know I will get the standard the person who wrote the script did a bad job but at some point it should be okay to blame the tools instead of the workman if workmen disproportionately create worse results with a set of tools.

JepZ 9y ago

I disagree too. Bash has many downsides, but there are very few languages out there which have the ability to 'connect' different programs so easily. Bash scripts are slow as hell (as most commands have to spawn new processes), it is hard to write "secure" code (if even possible), handling whitespaces can be a pain in the * and the amount of repetition is awful. If your kid has done something wrong, just tell it to write a bash script: it is the equivalent of writing a hundred times: x="$(...)" Nevertheless, I enjoy writing bash scripts. Many times it starts with a simple curl command and by the end of the day you have a new OS installer. Granted, there are tasks which other languages can do better, but thats the real power of bash, it doesn't care: Then off you go write your super complicated algorithm in Rust, Go, Python, R or whatever and just call the other program, that's what Bash is good at. It is the glue which keeps everything working together. Disclaimer: Please don't build a complete cathedral out of glue.

dgsb 9y ago
I'd rather use perl for these use cases. It's almost as concise as bash regarding subprocess management. It's ubiquitous as bash, and have the python like 2 vs 3 version issue.
joncrane 9y ago
"Bash is the glue which keeps everything working together. Please don't build a complete cathedral out of glue." --JepZ
IshKebab 9y ago
The reason it can connect programs so "easily" is because it basically ignores errors and robustness - it really relies on there being a user looking at the output and going "hmm that looked like it failed". Doing things properly in Python or Go may take a few more lines (not much more really) but it is 100 tones more robust, and you need that if you are writing anything more than a 10-line one-off hack.
- mbrock 9y ago
  No, it doesn't. Unix error codes are extremely well understood. Tell the Git maintainers that their program ignores errors and robustness—they'll be surprised.

shrumm 9y ago

I think the 10 line limit seems a little harsh. Bash is great to prototype something, especially because of all the great commands at your disposal versus writing it yourself. e.g. grep, sed, cut are all commands I use frequently. Once you've got a working script and it proves useful or needs to scale - that's a good time to go the 'real language' route like Python.

bch 9y ago
> I think the 10 line limit seems a little harsh. I think it's a fine guide. If I'm writing something that starts approaching a program rather than a few lines of utility-throw-away, it's time to at least immediately start co-developing in something more sane. My exception is for Makefiles, if one considers them "shell".

joncrane 9y ago

How do you feel about the use of bash for AWS userdata and the like? You pretty much have to put all the init stuff in a bash script. I can't think of any real-world examples of userdata scripts being less than 10 lines. The alternative of course would be a three line script that downloads a file and executes it. But what language would that file be in? Probably bash. And it would make the infrastructure-as-code tracking and deployment process much more complicated.

pas 9y ago
Not OP, but ... You can use Go/Rust for static linked bootstrap instead of Bash. (See rustup.) I'm thinking about combining GitLab (private token access + repository files: https://docs.gitlab.com/ce/api/repository_files.html#repository-files-api https://docs.gitlab.com/ce/api/repository_files.html#reposit... ) and a bootstrap script in Bash, that then launches something better. Plus self registration back into a GitLab repo (hey, it has the access token already, so it can push). Yes, Ansible/SaltStack/Chef/Puppet reinvented, but you're now not bound to the horrible database, language, bootstrap, speed (slowness) and workflow of any of them. If you want more access control, then creating a HTTP microservice that has the real private token (or proper OAuth2 access) to GitLab plus handles the self-registration and token handout/revocation is easy compared to the other parts. (And it can also store everything in GitLab.)
yeukhon 9y ago
I believe in your userdata script you can write a line to download a Python script from S3, then proceed to execute. Don't quote me on this since I don't use userdata. However, note that userdata will only run once at launch time. If you ever need to re-run userdata, you have to stop the instance, remove the userdata from settings, and then add again. YMMV. I do the Python way when launching a beanstalk instance (basically cloud init).

_kst_ 9y ago

I'd make the limit a lot more than 10 lines. I certainly prefer Perl for large scripts (Python is good too but I happen to know Perl better), but I often have to write scripts for targets that have bash but don't have Perl or Python. Since I've been forced to use bash, I've found it to be a better scripting language than I expected it to be.

sethrin 9y ago
10 lines is a rule which is meant to be broken. Rules exist to guide the novice, and to inform the expert. Bash is an extremely useful tool, but not one which should be employed carelessly or casually.