10 ms·

I'm glad this included the "Signs you should not be using a bash script" section. Bash is a very good solution for many cases, but it becomes downright unruly w

I'm glad this included the "Signs you should not be using a bash script" section. Bash is a very good solution for many cases, but it becomes downright unruly when dealing with a lot of string manipulation and more complex objects.

chubot 12y ago
your script is longer than a few hundred lines of code you need data structures beyond simple arrays you have a hard time working around quoting issues you do a lot of string manipulation you do not have much need for invoking other programs or pipe-lining them you worry about performance It's not a sign you shouldn't be using bash. It's a sign you shouldn't be using ONLY bash. People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man). You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes. You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution. Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.
- japhyr 12y ago
  I think I might have accidentally down voted you. That was helpful feedback for me.
  - chubot 12y ago
    No problem... the way I think about is that Python/Perl/Ruby are for data structures, logic and control flow; while shell is for configuration and data flow. As mentioned above, things that belong in shell, and NOT in Python: - file system paths (these are parameters to Python scripts, not hard coded) - port numbers - user names - other config values - concurrency -- you can get really far with xargs -P, and sometimes & - pipelines (try doing this in Python without shelling out; it's horrible) Things that belong in Python and NOT shell: - complex control flow (e.g. I rarely write loops in bash; never a nested loop) - data structures (bash has hash tables, but I never use them) - arithmetic - parsing and cleaning individual lines or fields of data Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.
    - rix0r 12y ago
      File system paths may belong in the shell, but what about paths with spaces? To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(
      - chubot 12y ago
        Here are the quoting rules I use: - Avoid paths with spaces :) - If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". - Always use "$@". I've never found a reason to use $@ or $* unquoted. I also write unit tests (in shell) if there is something unusual I know I have to handle. That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple. There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code. I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.
        onli 12y ago
        > If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". I'd like to add: Always use double quotes around paths. Or is there something I'm missing here?
        claudius 12y ago
        $ ls -l "*" ls: cannot access *: No such file or directory Of course, you might want that exact behaviour, but sometimes you want e.g. all .table files in a path with a space. I believe $ ls -l "path with/lots of spaces/"* is the correct solution, but it seems silly. I'm not entirely sure why variable expansion works inside double quotes but wildcard expansion doesn’t.
        elwin 12y ago
- nikatwork 12y ago
  Modularity is nice, but it's generally easier to debug a program in a single context. I work with some deployment systems that chain together small scripts in a bunch of different languages. They are a nightmare to troubleshoot. I'd much rather the spaghetti was all on one plate than follow it from table to table...
  - chubot 12y ago
    If the tools are coherently designed, it should be easier to debug, because you can just use -x to log the commands being run and paste them in to see what went wrong. It's debugging with a REPL. The biggest mistake I see people making is to hard code paths, ports, user names, configuration, etc. inside Python/Perl scripts. All that stuff belongs in shell. All the logic and control flow goes in the scripts. If it's factored this way, then you have a very natural way of testing the scripts with test parameters (i.e. not production parameters). I don't doubt that there are many multi language shell scripts that are spaghetti. Factoring into processes is definitely a skill that takes thought and effort, and I don't think anyone really teaches it or writes about it. The only books I can think of know of is The Art of Unix Programming by ESR and the Unix Programming Environment. But it's definitely made me way more productive once I started thinking like this. People say talk about the "Unix philosophy" for a reason. It's a real thing :) It's not an accident that Unix is wildly popular and has unprecedented longevity.
    - deleted 12y ago
      [deleted]
  - Toenex 12y ago
    I'm glad someone mentioned debugging. Typically Bash scripts evolve into programs and one of the first things I always notice is how much effort I'm having to put into debugging. Indeed, since I started working with [plumbum](http://plumbum.readthedocs.org/en/latest/ http://plumbum.readthedocs.org/en/latest/) I now typically reach for python in favour of bash even for small jobs.
  - Too 12y ago
    Yup, until someone can show me a bash editor that understands all the processes that are invoked and gives me at least context aware code completion, parameter documentation, "go to definition" and "find all references" I'd rather keep my code in a language where I can get those features.
- barrkel 12y ago
  This is how I work. Whenever possible, when I have to write a utility in whatever language, I write it such that it can easily be integrated into shell pipelines. That makes it far easier to glue multiple tools written in different languages together.
- __david__ 12y ago
  > …awkwardly and verbosely reimplements 'xargs -P'… Verbose, and almost guaranteed to be more accurate. The thing about python/ruby/perl is that when you do things the programmatic way you get safety by default. You don't have to remember to 'find -0 | xargs -0' and thing aren't going to bite you in the butt when filenames have spaces in them. Yes, you can make all that work with shell (heaven knows I've done it), but it's more work and sadly isn't shell's default state.
- anon4 12y ago
  To add, if you need/want to you can write python and perl inline in bash like so: python3 - <<'END' print('hello world') END The advantage is that you still have only one shell script to ship, which makes sense when the python code inside it is specific to that one script. Remember to use the <<'END' variant so variable expansion isn't done inside the heredoc. On the other hand, $ doesn't do anything in python and if you're feeling like it (please make sure everybody else is also feeling like it) you can use variable substitution there, in effect generating a python script and executing it.
  - vacri 12y ago
    Playing around with it, I see that using <<'END' does not expand $variables in the block, but <<END does expand them. My google-fu is failing me; what's happening here?
    - anon4 12y ago
      It's two different syntaxes. Like the difference between '$VAR' (no expansion, literal string) and "$VAR" (with expansion). See http://tldp.org/LDP/abs/html/quoting.html http://tldp.org/LDP/abs/html/quoting.html http://tldp.org/LDP/abs/html/here-docs.html http://tldp.org/LDP/abs/html/here-docs.html
yason 12y ago
I generally, whatever I'm writing, start with bash or considering bash. Usually that means I start with bash, excluding the obvious cases such as graphics or doing heavy data processing where I know from start that line-based approach isn't enough. Then, when bash isn't enough, I write the difficult parts in Python as standalone utilities and use those from bash. Consequently, I already have a library of little tools in my ~/bin that are generic enough so that I can just use them from bash right away. But every now and then I need to write a special tool that my bash program can use. Most of my programs "finish" after this stage and they can live for years as a completed bash script calling helper programs. They do their job and there's nothing to add. As long as the program continues to "live on" I just make changes to the bash script or the helper programs. If at some point I need features that bash can't offer, such as more speed or more complex data processing, I begin to consider rewriting my program in one language. But this is only after the first revision of the program is already complete and I know exactly what to do when porting. Thus, porting the program is initially only about rewriting in another language, not developing it further at the same time. The first months or years of using the bash script turn into a prototyping stage, and this allows my rewrite be concise, well-thought and clear-partitioned. In other words, the rewrite becomes a new stable baseline that contains only the best of my bash prototyping, and only solves a perfectly scoped problem. As soon as the port becomes a drop-in replacement for my original bash program I switch to using it and start a development branch to support the new features, if any. Usually the newly gained speed is enough already. And usually the rewrite happens using C or Scheme. There's rarely enough point in rewriting the script in Python which may already be used to some extent in utility programs that the script calls. At that point I don't want to extend the program but lift it onto a new platform to cut some old baggage that's not needed anymore. There's a certain kind of joy rewriting in a new language something that you know completely before hand. This allows you to only work with the language since the underlying problem is already solved. Coincidentally, this new C or Scheme program might then, one day in the future, be called from a new bash script...
- ejstronge 12y ago
  Do you have any examples of a tool/workflow that's made its way from bash to bash+python to C?
- deleted 12y ago
dllthomas 12y ago
God yes. I spent a year maintaining ~90k lines of bash. Interesting experience...
- migrantgeek 12y ago
  up-voted just to ease the pain
  - dllthomas 12y ago
    Thanks, man. Actually, it was kinda neat for the first 3 months or so, to have reason to dive deep and flesh out the dark corners of my bash knowledge. There's a few things I picked up that have served me well since - and I'd lived in the shell for years beforehand... The balance wasn't kinda neat.
dvirsky 12y ago
I'd go as far as saying "having more than a dozen lines". Bash syntax for anything other than basic execution and piping a few commands, is a disaster IMHO. Anything that requires logic should be written in a "real" scripting language. Python is my choice when I need a bit of logic in my scripts, but it's just a personal choice of course. A good addition for simplifying execution is the "sh" library that lets you call system commands easily as if they were python functions: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/
- massysett 12y ago
  Agreed; I would also say that you should run from Bash not when you need anything more than a simple array, but when you need any kind of array, period. The advice on functions also gives me pause because that helps you write longer Bash scripts...not a good idea. Functions are of limited use because they can't really return values (just an exit status.) If a function has to return something, it generally prints it to stdout...which requires use of command substitution to get the result...nasty. Shells are good for interactive use and for starting other programs, but it's a lot easier to take a full-featured language and make it work like a shell than it is to take a shell and make it behave like a full-featured language. I'd take this blog post, add considerably to the "when you shouldn't use Bash" list, and put that at the top of the blog post, and then say "but if you absolutely must use Bash for something that is not trivial, here are some things that can make the process manageably nasty instead of truly horrid."

chubot 12y ago

your script is longer than a few hundred lines of code you need data structures beyond simple arrays you have a hard time working around quoting issues you do a lot of string manipulation you do not have much need for invoking other programs or pipe-lining them you worry about performance It's not a sign you shouldn't be using bash. It's a sign you shouldn't be using ONLY bash. People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man). You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes. You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution. Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.

japhyr 12y ago
I think I might have accidentally down voted you. That was helpful feedback for me.
- chubot 12y ago
  No problem... the way I think about is that Python/Perl/Ruby are for data structures, logic and control flow; while shell is for configuration and data flow. As mentioned above, things that belong in shell, and NOT in Python: - file system paths (these are parameters to Python scripts, not hard coded) - port numbers - user names - other config values - concurrency -- you can get really far with xargs -P, and sometimes & - pipelines (try doing this in Python without shelling out; it's horrible) Things that belong in Python and NOT shell: - complex control flow (e.g. I rarely write loops in bash; never a nested loop) - data structures (bash has hash tables, but I never use them) - arithmetic - parsing and cleaning individual lines or fields of data Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.
  - rix0r 12y ago
    File system paths may belong in the shell, but what about paths with spaces? To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(
    - chubot 12y ago
      Here are the quoting rules I use: - Avoid paths with spaces :) - If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". - Always use "$@". I've never found a reason to use $@ or $* unquoted. I also write unit tests (in shell) if there is something unusual I know I have to handle. That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple. There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code. I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.
      - onli 12y ago
        > If you have to handle paths with spaces, use double quotes everywhere, e.g. "$pidfile". I'd like to add: Always use double quotes around paths. Or is there something I'm missing here?
        claudius 12y ago
        $ ls -l "*" ls: cannot access *: No such file or directory Of course, you might want that exact behaviour, but sometimes you want e.g. all .table files in a path with a space. I believe $ ls -l "path with/lots of spaces/"* is the correct solution, but it seems silly. I'm not entirely sure why variable expansion works inside double quotes but wildcard expansion doesn’t.
        elwin 12y ago
nikatwork 12y ago
Modularity is nice, but it's generally easier to debug a program in a single context. I work with some deployment systems that chain together small scripts in a bunch of different languages. They are a nightmare to troubleshoot. I'd much rather the spaghetti was all on one plate than follow it from table to table...
- chubot 12y ago
  If the tools are coherently designed, it should be easier to debug, because you can just use -x to log the commands being run and paste them in to see what went wrong. It's debugging with a REPL. The biggest mistake I see people making is to hard code paths, ports, user names, configuration, etc. inside Python/Perl scripts. All that stuff belongs in shell. All the logic and control flow goes in the scripts. If it's factored this way, then you have a very natural way of testing the scripts with test parameters (i.e. not production parameters). I don't doubt that there are many multi language shell scripts that are spaghetti. Factoring into processes is definitely a skill that takes thought and effort, and I don't think anyone really teaches it or writes about it. The only books I can think of know of is The Art of Unix Programming by ESR and the Unix Programming Environment. But it's definitely made me way more productive once I started thinking like this. People say talk about the "Unix philosophy" for a reason. It's a real thing :) It's not an accident that Unix is wildly popular and has unprecedented longevity.
  - deleted 12y ago
    [deleted]
- Toenex 12y ago
  I'm glad someone mentioned debugging. Typically Bash scripts evolve into programs and one of the first things I always notice is how much effort I'm having to put into debugging. Indeed, since I started working with [plumbum](http://plumbum.readthedocs.org/en/latest/ http://plumbum.readthedocs.org/en/latest/) I now typically reach for python in favour of bash even for small jobs.
- Too 12y ago
  Yup, until someone can show me a bash editor that understands all the processes that are invoked and gives me at least context aware code completion, parameter documentation, "go to definition" and "find all references" I'd rather keep my code in a language where I can get those features.
barrkel 12y ago
This is how I work. Whenever possible, when I have to write a utility in whatever language, I write it such that it can easily be integrated into shell pipelines. That makes it far easier to glue multiple tools written in different languages together.
__david__ 12y ago
> …awkwardly and verbosely reimplements 'xargs -P'… Verbose, and almost guaranteed to be more accurate. The thing about python/ruby/perl is that when you do things the programmatic way you get safety by default. You don't have to remember to 'find -0 | xargs -0' and thing aren't going to bite you in the butt when filenames have spaces in them. Yes, you can make all that work with shell (heaven knows I've done it), but it's more work and sadly isn't shell's default state.
anon4 12y ago
To add, if you need/want to you can write python and perl inline in bash like so: python3 - <<'END' print('hello world') END The advantage is that you still have only one shell script to ship, which makes sense when the python code inside it is specific to that one script. Remember to use the <<'END' variant so variable expansion isn't done inside the heredoc. On the other hand, $ doesn't do anything in python and if you're feeling like it (please make sure everybody else is also feeling like it) you can use variable substitution there, in effect generating a python script and executing it.
- vacri 12y ago
  Playing around with it, I see that using <<'END' does not expand $variables in the block, but <<END does expand them. My google-fu is failing me; what's happening here?
  - anon4 12y ago
    It's two different syntaxes. Like the difference between '$VAR' (no expansion, literal string) and "$VAR" (with expansion). See http://tldp.org/LDP/abs/html/quoting.html http://tldp.org/LDP/abs/html/quoting.html http://tldp.org/LDP/abs/html/here-docs.html http://tldp.org/LDP/abs/html/here-docs.html

yason 12y ago

I generally, whatever I'm writing, start with bash or considering bash. Usually that means I start with bash, excluding the obvious cases such as graphics or doing heavy data processing where I know from start that line-based approach isn't enough. Then, when bash isn't enough, I write the difficult parts in Python as standalone utilities and use those from bash. Consequently, I already have a library of little tools in my ~/bin that are generic enough so that I can just use them from bash right away. But every now and then I need to write a special tool that my bash program can use. Most of my programs "finish" after this stage and they can live for years as a completed bash script calling helper programs. They do their job and there's nothing to add. As long as the program continues to "live on" I just make changes to the bash script or the helper programs. If at some point I need features that bash can't offer, such as more speed or more complex data processing, I begin to consider rewriting my program in one language. But this is only after the first revision of the program is already complete and I know exactly what to do when porting. Thus, porting the program is initially only about rewriting in another language, not developing it further at the same time. The first months or years of using the bash script turn into a prototyping stage, and this allows my rewrite be concise, well-thought and clear-partitioned. In other words, the rewrite becomes a new stable baseline that contains only the best of my bash prototyping, and only solves a perfectly scoped problem. As soon as the port becomes a drop-in replacement for my original bash program I switch to using it and start a development branch to support the new features, if any. Usually the newly gained speed is enough already. And usually the rewrite happens using C or Scheme. There's rarely enough point in rewriting the script in Python which may already be used to some extent in utility programs that the script calls. At that point I don't want to extend the program but lift it onto a new platform to cut some old baggage that's not needed anymore. There's a certain kind of joy rewriting in a new language something that you know completely before hand. This allows you to only work with the language since the underlying problem is already solved. Coincidentally, this new C or Scheme program might then, one day in the future, be called from a new bash script...

ejstronge 12y ago
Do you have any examples of a tool/workflow that's made its way from bash to bash+python to C?
deleted 12y ago

dllthomas 12y ago

God yes. I spent a year maintaining ~90k lines of bash. Interesting experience...

migrantgeek 12y ago
up-voted just to ease the pain
- dllthomas 12y ago
  Thanks, man. Actually, it was kinda neat for the first 3 months or so, to have reason to dive deep and flesh out the dark corners of my bash knowledge. There's a few things I picked up that have served me well since - and I'd lived in the shell for years beforehand... The balance wasn't kinda neat.

dvirsky 12y ago

I'd go as far as saying "having more than a dozen lines". Bash syntax for anything other than basic execution and piping a few commands, is a disaster IMHO. Anything that requires logic should be written in a "real" scripting language. Python is my choice when I need a bit of logic in my scripts, but it's just a personal choice of course. A good addition for simplifying execution is the "sh" library that lets you call system commands easily as if they were python functions: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/

massysett 12y ago
Agreed; I would also say that you should run from Bash not when you need anything more than a simple array, but when you need any kind of array, period. The advice on functions also gives me pause because that helps you write longer Bash scripts...not a good idea. Functions are of limited use because they can't really return values (just an exit status.) If a function has to return something, it generally prints it to stdout...which requires use of command substitution to get the result...nasty. Shells are good for interactive use and for starting other programs, but it's a lot easier to take a full-featured language and make it work like a shell than it is to take a shell and make it behave like a full-featured language. I'd take this blog post, add considerably to the "when you shouldn't use Bash" list, and put that at the top of the blog post, and then say "but if you absolutely must use Bash for something that is not trivial, here are some things that can make the process manageably nasty instead of truly horrid."