11 ms·

Bashing the Bash – Replacing Shell Scripts with Python

43 points by tdurden 9y ago

visarga 9y ago
I think Bash is much more than that. For example, I mostly compose chains of commands through pipes. This extension is not pipe-centric.
- robinhoodexe 9y ago
  This so much. Using grep, cat, cut and sed to parse log files (in my case output from quantum chemical calculations) is really nice.
- pfranz 9y ago
  I think the scope for Bash has shrunk a lot since Perl and moreso Python have become widely adopted. The rule of thumb I usually have is "if it's more than 50 lines, reevaluate if shell scripting is the best language." I was working at a place that had been around since the early 80s. It looks like the "official" scripting language was tcsh until around the early 2000s, then Perl was preferred, and around the late 2000s they switched to Python. I ported a bunch of tcsh scripts that missed their Perl transition and was I impressed (I didn't often see "structured" shell scripts with actual design patterns) and the fact they were still used daily. I feel like my Python version had less boilerplate, better error checking, and was more legible. If you look at the OS itself, a lot of what used to be done with shell scripts migrated to Python2 years ago (which made the move to Python3 tougher)--but the scripts are more legible and maintainable. I do still love the challenge of running a command and trying to get my result in one line.
dguo 9y ago
This reminds me of the Google guidelines[1] for when to use Bash. Between Python's extensive standard library and the ability to call shell commands[2] if necessary, I rarely reach for Bash. [1]: https://google.github.io/styleguide/shell.xml?showone=When_to_use_Shell#When_to_use_Shell https://google.github.io/styleguide/shell.xml?showone=When_t... [2]: https://stackoverflow.com/questions/89228/calling-an-external-command-in-python https://stackoverflow.com/questions/89228/calling-an-externa...
rightisleft 9y ago
I think the author forgets to mention that the primary reason bash is used is because of its cross distribution support. I known if i write my script in bash it will work on any small linux distro. I think python is a close second, but it still use bash to bootstrap my environment for my real work.
- SwellJoe 9y ago
  I've been using POSIX shell exclusively, and spent a week or so converting all of my existing scripts to POSIX a few years ago (when Ubuntu switched its initscripts to run under dash instead of bash). Bash isn't available on tiny Linux distros by default, because it's quite large. Many use dash, including most busybox based distributions. With checkbashisms and shellcheck, it's pretty easy to spot compatibility issues and fix them.
  - yjftsjthsd-h 9y ago
    And if you're POSIX compliant, you get to target every POSIX platform (GNU/Linux, BSDs, MacOS, GNU/NT) for free:)
    - SwellJoe 9y ago
      It's not quite free, in my experience, but yes, it is more cross-platform than bash. The price you still have to pay for cross-platform is the different paths, different output formats for system commands, etc. This is particularly pronounced for the stuff most of my lines of shell scripts are used for, where it's interacting almost entirely with system utilities to figure out where it's running and to do things to make sure the system is able to run the software my scripts are installing (which also interacts with the system in intimate and intrusive ways). But, you'd pay this with just about any programming language. Then again, I also use shell scripts for building/deploying things where a makefile would be overkill (and of course makefiles interact with the shell, too, so building your makefiles for POSIX shell rather than bash is also a good practice), and I make those with POSIX, as well. And, in those cases, it would definitely be OK on damned near any platform. I even worked on Windows for a few months when I got my current laptop because HiDPI support and the graphics drives on Linux just weren't working well enough, and most of my stuff worked fine there...both in the WSL and from a git bash prompt.
brandonhsiao 9y ago
I like the idea, but that Python code doesn't look like "without much real work." The bash script is so much shorter.
chubot 9y ago
The point of my shell project Oil (http://www.oilshell.org/blog/ http://www.oilshell.org/blog/) is to get rid of the shell vs. Python debate. My claim is that, in general, X isn’t a good bash replacement, where X is Python/Ruby/Perl/JavaScript. I think that’s obvious to some people but not to others (typically X programmers who don’t know shell, which was me in the not-too-distant past). This article is good because it has a port of a realistic port from shell to Python. But to me it only succeeds in proving the opposite point -- that porting to Python is a lot of effort, and the result has a lot of detail that doesn't match the problem domain. Error handling is possible, which is good, but awkward. And now you need external librariesl like psutil, then a package manager and special deployment tools for Python. (And you probably need some shell scripts to build/deploy your Python...) One thing I don't see is the complete result, which from the looks of it is pretty clunky compared to the 20 or so lines of bash. ----- About Oil: I realized that not everyone was getting the point of it, so I resolved to write an “elevator pitch” [2]. I haven’t published that post yet, but the pitch is: Oil is the language you can convert bash scripts to automatically, once they become a maintenance problem That might happen at 50 lines for some people or 1000 lines for other people. Oil will have a more uniform syntax [6], not 4 different sublanguages [3]. It will get rid of word splitting, and have proper arrays [4] and hash tables. It should be easy to write a test framework in it, as opposed to bash test frameworks like bats [5] which actually modify/extend the language because it's not powerful enough. There are some cases where porting from bash to Python is a good idea. But I also had the revelation of porting Python to shell, and being relieved (e.g. more than one deployment script). I was a Python person and I came at it from the opposite side. Shell is really expressive, but it’s hampered horrible syntax and a bad/old implementation. In summary, the conundrum of porting things back and forth between bash <=> Python/Ruby/Perl/JavaScript is why Oil exists. It seems like this debate will never end, and I hope that the “automatic conversion” property [6] is a unique argument in favor of Oil vs. X. If anyone is skeptical of my elevator pitch I’m interested in the feedback :) You can also try the first release [7]. [1] http://www.oilshell.org/ http://www.oilshell.org/ [2] http://www.oilshell.org/blog/2017/07/31.html http://www.oilshell.org/blog/2017/07/31.html [3] http://www.oilshell.org/blog/2016/10/26.html http://www.oilshell.org/blog/2016/10/26.html [4] http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html [5] https://github.com/sstephenson/bats https://github.com/sstephenson/bats [6] http://www.oilshell.org/blog/2017/02/05.html http://www.oilshell.org/blog/2017/02/05.html [7] http://www.oilshell.org/blog/2017/07/23.html http://www.oilshell.org/blog/2017/07/23.html
chatmasta 9y ago
I love both bash and python. I almost always write the first version of a script in bash, especially if it involves piping between commands, managing processes, or moving files around the file system. Only when there is significant parsing involved do I reach for python -- and even then, it's often easier to just use python for the bit of parsing that's necessary, within a heredoc using python -c. The example script in this article hardly made use of pipes. I think if you tried to convert a shell script that used a lot of piping, the python might not come out so elegantly. As others have mentioned, bash, or at the very least shell, also has more cross platform support (certainly more than python3). Even on resource constrained or embedded systems, you're likely to have access to a shell, whereas python is not guaranteed to be installed. Both are good tools, and both excel in different ways; but I'm not prepared to throw out the baby with the bathwater and say bash is useless, always use python.
- deleted 9y ago
  [deleted]
grondilu 9y ago
> I’ve idealized the four steps as separate functions. Does the author know he can write functions in bash? I mean, if he wants to compare programming languages, he should at least compare similar programming styles. Bash is great for OS scripts because it has built-in commands and syntax for dealing with processus and their inter-communication (pipes and stuff). Python has NOTHING about that. If you want to do this, you have to use a module, read its documentation, and be prepared to write ugly, verbose method calls all around.
- l_t 9y ago
  The problem with Bash functions is that they aren't really functions -- more like subroutines. You can't even return a value from a Bash function (just a numeric status code). Most of the functions the author wrote wouldn't even be possible in Bash, if only because they utilize return values. So your suggestion that he "compare similar programming styles" is practically impossible without using implicit state-passing methods in Bash, which are much more awkward and error-prone than real functions. Having said that, I agree that writing scripts in Python adds a lot of overhead over Bash, and should only be done if you are implementing significant complicated logic more than just I/O. (For example, retries, exponential backoff, business logic, exception handling, etc.)
  - mbrock 9y ago
    You can output data from a function and pipe it just like any other program, right? So that's how you tend to return values from bash functions.
  - geofft 9y ago
    > You can't even return a value from a Bash function (just a numeric status code). You can print a textual result and store it in a variable, which is a pretty natural pattern in bash, e.g., set -e download_command () { if type wget >/dev/null 2>&1; then echo "wget -q -O-" elif type curl >/dev/null 2>&1; then echo "curl -sL" else echo "Error: curl or wget is required" >&2 exit 1 fi } download=$(download_command) public_v4=$($download http://whatismyip.akamai.com/) public_v6=$($download http://ipv6.whatismyip.akamai.com/) The numeric status code is best used to indicate errors (as above, with set -e), not to overload for data. I do agree that I'd prefer being able to return actual structured data and not strings -- but honestly you can pass e.g. JSON back-and-forth between commands that take JSON, and into `python -c 'import sys, json; something(json.loads(sys.stdin))'` if you need it for a small part of a script that's generally better written in sh.
skywhopper 9y ago
The key is knowing what to use where. Writing certain scripts in Bash is miles more efficient use of time than using python because it's "better". While it's certainly possible to write bad scripts in Bash like the example given, it's just as possible to write bad scripts in Python. All the time spent learning to do bash-like stuff in Python could just as well be spent learning to write better Bash. I think the readability of the program is key as well. It's interesting that the author never included the full Python program. Based on the snippets I would expect it was longer and was definitely far more verbose. One of Bash's advantages in terms of working with files and shell commands is that most people already know how to work with files and shell commands at the command prompt and so the clarity is already there. If you replace all that with Python library calls all the sudden you're speaking a different language and comprehension will be slower unless your reader works with those Python libs just as often as they use the command line. In other words, this may all sound good but be careful with this sort of advice. It's often counterproductive to go down this particular path
- binocarlos 9y ago
  There are times writing a bash script it feels like "omg - a programming language would be much easier for this task"... So I write a tiny program in the lang of my choice and invoke it from that place in the bash script. The balance between the two tools is great - your advice is spot on for me.
- yes_or_gnome 9y ago
  I work with a couple people who insistently write shell scripts in Python. I know it's anecdotal, but to me, it's like the metaphor 'When all you have is a hammer, everything looks like a nail.' The scripts use functions liberally, but the code isn't DRY nor self-documented. Written in shell, it would take a fifth of the LOC because there's no need to use subprocess or Popen and parse the output. They're not the best developers, but I certainly trust their Python code than I would trust their shell code. And, there's plenty of work to go around, so it's better than nothing.
  - remline 9y ago
dlandis 9y ago
I wonder why the author split up the Python program into functions but left the Bash program as just a single blob of code.
deleted 9y ago
[deleted]
ttd 9y ago
A while ago I wrote a tiny language (called bish) that was an attempt at finding a middle ground for this problem. Essentially, you write your shell scripts in bish, with sane syntax and semantics (example: function return values, which are not really present in bash). The bish compiler then compiles your script to bash. I haven't touched it in a few years now, but maybe it's of interest to someone: https://github.com/tdenniston/bish https://github.com/tdenniston/bish
mbrock 9y ago
Unit testing just means running your programs or subprograms with different inputs and checking the result. Nothing about this is hard in bash. Why would it be?
- octalmage 9y ago
  This is usually referred to as functional or acceptance testing. The goal of these types of tests is to confirm that the overall application works as expected from a users perspective. This is easy with bash. Testing individual units of code inside of a bash script is more difficult. This can be useful for pin pointing which function introduced a bug. Edit: want to add that difficult does not mean impossible. Bats exists, and you could structure a bash script to be more easily testable.
  - mbrock 9y ago
    Yeah, I just don't understand what actually makes it difficult.
    - octalmage 9y ago
      A big part of it might be lack of testing utilities. Things like mocking HTTP calls or stubbing methods will be difficult without modifying the code you're testing.
octalmage 9y ago
The thing other languages lack for me is a clean way to execute other programs. With bash it's just "grep whatever", in other languages it's "exec('grep whatever');". Then piping that output to another program is a lot of work. Bash makes for really good program glue. I've considered creating a library that scanned your PATH for binaries then created dynamic function signatures to allow calling the function by name. Then that dynamic module can be included, and you'll be able to grep by calling "grep". You could make it chainable and get something like: grep('something').cut('args'); or whatever. Of course this would only be used for personal utility scripts.
- catdog 9y ago
  Yes it's entirely possible to create a shell like "program glue" experience in python and somebody already has done it [1][2]. It's only the python standard library which is missing some convenient way to shell out. [1] http://plumbum.readthedocs.io/en/latest/ http://plumbum.readthedocs.io/en/latest/ [2] http://amoffat.github.io/sh/ http://amoffat.github.io/sh/
jwilk 9y ago
> The shell language’s only data structure is the string. Bash has arrays. > The expr program can convert strings to numbers to do arithmetic. You don't need to execute expr to do arithmetic: $ echo $((37 * 42)) 1554
carapace 9y ago
This is ignorant and unsubstantiated nonsense. I love Python and I don't enjoy shell scripting, but as a professional programmer I believe it's part of my job to understand bash and sh well enough. Especially if I'm going to throw around buzzy buzzwords like "devops". "Rewrite all your shell scripts in Python! Now you can ignore shell!" No. (I also have a hard time seeing the upside of e.g. Fabric. It's a Python wrapper around the shell, so now you have two things to learn: the actual commands that you need to execute, and how to express those commands in this bespoke Python-to-shell mapping.) Just learn shell. It's not going anywhere. You'll always have occasion to use it. People will respect you. (Edit: One trick I like: If I'm writing a shell script an there's some fiddly bit of processing that is PITA to express in sh I'll write a tiny Python script that does it and call that from the shell script. Not great for performance, but usually it's a one-off or small batch so you don't care. Sometimes it's a small enough Python snippet that you can just use `python -c "..."`)
denfromufa 9y ago
Strange that no one has mentioned xonsh here: http://xon.sh http://xon.sh
- jpitz 9y ago
  Or Ammonite, for the strongly-typed crowd. http://ammonite.io/ http://ammonite.io/
binaryphile 9y ago
Here's what my take on the script would be in bash (albeit untested): https://gist.github.com/binaryphile/3cf01870516d5fffafbc84a054540160 https://gist.github.com/binaryphile/3cf01870516d5fffafbc84a0... Whoever wrote the initial bash script was clearly in a rush. I don't hold that against anyone, but it makes it poor fodder for an example of why bash is "bad". In particular, for the kind of task illustrated, it's a far better choice than python and I say that as a (formerly) unabashed pythonista. It takes some time to get chops in bash, especially since there's no culture of developing shareable libraries, but it's no harder to learn bash via stackoverflow/google searches than it is to learn python. Use python when you need data structures and objects. Use bash to automate what you would normally do on the command line. Is that so hard?
burntrelish1273 9y ago
There's nuance and opportunity for shell, Python, Ruby, C, etc. It's a good practice standardize on a couple of languages and treat executables as opaque processes with a limited, well-defined API/ABI. The advantage of scripting languages is reuse across scripts. There is a limited ability to reuse across shell scripts, but it's often somewhat awkward.

43 points by tdurden 9y ago

I think Bash is much more than that. For example, I mostly compose chains of commands through pipes. This extension is not pipe-centric.

robinhoodexe 9y ago
This so much. Using grep, cat, cut and sed to parse log files (in my case output from quantum chemical calculations) is really nice.
pfranz 9y ago
I think the scope for Bash has shrunk a lot since Perl and moreso Python have become widely adopted. The rule of thumb I usually have is "if it's more than 50 lines, reevaluate if shell scripting is the best language." I was working at a place that had been around since the early 80s. It looks like the "official" scripting language was tcsh until around the early 2000s, then Perl was preferred, and around the late 2000s they switched to Python. I ported a bunch of tcsh scripts that missed their Perl transition and was I impressed (I didn't often see "structured" shell scripts with actual design patterns) and the fact they were still used daily. I feel like my Python version had less boilerplate, better error checking, and was more legible. If you look at the OS itself, a lot of what used to be done with shell scripts migrated to Python2 years ago (which made the move to Python3 tougher)--but the scripts are more legible and maintainable. I do still love the challenge of running a command and trying to get my result in one line.

dguo 9y ago

This reminds me of the Google guidelines[1] for when to use Bash. Between Python's extensive standard library and the ability to call shell commands[2] if necessary, I rarely reach for Bash. [1]: https://google.github.io/styleguide/shell.xml?showone=When_to_use_Shell#When_to_use_Shell https://google.github.io/styleguide/shell.xml?showone=When_t... [2]: https://stackoverflow.com/questions/89228/calling-an-external-command-in-python https://stackoverflow.com/questions/89228/calling-an-externa...

rightisleft 9y ago

I think the author forgets to mention that the primary reason bash is used is because of its cross distribution support. I known if i write my script in bash it will work on any small linux distro. I think python is a close second, but it still use bash to bootstrap my environment for my real work.

SwellJoe 9y ago
I've been using POSIX shell exclusively, and spent a week or so converting all of my existing scripts to POSIX a few years ago (when Ubuntu switched its initscripts to run under dash instead of bash). Bash isn't available on tiny Linux distros by default, because it's quite large. Many use dash, including most busybox based distributions. With checkbashisms and shellcheck, it's pretty easy to spot compatibility issues and fix them.
- yjftsjthsd-h 9y ago
  And if you're POSIX compliant, you get to target every POSIX platform (GNU/Linux, BSDs, MacOS, GNU/NT) for free:)
  - SwellJoe 9y ago
    It's not quite free, in my experience, but yes, it is more cross-platform than bash. The price you still have to pay for cross-platform is the different paths, different output formats for system commands, etc. This is particularly pronounced for the stuff most of my lines of shell scripts are used for, where it's interacting almost entirely with system utilities to figure out where it's running and to do things to make sure the system is able to run the software my scripts are installing (which also interacts with the system in intimate and intrusive ways). But, you'd pay this with just about any programming language. Then again, I also use shell scripts for building/deploying things where a makefile would be overkill (and of course makefiles interact with the shell, too, so building your makefiles for POSIX shell rather than bash is also a good practice), and I make those with POSIX, as well. And, in those cases, it would definitely be OK on damned near any platform. I even worked on Windows for a few months when I got my current laptop because HiDPI support and the graphics drives on Linux just weren't working well enough, and most of my stuff worked fine there...both in the WSL and from a git bash prompt.

brandonhsiao 9y ago

I like the idea, but that Python code doesn't look like "without much real work." The bash script is so much shorter.

chubot 9y ago

The point of my shell project Oil (http://www.oilshell.org/blog/ http://www.oilshell.org/blog/) is to get rid of the shell vs. Python debate. My claim is that, in general, X isn’t a good bash replacement, where X is Python/Ruby/Perl/JavaScript. I think that’s obvious to some people but not to others (typically X programmers who don’t know shell, which was me in the not-too-distant past). This article is good because it has a port of a realistic port from shell to Python. But to me it only succeeds in proving the opposite point -- that porting to Python is a lot of effort, and the result has a lot of detail that doesn't match the problem domain. Error handling is possible, which is good, but awkward. And now you need external librariesl like psutil, then a package manager and special deployment tools for Python. (And you probably need some shell scripts to build/deploy your Python...) One thing I don't see is the complete result, which from the looks of it is pretty clunky compared to the 20 or so lines of bash. ----- About Oil: I realized that not everyone was getting the point of it, so I resolved to write an “elevator pitch” [2]. I haven’t published that post yet, but the pitch is: Oil is the language you can convert bash scripts to automatically, once they become a maintenance problem That might happen at 50 lines for some people or 1000 lines for other people. Oil will have a more uniform syntax [6], not 4 different sublanguages [3]. It will get rid of word splitting, and have proper arrays [4] and hash tables. It should be easy to write a test framework in it, as opposed to bash test frameworks like bats [5] which actually modify/extend the language because it's not powerful enough. There are some cases where porting from bash to Python is a good idea. But I also had the revelation of porting Python to shell, and being relieved (e.g. more than one deployment script). I was a Python person and I came at it from the opposite side. Shell is really expressive, but it’s hampered horrible syntax and a bad/old implementation. In summary, the conundrum of porting things back and forth between bash <=> Python/Ruby/Perl/JavaScript is why Oil exists. It seems like this debate will never end, and I hope that the “automatic conversion” property [6] is a unique argument in favor of Oil vs. X. If anyone is skeptical of my elevator pitch I’m interested in the feedback :) You can also try the first release [7]. [1] http://www.oilshell.org/ http://www.oilshell.org/ [2] http://www.oilshell.org/blog/2017/07/31.html http://www.oilshell.org/blog/2017/07/31.html [3] http://www.oilshell.org/blog/2016/10/26.html http://www.oilshell.org/blog/2016/10/26.html [4] http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html [5] https://github.com/sstephenson/bats https://github.com/sstephenson/bats [6] http://www.oilshell.org/blog/2017/02/05.html http://www.oilshell.org/blog/2017/02/05.html [7] http://www.oilshell.org/blog/2017/07/23.html http://www.oilshell.org/blog/2017/07/23.html

chatmasta 9y ago

I love both bash and python. I almost always write the first version of a script in bash, especially if it involves piping between commands, managing processes, or moving files around the file system. Only when there is significant parsing involved do I reach for python -- and even then, it's often easier to just use python for the bit of parsing that's necessary, within a heredoc using python -c. The example script in this article hardly made use of pipes. I think if you tried to convert a shell script that used a lot of piping, the python might not come out so elegantly. As others have mentioned, bash, or at the very least shell, also has more cross platform support (certainly more than python3). Even on resource constrained or embedded systems, you're likely to have access to a shell, whereas python is not guaranteed to be installed. Both are good tools, and both excel in different ways; but I'm not prepared to throw out the baby with the bathwater and say bash is useless, always use python.

deleted 9y ago
[deleted]

grondilu 9y ago

> I’ve idealized the four steps as separate functions. Does the author know he can write functions in bash? I mean, if he wants to compare programming languages, he should at least compare similar programming styles. Bash is great for OS scripts because it has built-in commands and syntax for dealing with processus and their inter-communication (pipes and stuff). Python has NOTHING about that. If you want to do this, you have to use a module, read its documentation, and be prepared to write ugly, verbose method calls all around.

l_t 9y ago
The problem with Bash functions is that they aren't really functions -- more like subroutines. You can't even return a value from a Bash function (just a numeric status code). Most of the functions the author wrote wouldn't even be possible in Bash, if only because they utilize return values. So your suggestion that he "compare similar programming styles" is practically impossible without using implicit state-passing methods in Bash, which are much more awkward and error-prone than real functions. Having said that, I agree that writing scripts in Python adds a lot of overhead over Bash, and should only be done if you are implementing significant complicated logic more than just I/O. (For example, retries, exponential backoff, business logic, exception handling, etc.)
- mbrock 9y ago
  You can output data from a function and pipe it just like any other program, right? So that's how you tend to return values from bash functions.
- geofft 9y ago
  > You can't even return a value from a Bash function (just a numeric status code). You can print a textual result and store it in a variable, which is a pretty natural pattern in bash, e.g., set -e download_command () { if type wget >/dev/null 2>&1; then echo "wget -q -O-" elif type curl >/dev/null 2>&1; then echo "curl -sL" else echo "Error: curl or wget is required" >&2 exit 1 fi } download=$(download_command) public_v4=$($download http://whatismyip.akamai.com/) public_v6=$($download http://ipv6.whatismyip.akamai.com/) The numeric status code is best used to indicate errors (as above, with set -e), not to overload for data. I do agree that I'd prefer being able to return actual structured data and not strings -- but honestly you can pass e.g. JSON back-and-forth between commands that take JSON, and into `python -c 'import sys, json; something(json.loads(sys.stdin))'` if you need it for a small part of a script that's generally better written in sh.

skywhopper 9y ago

The key is knowing what to use where. Writing certain scripts in Bash is miles more efficient use of time than using python because it's "better". While it's certainly possible to write bad scripts in Bash like the example given, it's just as possible to write bad scripts in Python. All the time spent learning to do bash-like stuff in Python could just as well be spent learning to write better Bash. I think the readability of the program is key as well. It's interesting that the author never included the full Python program. Based on the snippets I would expect it was longer and was definitely far more verbose. One of Bash's advantages in terms of working with files and shell commands is that most people already know how to work with files and shell commands at the command prompt and so the clarity is already there. If you replace all that with Python library calls all the sudden you're speaking a different language and comprehension will be slower unless your reader works with those Python libs just as often as they use the command line. In other words, this may all sound good but be careful with this sort of advice. It's often counterproductive to go down this particular path

binocarlos 9y ago
There are times writing a bash script it feels like "omg - a programming language would be much easier for this task"... So I write a tiny program in the lang of my choice and invoke it from that place in the bash script. The balance between the two tools is great - your advice is spot on for me.
yes_or_gnome 9y ago
I work with a couple people who insistently write shell scripts in Python. I know it's anecdotal, but to me, it's like the metaphor 'When all you have is a hammer, everything looks like a nail.' The scripts use functions liberally, but the code isn't DRY nor self-documented. Written in shell, it would take a fifth of the LOC because there's no need to use subprocess or Popen and parse the output. They're not the best developers, but I certainly trust their Python code than I would trust their shell code. And, there's plenty of work to go around, so it's better than nothing.
- remline 9y ago

dlandis 9y ago

I wonder why the author split up the Python program into functions but left the Bash program as just a single blob of code.

deleted 9y ago

[deleted]

ttd 9y ago

A while ago I wrote a tiny language (called bish) that was an attempt at finding a middle ground for this problem. Essentially, you write your shell scripts in bish, with sane syntax and semantics (example: function return values, which are not really present in bash). The bish compiler then compiles your script to bash. I haven't touched it in a few years now, but maybe it's of interest to someone: https://github.com/tdenniston/bish https://github.com/tdenniston/bish

mbrock 9y ago

Unit testing just means running your programs or subprograms with different inputs and checking the result. Nothing about this is hard in bash. Why would it be?

octalmage 9y ago
This is usually referred to as functional or acceptance testing. The goal of these types of tests is to confirm that the overall application works as expected from a users perspective. This is easy with bash. Testing individual units of code inside of a bash script is more difficult. This can be useful for pin pointing which function introduced a bug. Edit: want to add that difficult does not mean impossible. Bats exists, and you could structure a bash script to be more easily testable.
- mbrock 9y ago
  Yeah, I just don't understand what actually makes it difficult.
  - octalmage 9y ago
    A big part of it might be lack of testing utilities. Things like mocking HTTP calls or stubbing methods will be difficult without modifying the code you're testing.

octalmage 9y ago

The thing other languages lack for me is a clean way to execute other programs. With bash it's just "grep whatever", in other languages it's "exec('grep whatever');". Then piping that output to another program is a lot of work. Bash makes for really good program glue. I've considered creating a library that scanned your PATH for binaries then created dynamic function signatures to allow calling the function by name. Then that dynamic module can be included, and you'll be able to grep by calling "grep". You could make it chainable and get something like: grep('something').cut('args'); or whatever. Of course this would only be used for personal utility scripts.

catdog 9y ago
Yes it's entirely possible to create a shell like "program glue" experience in python and somebody already has done it [1][2]. It's only the python standard library which is missing some convenient way to shell out. [1] http://plumbum.readthedocs.io/en/latest/ http://plumbum.readthedocs.io/en/latest/ [2] http://amoffat.github.io/sh/ http://amoffat.github.io/sh/

jwilk 9y ago

> The shell language’s only data structure is the string. Bash has arrays. > The expr program can convert strings to numbers to do arithmetic. You don't need to execute expr to do arithmetic: $ echo $((37 * 42)) 1554

carapace 9y ago

This is ignorant and unsubstantiated nonsense. I love Python and I don't enjoy shell scripting, but as a professional programmer I believe it's part of my job to understand bash and sh well enough. Especially if I'm going to throw around buzzy buzzwords like "devops". "Rewrite all your shell scripts in Python! Now you can ignore shell!" No. (I also have a hard time seeing the upside of e.g. Fabric. It's a Python wrapper around the shell, so now you have two things to learn: the actual commands that you need to execute, and how to express those commands in this bespoke Python-to-shell mapping.) Just learn shell. It's not going anywhere. You'll always have occasion to use it. People will respect you. (Edit: One trick I like: If I'm writing a shell script an there's some fiddly bit of processing that is PITA to express in sh I'll write a tiny Python script that does it and call that from the shell script. Not great for performance, but usually it's a one-off or small batch so you don't care. Sometimes it's a small enough Python snippet that you can just use `python -c "..."`)

denfromufa 9y ago

Strange that no one has mentioned xonsh here: http://xon.sh http://xon.sh

jpitz 9y ago
Or Ammonite, for the strongly-typed crowd. http://ammonite.io/ http://ammonite.io/

binaryphile 9y ago

Here's what my take on the script would be in bash (albeit untested): https://gist.github.com/binaryphile/3cf01870516d5fffafbc84a054540160 https://gist.github.com/binaryphile/3cf01870516d5fffafbc84a0... Whoever wrote the initial bash script was clearly in a rush. I don't hold that against anyone, but it makes it poor fodder for an example of why bash is "bad". In particular, for the kind of task illustrated, it's a far better choice than python and I say that as a (formerly) unabashed pythonista. It takes some time to get chops in bash, especially since there's no culture of developing shareable libraries, but it's no harder to learn bash via stackoverflow/google searches than it is to learn python. Use python when you need data structures and objects. Use bash to automate what you would normally do on the command line. Is that so hard?

burntrelish1273 9y ago

There's nuance and opportunity for shell, Python, Ruby, C, etc. It's a good practice standardize on a couple of languages and treat executables as opaque processes with a limited, well-defined API/ABI. The advantage of scripting languages is reuse across scripts. There is a limited ability to reuse across shell scripts, but it's often somewhat awkward.