6 ms·

A follow-up recommendation I give (which I suspect might be unpopular with many around here) is to use Python for all but the most trivial one-liner scripts, in

A follow-up recommendation I give (which I suspect might be unpopular with many around here) is to use Python for all but the most trivial one-liner scripts, instead of shell. Since 3.5 added `subprocess.run` (https://docs.python.org/3/library/subprocess.html#subprocess.run https://docs.python.org/3/library/subprocess.html#subprocess...) it's really easy to write CLI-style scripts in Python. In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow. The equivalent Python for a script is seldom harder to understand, and as soon as you start doing any nontrivial logic it is (in my experience) always easier to understand. For example: subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status Combine this with `docopt` and you can very quickly and easily write helptext/arg parsing wrappers for your scripts; e.g. """Create a backup from a Google Cloud SQL instance, storing the backup in a Storage bucket. Usage: db_backup.py INSTANCE """ if __name__ == '__main__': args = docopt.docopt(__doc__) make_backup(instance=args['INSTANCE']) Which to my eyes is much easier to grok than the equivalent bash for providing help text and requiring args. There's an argument to be made that "shell is more universal", but my claim here is that this is actually false, and simple Python is going to be more widely-understood and less error-prone these days.

teknopaul 6y ago
for the record... if __name__ == '__main__': args = docopt.docopt(__doc__) means nowt to me. In fact I don't really understand your post and I've written a lot of python. No language has a simpler api to executing a cli command than bash itself. By definition.
- divbzero 6y ago
  The conditional expression __name___ == '__main__' simply tests whether this Python file is being executed from command line. It’s one of the ubiquitous Python idioms that doesn’t actually follow Python’s zen of “readability”.
  - theptip 6y ago
    Agreed on this point, it's the `public void static main` of Python. Perhaps I shouldn't have tried to make two points in one post; I wouldn't advocate using docopt or a main function for a simple script, I was more making the case for how easy it is to add proper parameter parsing when you need it; that is easier to remember than what you'd end up writing in bash, which is something like: PARAMS="" while (( "$#" )); do case "$1" in -a|--my-boolean-flag) MY_FLAG=0 shift ;; -b|--my-flag-with-argument) if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then MY_FLAG_ARG=$2 shift 2 else echo "Error: Argument for $1 is missing" >&2 exit 1 fi ;; -*|--*=) # unsupported flags echo "Error: Unsupported flag $1" >&2 exit 1 ;; *) # preserve positional arguments PARAMS="$PARAMS $1" shift ;; esac done # set positional arguments in their proper place eval set -- "$PARAMS" I think you've got to write a lot of bash before you can remember how to write `while (( "$#" )); do` off the top of your head; the double-brackets and [ vs ( are particularly error-prone pieces of syntax.
    - rmetzler 6y ago
      But the point of the article is not to make complicated scripts readable again, but to put even simpler commands (maybe even one long mysqldump command) into very short scripts. I agree that a cli framework is often easier to use than bash, but it also is a dependency. I think everyone should use what he’s familiar with.
    - wahern 6y ago
      Idiomatic shell code would be: while getopts "ab:" OPTC; do case "$OPTC" in a) MY_FLAG=0 ;; b) MY_FLAG_ARG="$OPTARG" ;; *) # shell already printed diagnostic to stderr exit 1 ;; esac done shift $((OPTIND - 1)) Yes, I realize it's more difficult to support long options (though not that difficult), but the best tool for the job will rarely check all the boxes. Anyhow, the arguments for long options are weakest in the case of simple shell scripts. (Above code doesn't handle mixed arguments either but GNU-style permuted arguments are evil. But unlike the Bash example the above code does support bundling, which is something I and many Unix users would expect to always be supported.) Also, I realize there's a ton of Bash code that looks exactly like you wrote. But that's on Google--in promoting Bash to the exclusion of learning good shell programming style they've created an army of Bash zombies who try to write Bash code like they would Python or JavaScript code, with predictable results.
- dllthomas 6y ago
  > No language has a simpler api to executing a cli command than bash itself. By definition. Things that are true "by definition" are not usually useful. "No language has a simpler api [...] [b]y definition" only if by "executing a cli command" we mean literally interpreting bash (or passing the string through to a bash process). But that's never the terminal[0] goal. The useful question is whether, for the task you might want to achieve, for which you might usually reach for the shell, is there in fact a simpler way. It may very well be that the answer is "no", but support for that is not "by definition". [0]: Edited to add: ugh. Believe it or not, this was not intended.
- m463 6y ago
  look above it to the line that says: Usage: db_backup.py INSTANCE then just go with it.
nicbou 6y ago
At my last company, I used Python for most of our scripts. Bash scripts were too tedious to write, harden and maintain.
yjftsjthsd-h 6y ago
How does this handle pipes and such? What's the nicest python equivalent to `grep ^$(whoami) /etc/passwd | awk -F : '{print $NF}'`?
- jniedrauer 6y ago
  Nothing is quite as succinct as shell. The python equivalent to this would probably be at least 10 lines of code.
  - tyingq 6y ago
    Everyone tends to rag on Perl, but it's excellent in this space. Edit: The equivalent of above would be: #!/usr/bin/perl my $shell=(getpwuid($>))[8]; print "$shell\n"; Though it does have simple syntax for pipes as well.
  - yjftsjthsd-h 6y ago
    https://news.ycombinator.com/item?id=24558719 https://news.ycombinator.com/item?id=24558719 does it in 5, which is, I think, proof that you're both right. (Of course, perhaps shell is cheating, since I chained 3 commands into that one-liner, and 2 of those are really their own languages with further subcommands.)
- johndough 6y ago
  Here are three options: # Option 1 (no pipes) import os for line in open("/etc/passwd").read().splitlines(): fields = line.split(":") if fields[0] == os.getlogin(): print(fields[-1]) # Option 2 (Cheating, but not really. For most problems worth solving, there exists a library to do the work with little code.) import os print(os.environ['SHELL']) # Option 3 (pipes, not sure if the utf-8 thing can be done nicer somehow) import subprocess username = subprocess.check_output("whoami", encoding="utf-8").rstrip() p1 = subprocess.Popen(["grep", "^" + username, "/etc/passwd"], stdout=subprocess.PIPE) p2 = subprocess.Popen(["awk", "-F", ":", "{print $NF}"], stdin=p1.stdout, stdout=subprocess.PIPE) print(p2.communicate()[0].decode("utf-8"))
freedomben 6y ago
As someone who hasn't used Python in ages, and finds it quite unreadable, I would be saddened and dismayed if instead of shell (which is the lingua franca) a project used Python everywhere. The exception being if it's a Python app in the first place, then it's fine since it can be assumed you need to be familiar with Python to hack on it anyway. For example I write a lot of Ruby scripts to include with Sinatra and Rails apps. For a general purpose approach however, everybody should learn basic shell. It's not that hard and it is universal.
- ajuc 6y ago
  Ever tried to parse XML or json in linux shell? It's not a matter of familiarity with linux shell. It's a matter of wasting time debugging/implementing stuff in shell that is trivial to implement in any modern scripting language.
  - freedomben 6y ago
    Haven't parsed XML in quite a number of years but I parse JSON to extract values from a curl command all the time with jq and it's really not that bad. If there's anything more than simple extraction required then I agree it's not good for the shell. For that I turn to the primary language of the project preferably (which these days is Elixir and sometimes Ruby for me). But even JSON and (X|HT)ML you'd be amazed how often a simple grep can snag what you want, and regular expressions are also mostly universal so nearly everybody can read them.
    - ajuc 6y ago
      We (2-person team doing embedded QT app on linux) did it the exact way you're arguing for - starting with shell, struggling with it for several months every time something went wrong or we had to change it, and finally giving up and rewriting it in python. Python wasn't our main language (the app was in C++ and QML which is basically java script). But we both knew Python and it's the easiest language for these sort of things that we both knew and that comes with the system. My main conclussion is - I'm starting with python the next time. Life's too short to check for the number of arguments in each function.
jiofih 6y ago
Ruby is a much better candidate for this. Nicer syntax and a vastly better standard library.
- cle 6y ago
  This kind of conversation is never productive in a team setting. The strongest criteria is "what would the majority of people on the team prefer to use?". That'll yield the highest productivity, regardless of whatever syntactic sugar it has.
- nfrmatk 6y ago
  Nicer syntax? Vastly better standard library? Those seem like subjective statements to me. Can you provide some examples to support those claims?
  - deeg 6y ago
    I dunno about a better standard library but I do think Ruby has nicer syntax. You can use back-ticks to run system commands, like: listing = `ls -oh` puts listing puts $?.to_i # Prints status of last system command.
    - nfrmatk 6y ago
      That might just come down to the usual differences between Python and Ruby. A Python programmer would likely appreciate an explicit call to `subprocess.run`. Some equivalent Python 3.7+ would be import subprocess listing = subprocess.run(("ls", "-oh"), capture_output=True, encoding="utf-8") print(listing.stdout) print(listing.returncode) As an experienced Ruby programmer I imagine the code you provided looks like a no-brainer. To a Ruby neophyte the backticks, $? sigil, and .to_i method don't strike me as intuitive (but maybe they would be do someone else). We may just have to disagree about the nicety of syntax. As cle mentioned[1], the strongest criteria is > what would the majority of people on the team prefer to use? and I'd agree that ultimately that's what makes the most sense. [1]: https://news.ycombinator.com/item?id=24558989 https://news.ycombinator.com/item?id=24558989
fra 6y ago
Every project I work on uses the `invoke`[1] Python package to create a project CLI. It's super powerful, comes with batteries included, and allows for some better code sharing and compartmentalization than a bunch of bash scripts. [1] http://www.pyinvoke.org/ http://www.pyinvoke.org/
- jgable 6y ago
  I remember the memfault blog post that introduced me to this package. Great find, and I’ve started doing the same on all my embedded dev projects. Thanks!
- theptip 6y ago
  Thanks for the recommendation, I used Fabric for local scripting ages ago and haven’t revisited in some time. Is it mostly the Make-style task decorators for managing the project task list that you consider a win vs. native subprocess.run? Are there other features that you find valuable too? (Back when the original fabric was written subprocess was much less friendly to work with, of course).
cjohnson318 6y ago
Shell is definitely not universal on Windows machines. If you work with mechanical engineers, at least half of them will be using Windows. That being said, Python is gross on Windows, too.
0df8dkdf 6y ago
At this point, one can also argue to use node. It is probably faster, and it doesn't suffer the 2.7 vs 3.x problem. just put this on the first line #!/usr/local/bin/node instead of #!/usr/local/bin/python
- slightwinder 6y ago
  Isn't nodejs breaking every some months with the newest major-update? That's even worse than Python3, which at least is now mostly established.
ivan_ah 6y ago
I'm +1 about the use of Python, but I'd say ONLY when complexity gets higher than simply calling commands (e.g. cat/sed/etc). For basic commands, I'd say Makefile is the way to go, or scripts in a package.json. I find Python becomes a must when you want to reuse commands and pass data around, or support the same kind of commands locally and on remote hosts. Fabric is the best for this: you put a `fabfile.py` (name inspired by Makefile) in the project root and add @task commands in there. Specifically the `fab-classic` fork of the Fabric project https://github.com/ploxiln/fab-classic#usage-introduction https://github.com/ploxiln/fab-classic#usage-introduction (the mainline Fabric changed it's API significantly, see `invoke` package). Fabric can run local or remote commands (via ssh). Each task is self-documenting and the best part it that it's super easy to read... almost like literate programming (as opposed to more magical tools like ansible).
- throwaway33339 6y ago
  Why choose between Makefiles and Python when you can have both? [0] [0] https://snakemake.readthedocs.io/en/stable/ https://snakemake.readthedocs.io/en/stable/
slightwinder 6y ago
subprocess is far to complicated for casual levels of scripting. We have a handful python-devs here, and everyone is avoiding it for scripts, because it's just so poor designend for this job. Shell is simple and straight forward, and can bring you quite far for most stuff. Only if you start using complex datastructures it makes more sense to use a mature language like python.
specialist 6y ago
I now agree. My last gig, I stumbled onto shebang, which allows you to invoke shell scripts in any scripting language from the command line. https://en.wikipedia.org/wiki/Shebang_(Unix) https://en.wikipedia.org/wiki/Shebang_(Unix) Major face-palm-slap. I'm certain I've read about shebang many times. But never realized that it's general purpose.
tarruda 6y ago
I've made a single file python library that implements an idiomatic shell API in python: https://github.com/tarruda/python-ush https://github.com/tarruda/python-ush One nice thing about it is the overloading of pipe operator to allow easy creation of pipelines and/or redirection (plus it is cross platform, so you don't depend on the system having a certain shell).
swirepe 6y ago
>In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow. Okay, but what about software engineers?

teknopaul 6y ago

for the record... if __name__ == '__main__': args = docopt.docopt(__doc__) means nowt to me. In fact I don't really understand your post and I've written a lot of python. No language has a simpler api to executing a cli command than bash itself. By definition.

divbzero 6y ago
The conditional expression __name___ == '__main__' simply tests whether this Python file is being executed from command line. It’s one of the ubiquitous Python idioms that doesn’t actually follow Python’s zen of “readability”.
- theptip 6y ago
  Agreed on this point, it's the `public void static main` of Python. Perhaps I shouldn't have tried to make two points in one post; I wouldn't advocate using docopt or a main function for a simple script, I was more making the case for how easy it is to add proper parameter parsing when you need it; that is easier to remember than what you'd end up writing in bash, which is something like: PARAMS="" while (( "$#" )); do case "$1" in -a|--my-boolean-flag) MY_FLAG=0 shift ;; -b|--my-flag-with-argument) if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then MY_FLAG_ARG=$2 shift 2 else echo "Error: Argument for $1 is missing" >&2 exit 1 fi ;; -*|--*=) # unsupported flags echo "Error: Unsupported flag $1" >&2 exit 1 ;; *) # preserve positional arguments PARAMS="$PARAMS $1" shift ;; esac done # set positional arguments in their proper place eval set -- "$PARAMS" I think you've got to write a lot of bash before you can remember how to write `while (( "$#" )); do` off the top of your head; the double-brackets and [ vs ( are particularly error-prone pieces of syntax.
  - rmetzler 6y ago
    But the point of the article is not to make complicated scripts readable again, but to put even simpler commands (maybe even one long mysqldump command) into very short scripts. I agree that a cli framework is often easier to use than bash, but it also is a dependency. I think everyone should use what he’s familiar with.
  - wahern 6y ago
    Idiomatic shell code would be: while getopts "ab:" OPTC; do case "$OPTC" in a) MY_FLAG=0 ;; b) MY_FLAG_ARG="$OPTARG" ;; *) # shell already printed diagnostic to stderr exit 1 ;; esac done shift $((OPTIND - 1)) Yes, I realize it's more difficult to support long options (though not that difficult), but the best tool for the job will rarely check all the boxes. Anyhow, the arguments for long options are weakest in the case of simple shell scripts. (Above code doesn't handle mixed arguments either but GNU-style permuted arguments are evil. But unlike the Bash example the above code does support bundling, which is something I and many Unix users would expect to always be supported.) Also, I realize there's a ton of Bash code that looks exactly like you wrote. But that's on Google--in promoting Bash to the exclusion of learning good shell programming style they've created an army of Bash zombies who try to write Bash code like they would Python or JavaScript code, with predictable results.
dllthomas 6y ago
> No language has a simpler api to executing a cli command than bash itself. By definition. Things that are true "by definition" are not usually useful. "No language has a simpler api [...] [b]y definition" only if by "executing a cli command" we mean literally interpreting bash (or passing the string through to a bash process). But that's never the terminal[0] goal. The useful question is whether, for the task you might want to achieve, for which you might usually reach for the shell, is there in fact a simpler way. It may very well be that the answer is "no", but support for that is not "by definition". [0]: Edited to add: ugh. Believe it or not, this was not intended.
m463 6y ago
look above it to the line that says: Usage: db_backup.py INSTANCE then just go with it.

nicbou 6y ago

At my last company, I used Python for most of our scripts. Bash scripts were too tedious to write, harden and maintain.

yjftsjthsd-h 6y ago

How does this handle pipes and such? What's the nicest python equivalent to `grep ^$(whoami) /etc/passwd | awk -F : '{print $NF}'`?

jniedrauer 6y ago
Nothing is quite as succinct as shell. The python equivalent to this would probably be at least 10 lines of code.
- tyingq 6y ago
  Everyone tends to rag on Perl, but it's excellent in this space. Edit: The equivalent of above would be: #!/usr/bin/perl my $shell=(getpwuid($>))[8]; print "$shell\n"; Though it does have simple syntax for pipes as well.
- yjftsjthsd-h 6y ago
  https://news.ycombinator.com/item?id=24558719 https://news.ycombinator.com/item?id=24558719 does it in 5, which is, I think, proof that you're both right. (Of course, perhaps shell is cheating, since I chained 3 commands into that one-liner, and 2 of those are really their own languages with further subcommands.)
johndough 6y ago
Here are three options: # Option 1 (no pipes) import os for line in open("/etc/passwd").read().splitlines(): fields = line.split(":") if fields[0] == os.getlogin(): print(fields[-1]) # Option 2 (Cheating, but not really. For most problems worth solving, there exists a library to do the work with little code.) import os print(os.environ['SHELL']) # Option 3 (pipes, not sure if the utf-8 thing can be done nicer somehow) import subprocess username = subprocess.check_output("whoami", encoding="utf-8").rstrip() p1 = subprocess.Popen(["grep", "^" + username, "/etc/passwd"], stdout=subprocess.PIPE) p2 = subprocess.Popen(["awk", "-F", ":", "{print $NF}"], stdin=p1.stdout, stdout=subprocess.PIPE) print(p2.communicate()[0].decode("utf-8"))

freedomben 6y ago

As someone who hasn't used Python in ages, and finds it quite unreadable, I would be saddened and dismayed if instead of shell (which is the lingua franca) a project used Python everywhere. The exception being if it's a Python app in the first place, then it's fine since it can be assumed you need to be familiar with Python to hack on it anyway. For example I write a lot of Ruby scripts to include with Sinatra and Rails apps. For a general purpose approach however, everybody should learn basic shell. It's not that hard and it is universal.

ajuc 6y ago
Ever tried to parse XML or json in linux shell? It's not a matter of familiarity with linux shell. It's a matter of wasting time debugging/implementing stuff in shell that is trivial to implement in any modern scripting language.
- freedomben 6y ago
  Haven't parsed XML in quite a number of years but I parse JSON to extract values from a curl command all the time with jq and it's really not that bad. If there's anything more than simple extraction required then I agree it's not good for the shell. For that I turn to the primary language of the project preferably (which these days is Elixir and sometimes Ruby for me). But even JSON and (X|HT)ML you'd be amazed how often a simple grep can snag what you want, and regular expressions are also mostly universal so nearly everybody can read them.
  - ajuc 6y ago
    We (2-person team doing embedded QT app on linux) did it the exact way you're arguing for - starting with shell, struggling with it for several months every time something went wrong or we had to change it, and finally giving up and rewriting it in python. Python wasn't our main language (the app was in C++ and QML which is basically java script). But we both knew Python and it's the easiest language for these sort of things that we both knew and that comes with the system. My main conclussion is - I'm starting with python the next time. Life's too short to check for the number of arguments in each function.

jiofih 6y ago

Ruby is a much better candidate for this. Nicer syntax and a vastly better standard library.

cle 6y ago
This kind of conversation is never productive in a team setting. The strongest criteria is "what would the majority of people on the team prefer to use?". That'll yield the highest productivity, regardless of whatever syntactic sugar it has.
nfrmatk 6y ago
Nicer syntax? Vastly better standard library? Those seem like subjective statements to me. Can you provide some examples to support those claims?
- deeg 6y ago
  I dunno about a better standard library but I do think Ruby has nicer syntax. You can use back-ticks to run system commands, like: listing = `ls -oh` puts listing puts $?.to_i # Prints status of last system command.
  - nfrmatk 6y ago
    That might just come down to the usual differences between Python and Ruby. A Python programmer would likely appreciate an explicit call to `subprocess.run`. Some equivalent Python 3.7+ would be import subprocess listing = subprocess.run(("ls", "-oh"), capture_output=True, encoding="utf-8") print(listing.stdout) print(listing.returncode) As an experienced Ruby programmer I imagine the code you provided looks like a no-brainer. To a Ruby neophyte the backticks, $? sigil, and .to_i method don't strike me as intuitive (but maybe they would be do someone else). We may just have to disagree about the nicety of syntax. As cle mentioned[1], the strongest criteria is > what would the majority of people on the team prefer to use? and I'd agree that ultimately that's what makes the most sense. [1]: https://news.ycombinator.com/item?id=24558989 https://news.ycombinator.com/item?id=24558989

fra 6y ago

Every project I work on uses the `invoke`[1] Python package to create a project CLI. It's super powerful, comes with batteries included, and allows for some better code sharing and compartmentalization than a bunch of bash scripts. [1] http://www.pyinvoke.org/ http://www.pyinvoke.org/

jgable 6y ago
I remember the memfault blog post that introduced me to this package. Great find, and I’ve started doing the same on all my embedded dev projects. Thanks!
theptip 6y ago
Thanks for the recommendation, I used Fabric for local scripting ages ago and haven’t revisited in some time. Is it mostly the Make-style task decorators for managing the project task list that you consider a win vs. native subprocess.run? Are there other features that you find valuable too? (Back when the original fabric was written subprocess was much less friendly to work with, of course).

cjohnson318 6y ago

Shell is definitely not universal on Windows machines. If you work with mechanical engineers, at least half of them will be using Windows. That being said, Python is gross on Windows, too.

0df8dkdf 6y ago

At this point, one can also argue to use node. It is probably faster, and it doesn't suffer the 2.7 vs 3.x problem. just put this on the first line #!/usr/local/bin/node instead of #!/usr/local/bin/python

slightwinder 6y ago
Isn't nodejs breaking every some months with the newest major-update? That's even worse than Python3, which at least is now mostly established.

ivan_ah 6y ago

I'm +1 about the use of Python, but I'd say ONLY when complexity gets higher than simply calling commands (e.g. cat/sed/etc). For basic commands, I'd say Makefile is the way to go, or scripts in a package.json. I find Python becomes a must when you want to reuse commands and pass data around, or support the same kind of commands locally and on remote hosts. Fabric is the best for this: you put a `fabfile.py` (name inspired by Makefile) in the project root and add @task commands in there. Specifically the `fab-classic` fork of the Fabric project https://github.com/ploxiln/fab-classic#usage-introduction https://github.com/ploxiln/fab-classic#usage-introduction (the mainline Fabric changed it's API significantly, see `invoke` package). Fabric can run local or remote commands (via ssh). Each task is self-documenting and the best part it that it's super easy to read... almost like literate programming (as opposed to more magical tools like ansible).

throwaway33339 6y ago
Why choose between Makefiles and Python when you can have both? [0] [0] https://snakemake.readthedocs.io/en/stable/ https://snakemake.readthedocs.io/en/stable/

slightwinder 6y ago

subprocess is far to complicated for casual levels of scripting. We have a handful python-devs here, and everyone is avoiding it for scripts, because it's just so poor designend for this job. Shell is simple and straight forward, and can bring you quite far for most stuff. Only if you start using complex datastructures it makes more sense to use a mature language like python.

specialist 6y ago

I now agree. My last gig, I stumbled onto shebang, which allows you to invoke shell scripts in any scripting language from the command line. https://en.wikipedia.org/wiki/Shebang_(Unix) https://en.wikipedia.org/wiki/Shebang_(Unix) Major face-palm-slap. I'm certain I've read about shebang many times. But never realized that it's general purpose.

tarruda 6y ago

I've made a single file python library that implements an idiomatic shell API in python: https://github.com/tarruda/python-ush https://github.com/tarruda/python-ush One nice thing about it is the overloading of pipe operator to allow easy creation of pipelines and/or redirection (plus it is cross platform, so you don't depend on the system having a certain shell).

swirepe 6y ago

>In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow. Okay, but what about software engineers?