5 ms·

Quite an impressive project all things considered; definitely something to keep in mind for the next project that inevitably starts out as a simple Bash script.

by usrme 6y ago

Quite an impressive project all things considered; definitely something to keep in mind for the next project that inevitably starts out as a simple Bash script. Out of curiosity: if you (the reader) get to a point where you need more complicated and esoteric functionality that Bash can provide, do you still keep hammering at it or turn to a more capable (and readable) language? Just wondering if people are usually working in very constrained environments where other languages aren't able to be readily used.

lysium 6y ago
I've tried that path with little success. Usually, arg parsing makes me try some other language (perl, python) but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output etc. In the end, it was easier to hammer onto bash. Also, there are great linters for bash scripts, so things work out.
- IggleSniggle 6y ago
  tcl is designed for this situation
  - lysium 6y ago
    Thx, I’ll give it a shot!
- lou1306 6y ago
  > but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output Note that Python's subprocess module [0] has received many updates since v3.5, and now I find it relatively painless. Most of the times, subprocess.check_output() or subprocess.run() will do the trick. [0]: https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html
  - lysium 6y ago
    Thx, I’ll check it out again!
- kortex 6y ago
  I've been in a similar boat, yet despite as my experience with it grows, bash just wears on me. Everything feels like a hack. Sure, piping is easy, but if statements are atrocious. Starting to get into Xonsh. Still a learning curve, cause it's neither bash nor python, but it's really refreshing.
xurukefi 6y ago
As soon as as the cli becomes non-trivial I switch to python and use docopt for that (really cool module btw.). Pipelining processes is not as idiomatic in Python as it is in Bash though, but everything is better than getopts. I'd rather decapitate myself with a teaspoon than using getopts.
- OJFord 6y ago
  Docopt exists (now, think it was originally python) for many languages including bash [0] (or perhaps it's POSIX, haven't checked). I like it too, not least because I can use ~the same thing in multiple languages and not have to remind myself how the arg parser for language X works each time. [0] - https://github.com/docopt/docopts https://github.com/docopt/docopts
- vlovich123 6y ago
  Yeah, I’m kind of shocked to see so many masochists in this thread. Built-in subprocess is nice and all but I’ve had a lot of joy using https://amoffat.github.io/sh/ https://amoffat.github.io/sh/ to do the work (and the maintainer is super responsive). I sometimes don’t have the flexibility to have external dependencies but I’d use it everywhere if Python just folded it into the default library.
raziel2p 6y ago
It's a trade-off. Especially if the script contains a lot of pipes and redirection, it's difficult to reproduce with the same elegance in something like Python. My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes, which gives you the best of both worlds. The best example off the top of my head is `mysqldump | sed ... | gzip` with decent error handling (i.e. set -o pipefail in bash). My environment isn't all that constrained, but I do have to deal with old Python versions (though at least I don't have to support 2.6 any more...), and a surprising amount of the features that would make replacing bash easier (subprocess.call especially) are only available fairly recently.
- vlovich123 6y ago
  I’m curious about your experience that pipes and redirects are harder with python. Do you mean slightly more verbose or just complicated and difficult? I’m curious if you’ve tried the sh Python module [1]. In some ways it’s much prettier to read although the initial writing of it can be awkward to get used to at first. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
  - raziel2p 6y ago
    I mean significantly more verbose and more complicated. Fee free to try it yourself - replicate `bash -eo pipefail -c 'mysqldump example_db | gzip > dump.sql.gz'` in Python with streaming (databases can be several GB, so you can't read it all in memory). I have not tried any of these libraries, though they look nice. The only times I've had to write scripts that make heavy use of subprocesses and pipes, I want them to work with only the standard library so I can just rsync them and they just work™.
    - vlovich123 6y ago
      You have to dig through the documentation a bit on the website but all the info is there[1] from sh import mysqldump from sh import gzip gzip(mysqldump("example_db", _piped=True), _out="dump.sql.gz") or if you don't want the magic import thing: import sh sh.gzip(sh.mysqldump("example_db", _piped=True), _out="dump.sql.gz") It's definitely foreign to shell scripting languages since the piping syntax is a bit different & you have to remember to do `_piped=True` since it's not parallel by default. But the default behavior is pipefail & exit on the command failing IIRC so it's more of a choose your poison thing (do you want a subtle perf issue or a subtle bug in your script not handling error states correctly). And I find it easier to read. + if you want to customize anything about gzip or not rely on needing the binary in the path, then you can just switch it to Python-native gzip pretty easily. That's my favorite feature. Conciseness if I'm just translating a script with progressive complexity options to migrate things that need more complexity or different requirements within the same script without having to rewrite it from scratch. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
techdragon 6y ago
Speaking as someone who frequently injects the most ridiculous of things into a makefile in order to bootstrap build tools and such things... and as someone who had learned my way through enough of the Chrome OS build tool chain to customise a fork of it for building my own variant of CoreOS back in the early days before Kubernetes won the container platform wars... It depends. Sometimes you use these tools because they are already responsible for so much stuff it makes sense to just extend it no matter how crazy it is (Chrome Os build tool was built as hundreds of lines of shell script which built on top of portage which is even more hundred of lines of shell script) ... sometimes it’s because that’s the tool you know will be available (the dozens of bootstrap scripts I’ve beaten crushed and smashed into makefiles complete with OS detection and other stuff) ... and sometimes you don’t have any read you just do it because it’s what your comfortable thinking in. I’ve definitely written bash scripts that would have been better in Python but at the time my mind was building the script up as a sequence of Unix tool operations and pipelines not as a Python program.

by usrme 6y ago

lysium 6y ago

I've tried that path with little success. Usually, arg parsing makes me try some other language (perl, python) but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output etc. In the end, it was easier to hammer onto bash. Also, there are great linters for bash scripts, so things work out.

IggleSniggle 6y ago
tcl is designed for this situation
- lysium 6y ago
  Thx, I’ll give it a shot!
lou1306 6y ago
> but other things got complicated there, mostly running other processes, capturing their exit status, pipe-ing their output Note that Python's subprocess module [0] has received many updates since v3.5, and now I find it relatively painless. Most of the times, subprocess.check_output() or subprocess.run() will do the trick. [0]: https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html
- lysium 6y ago
  Thx, I’ll check it out again!
kortex 6y ago
I've been in a similar boat, yet despite as my experience with it grows, bash just wears on me. Everything feels like a hack. Sure, piping is easy, but if statements are atrocious. Starting to get into Xonsh. Still a learning curve, cause it's neither bash nor python, but it's really refreshing.

xurukefi 6y ago

As soon as as the cli becomes non-trivial I switch to python and use docopt for that (really cool module btw.). Pipelining processes is not as idiomatic in Python as it is in Bash though, but everything is better than getopts. I'd rather decapitate myself with a teaspoon than using getopts.

OJFord 6y ago
Docopt exists (now, think it was originally python) for many languages including bash [0] (or perhaps it's POSIX, haven't checked). I like it too, not least because I can use ~the same thing in multiple languages and not have to remind myself how the arg parser for language X works each time. [0] - https://github.com/docopt/docopts https://github.com/docopt/docopts
vlovich123 6y ago
Yeah, I’m kind of shocked to see so many masochists in this thread. Built-in subprocess is nice and all but I’ve had a lot of joy using https://amoffat.github.io/sh/ https://amoffat.github.io/sh/ to do the work (and the maintainer is super responsive). I sometimes don’t have the flexibility to have external dependencies but I’d use it everywhere if Python just folded it into the default library.

raziel2p 6y ago

It's a trade-off. Especially if the script contains a lot of pipes and redirection, it's difficult to reproduce with the same elegance in something like Python. My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes, which gives you the best of both worlds. The best example off the top of my head is `mysqldump | sed ... | gzip` with decent error handling (i.e. set -o pipefail in bash). My environment isn't all that constrained, but I do have to deal with old Python versions (though at least I don't have to support 2.6 any more...), and a surprising amount of the features that would make replacing bash easier (subprocess.call especially) are only available fairly recently.

vlovich123 6y ago
I’m curious about your experience that pipes and redirects are harder with python. Do you mean slightly more verbose or just complicated and difficult? I’m curious if you’ve tried the sh Python module [1]. In some ways it’s much prettier to read although the initial writing of it can be awkward to get used to at first. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
- raziel2p 6y ago
  I mean significantly more verbose and more complicated. Fee free to try it yourself - replicate `bash -eo pipefail -c 'mysqldump example_db | gzip > dump.sql.gz'` in Python with streaming (databases can be several GB, so you can't read it all in memory). I have not tried any of these libraries, though they look nice. The only times I've had to write scripts that make heavy use of subprocesses and pipes, I want them to work with only the standard library so I can just rsync them and they just work™.
  - vlovich123 6y ago
    You have to dig through the documentation a bit on the website but all the info is there[1] from sh import mysqldump from sh import gzip gzip(mysqldump("example_db", _piped=True), _out="dump.sql.gz") or if you don't want the magic import thing: import sh sh.gzip(sh.mysqldump("example_db", _piped=True), _out="dump.sql.gz") It's definitely foreign to shell scripting languages since the piping syntax is a bit different & you have to remember to do `_piped=True` since it's not parallel by default. But the default behavior is pipefail & exit on the command failing IIRC so it's more of a choose your poison thing (do you want a subtle perf issue or a subtle bug in your script not handling error states correctly). And I find it easier to read. + if you want to customize anything about gzip or not rely on needing the binary in the path, then you can just switch it to Python-native gzip pretty easily. That's my favorite feature. Conciseness if I'm just translating a script with progressive complexity options to migrate things that need more complexity or different requirements within the same script without having to rewrite it from scratch. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/

techdragon 6y ago

Speaking as someone who frequently injects the most ridiculous of things into a makefile in order to bootstrap build tools and such things... and as someone who had learned my way through enough of the Chrome OS build tool chain to customise a fork of it for building my own variant of CoreOS back in the early days before Kubernetes won the container platform wars... It depends. Sometimes you use these tools because they are already responsible for so much stuff it makes sense to just extend it no matter how crazy it is (Chrome Os build tool was built as hundreds of lines of shell script which built on top of portage which is even more hundred of lines of shell script) ... sometimes it’s because that’s the tool you know will be available (the dozens of bootstrap scripts I’ve beaten crushed and smashed into makefiles complete with OS detection and other stuff) ... and sometimes you don’t have any read you just do it because it’s what your comfortable thinking in. I’ve definitely written bash scripts that would have been better in Python but at the time my mind was building the script up as a sequence of Unix tool operations and pipelines not as a Python program.