4 ms·

It's a trade-off. Especially if the script contains a lot of pipes and redirection, it's difficult to reproduce with the same elegance in something like Python.

It's a trade-off. Especially if the script contains a lot of pipes and redirection, it's difficult to reproduce with the same elegance in something like Python. My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes, which gives you the best of both worlds. The best example off the top of my head is `mysqldump | sed ... | gzip` with decent error handling (i.e. set -o pipefail in bash). My environment isn't all that constrained, but I do have to deal with old Python versions (though at least I don't have to support 2.6 any more...), and a surprising amount of the features that would make replacing bash easier (subprocess.call especially) are only available fairly recently.

vlovich123 6y ago
I’m curious about your experience that pipes and redirects are harder with python. Do you mean slightly more verbose or just complicated and difficult? I’m curious if you’ve tried the sh Python module [1]. In some ways it’s much prettier to read although the initial writing of it can be awkward to get used to at first. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
- raziel2p 6y ago
  I mean significantly more verbose and more complicated. Fee free to try it yourself - replicate `bash -eo pipefail -c 'mysqldump example_db | gzip > dump.sql.gz'` in Python with streaming (databases can be several GB, so you can't read it all in memory). I have not tried any of these libraries, though they look nice. The only times I've had to write scripts that make heavy use of subprocesses and pipes, I want them to work with only the standard library so I can just rsync them and they just work™.
  - vlovich123 6y ago
    You have to dig through the documentation a bit on the website but all the info is there[1] from sh import mysqldump from sh import gzip gzip(mysqldump("example_db", _piped=True), _out="dump.sql.gz") or if you don't want the magic import thing: import sh sh.gzip(sh.mysqldump("example_db", _piped=True), _out="dump.sql.gz") It's definitely foreign to shell scripting languages since the piping syntax is a bit different & you have to remember to do `_piped=True` since it's not parallel by default. But the default behavior is pipefail & exit on the command failing IIRC so it's more of a choose your poison thing (do you want a subtle perf issue or a subtle bug in your script not handling error states correctly). And I find it easier to read. + if you want to customize anything about gzip or not rely on needing the binary in the path, then you can just switch it to Python-native gzip pretty easily. That's my favorite feature. Conciseness if I'm just translating a script with progressive complexity options to migrate things that need more complexity or different requirements within the same script without having to rewrite it from scratch. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
mturmon 6y ago
> My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes ... That's a clever way to go. Sometimes the other way around works too: taking the most fiddly manipulations (say, date or time intervals) and re-homing them to a small Python script, but leaving the bash intact.
kortex 6y ago
There is also plumbum for more ergonomic pipelines in python: https://pypi.org/project/plumbum/ https://pypi.org/project/plumbum/

vlovich123 6y ago

I’m curious about your experience that pipes and redirects are harder with python. Do you mean slightly more verbose or just complicated and difficult? I’m curious if you’ve tried the sh Python module [1]. In some ways it’s much prettier to read although the initial writing of it can be awkward to get used to at first. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/

raziel2p 6y ago
I mean significantly more verbose and more complicated. Fee free to try it yourself - replicate `bash -eo pipefail -c 'mysqldump example_db | gzip > dump.sql.gz'` in Python with streaming (databases can be several GB, so you can't read it all in memory). I have not tried any of these libraries, though they look nice. The only times I've had to write scripts that make heavy use of subprocesses and pipes, I want them to work with only the standard library so I can just rsync them and they just work™.
- vlovich123 6y ago
  You have to dig through the documentation a bit on the website but all the info is there[1] from sh import mysqldump from sh import gzip gzip(mysqldump("example_db", _piped=True), _out="dump.sql.gz") or if you don't want the magic import thing: import sh sh.gzip(sh.mysqldump("example_db", _piped=True), _out="dump.sql.gz") It's definitely foreign to shell scripting languages since the piping syntax is a bit different & you have to remember to do `_piped=True` since it's not parallel by default. But the default behavior is pipefail & exit on the command failing IIRC so it's more of a choose your poison thing (do you want a subtle perf issue or a subtle bug in your script not handling error states correctly). And I find it easier to read. + if you want to customize anything about gzip or not rely on needing the binary in the path, then you can just switch it to Python-native gzip pretty easily. That's my favorite feature. Conciseness if I'm just translating a script with progressive complexity options to migrate things that need more complexity or different requirements within the same script without having to rewrite it from scratch. [1] https://amoffat.github.io/sh/ https://amoffat.github.io/sh/

mturmon 6y ago

> My biggest successes have been extracting these parts into very small stand-alone shell scripts which the Python application then invokes ... That's a clever way to go. Sometimes the other way around works too: taking the most fiddly manipulations (say, date or time intervals) and re-homing them to a small Python script, but leaving the bash intact.

kortex 6y ago

There is also plumbum for more ergonomic pipelines in python: https://pypi.org/project/plumbum/ https://pypi.org/project/plumbum/