3 ms·

I've always liked Python's "batteries-included" approach, and my take on libraries like this is that they introduce extra dependencies into scripts I write. And

I've always liked Python's "batteries-included" approach, and my take on libraries like this is that they introduce extra dependencies into scripts I write. And often, the juice doesn't feel like it's worth the squeeze. For me, a little bit of syntax sugar (and maybe a line or two of code savings) isn't worth taking on the dependency burden, especially in production code. Each and every dependency is one where I invite risk of requirements collision, or that the upstream goes away, or that the maintainer breaks compatibility, etc. It's more time spent installing dependencies, and more time spent processing lockfiles. Why carry that burden when subprocess.Popen() exists and works fine?

sneed_chucker 3y ago
I agree with you for the most part, I really like writing shebangable dependency-free python scripts. But I 100% understand the motivation for this project. Like you said, subprocess.Popen works well enough, but it's so much less ergonomic than shelling out in Ruby, Perl, or even Awk.
grrandalf 3y ago
Great points. Agreed. In certain situations though, the readability/maintainability gains from using this library might outweigh the added complexity of a layer of indirection. Situations like this: whether to build a utility solution completely [1] in shell, or invert things and use mix of Python and shell. E.g., wrapping 'git' to create a note-taking CLI, or shelling out to $WEIRD_INTERNAL_TOOL and scraping its logs and stdout/stderr. I think infrastructure like this repo can result in more readable and maintainable code than an all-shell OR all-Python solution in such situations. You'll be less tempted to start doing things completely in shell and then get locked into that lock minimum. However, you MUST be prepared to understand the source code of all dependencies you take [except the language's standard library, or similarly mature external libs]. e.g., I pick bottlepy.org vs Flask for personal projects. With a small/medium library like this, I might import it to get a headstart and will be ok with it ultimately ending up as an incompatible, heavily-modified fork of the upstream project. Which IMO doesn't reflect poorly on the user or the library author. [1] "completely" means some non-trivial processing that makes you use hash tables, arrays, lightweight tokenizing using read, arithmetic evaluation, sed+awk to massage command outputs so Bash's lightweight data structures can handle them, os.path manipulations.
- bsimpson 3y ago
  Also, `>> 'some-cmd --with --args'` is code that's more likely to end up in a deployment script than in a production app. Less likely to be a project that's being edited by many people at once, served to end users, etc. I respect and often agree with the parent's hesitation to introduce inessential dependencies, but the cost/benefit of making a shell script more comfortable to read/write is very different than that of adding a mystery brick to the foundation of your app.
  - nrclark 3y ago
    Deployment scripts are production code, same as whatever you're deploying. If your CI/CD docker images break, you're just as stuck as if your payload breaks. I totally see the benefit of moving complex shell logic to Python (and still calling some shell helpers) but that doesn't mean you need to have a whole extra dependency for it. You could write a two-line function that wraps subprocess.Popen.communicate (or subprocess.check_output), and use it instead for 90% of the benefit. If the argument is "I don't want to learn it", or "too much effort", then I'd ask - is saving 15 minutes one time during development really worth a permanent dependency? I remember Leftpad, and the lessons that it taught. I've also had to deal with codebases where the build kept breaking because a large team of developers all Pip-installed whatever they wanted. A package here, two packages there. Overall a maintenance disaster, and you'd be surprised at how often those dependencies outlasted any code that used them.
dimator 3y ago
Unfortunately, the subprocess package is too low level, and usually ends up making scripts excessively verbose. I wish there was a stdlib package that sat between subprocess and something else with a higher level syntax for process management. Running binary `foo` takes 3 letters in shell, and many many more in subprocess.
- krackers 3y ago
  Isn't that https://pypi.org/project/sh/ https://pypi.org/project/sh/ or https://plumbum.readthedocs.io/en/latest/#api-reference https://plumbum.readthedocs.io/en/latest/#api-reference
  - throwaway290 3y ago
    No because they are not stdlib.
slotrans 3y ago
because subprocess.Popen() is a giant pain in the ass that's easy to screw up, and I have to read the docs every single time and I'm still not sure it's right

sneed_chucker 3y ago

I agree with you for the most part, I really like writing shebangable dependency-free python scripts. But I 100% understand the motivation for this project. Like you said, subprocess.Popen works well enough, but it's so much less ergonomic than shelling out in Ruby, Perl, or even Awk.

grrandalf 3y ago

Great points. Agreed. In certain situations though, the readability/maintainability gains from using this library might outweigh the added complexity of a layer of indirection. Situations like this: whether to build a utility solution completely [1] in shell, or invert things and use mix of Python and shell. E.g., wrapping 'git' to create a note-taking CLI, or shelling out to $WEIRD_INTERNAL_TOOL and scraping its logs and stdout/stderr. I think infrastructure like this repo can result in more readable and maintainable code than an all-shell OR all-Python solution in such situations. You'll be less tempted to start doing things completely in shell and then get locked into that lock minimum. However, you MUST be prepared to understand the source code of all dependencies you take [except the language's standard library, or similarly mature external libs]. e.g., I pick bottlepy.org vs Flask for personal projects. With a small/medium library like this, I might import it to get a headstart and will be ok with it ultimately ending up as an incompatible, heavily-modified fork of the upstream project. Which IMO doesn't reflect poorly on the user or the library author. [1] "completely" means some non-trivial processing that makes you use hash tables, arrays, lightweight tokenizing using read, arithmetic evaluation, sed+awk to massage command outputs so Bash's lightweight data structures can handle them, os.path manipulations.

bsimpson 3y ago
Also, `>> 'some-cmd --with --args'` is code that's more likely to end up in a deployment script than in a production app. Less likely to be a project that's being edited by many people at once, served to end users, etc. I respect and often agree with the parent's hesitation to introduce inessential dependencies, but the cost/benefit of making a shell script more comfortable to read/write is very different than that of adding a mystery brick to the foundation of your app.
- nrclark 3y ago
  Deployment scripts are production code, same as whatever you're deploying. If your CI/CD docker images break, you're just as stuck as if your payload breaks. I totally see the benefit of moving complex shell logic to Python (and still calling some shell helpers) but that doesn't mean you need to have a whole extra dependency for it. You could write a two-line function that wraps subprocess.Popen.communicate (or subprocess.check_output), and use it instead for 90% of the benefit. If the argument is "I don't want to learn it", or "too much effort", then I'd ask - is saving 15 minutes one time during development really worth a permanent dependency? I remember Leftpad, and the lessons that it taught. I've also had to deal with codebases where the build kept breaking because a large team of developers all Pip-installed whatever they wanted. A package here, two packages there. Overall a maintenance disaster, and you'd be surprised at how often those dependencies outlasted any code that used them.

dimator 3y ago

Unfortunately, the subprocess package is too low level, and usually ends up making scripts excessively verbose. I wish there was a stdlib package that sat between subprocess and something else with a higher level syntax for process management. Running binary `foo` takes 3 letters in shell, and many many more in subprocess.

krackers 3y ago
Isn't that https://pypi.org/project/sh/ https://pypi.org/project/sh/ or https://plumbum.readthedocs.io/en/latest/#api-reference https://plumbum.readthedocs.io/en/latest/#api-reference
- throwaway290 3y ago
  No because they are not stdlib.

slotrans 3y ago

because subprocess.Popen() is a giant pain in the ass that's easy to screw up, and I have to read the docs every single time and I'm still not sure it's right