4 ms·

Please, do not use subprocess.Popen. Use something like plumbum, way safer and more robust.

deleted 2y ago
[deleted]
theamk 2y ago
If you can make a python program which only uses stdlib, it becomes wonderfully portable and easy to work with. Also, significantly more people use stdlib, there is more knowledge on the internet, and xz-style supply chain attacks are significantly less likely. This is why my advice to everyone is to use python's stdlib as much as possible, and avoid using Python's external libraries unless they significantly simplify code. Plumbum seems nice (and also is packaged in debian/ubuntu, which is a plus), but it does not seem to be significantly safer than correctly written subprocess code, and it won't even save that much lines in this particular example.
- zekrioca 2y ago
  I agree and disagree. Python’s Subprocess has been the reason for many unfortunate, time-consuming bugs among users who think they are properly executing external commands, but that in end cannot realize their logic are full of errors, with no indication that something went bad. I agree that a standard interface is always better, however not at the cost of productivity. A better than the current Subprocess interface is needed, and I think plumbum is the direction to go.
  - theamk 2y ago
    I am curious which errors do you find most problematic? We have internal codebase with hundreds of developers, and we haven't observed many subprocess related bugs. And the ability to print command being executed (via shlex.join) so it can be copied to shell and debugged there is very nice. That said, there is a bunch of rules in our internal style guide about this process, such as: avoid shell=True unless you need shell functionality and know how shell quoting works; use python instead of tools like grep/head/aws/etc.. when performance permits; check returncode after Popen calls; correctly quote ssh args (they are tricky!)

deleted 2y ago

[deleted]

If you can make a python program which only uses stdlib, it becomes wonderfully portable and easy to work with. Also, significantly more people use stdlib, there is more knowledge on the internet, and xz-style supply chain attacks are significantly less likely. This is why my advice to everyone is to use python's stdlib as much as possible, and avoid using Python's external libraries unless they significantly simplify code. Plumbum seems nice (and also is packaged in debian/ubuntu, which is a plus), but it does not seem to be significantly safer than correctly written subprocess code, and it won't even save that much lines in this particular example.

zekrioca 2y ago
I agree and disagree. Python’s Subprocess has been the reason for many unfortunate, time-consuming bugs among users who think they are properly executing external commands, but that in end cannot realize their logic are full of errors, with no indication that something went bad. I agree that a standard interface is always better, however not at the cost of productivity. A better than the current Subprocess interface is needed, and I think plumbum is the direction to go.
- theamk 2y ago
  I am curious which errors do you find most problematic? We have internal codebase with hundreds of developers, and we haven't observed many subprocess related bugs. And the ability to print command being executed (via shlex.join) so it can be copied to shell and debugged there is very nice. That said, there is a bunch of rules in our internal style guide about this process, such as: avoid shell=True unless you need shell functionality and know how shell quoting works; use python instead of tools like grep/head/aws/etc.. when performance permits; check returncode after Popen calls; correctly quote ssh args (they are tricky!)