10 ms·

Working with Python subprocess

41 points by llambda 15y ago

rollypolly 15y ago
In regards to "Accessing Return Values, poll() and wait()": Another easy way to get a returncode without polling is to call communicate(). returncode will be set when it returns from communicate().
Pewpewarrows 15y ago
As someone who to this day still cringes every time they need to work with the subprocess module, do yourself a favor and use Kenneth Reitz's excellent wrapper library called Envoy: https://github.com/kennethreitz/envoy https://github.com/kennethreitz/envoy For reference, he's the same guy that did the Requests module, which is pretty much the defacto standard for avoiding urllib hell.
- shuzchen 15y ago
  As someone who has used (and contributed code) to envoy, I must give warning that envoy is still very much a work in progress. Don't go in there expecting the same quality of the Requests module.
- falcolas 15y ago
  Envoy doesn't work if your data is larger than what you can safely store in memory (a few times over). I would guess that subprocess is so ugly just because it needs to be so flexible. Personally, I don't find it so offensive, mostly because I have to take advantage of its "warts" on almost a daily basis.
falcolas 15y ago
A few notes from a recent attempt to use subprocess in the middle of a few generators: p.communicate() only works if the output of the program can fit in memory. If it can't, python will gladly crash your box for you. The buffer size of the os pipes on Linux are 64k. If you don't do your reads and writes in this size, it's possible to deadlock yourself. Reads on the pipes has to be done in a non-blocking fashion, using os.read(), if you're dealing with streams of data that are larger than the memory you have available to yourself. You can chain multiple subprocesses together directly, if you feed the stdout pipe to stdin on the next process. p1 = subprocess.Popen(['cat', 'foo'], stdout=subprocess.PIPE) p2 = subprocess.Popen(['grep', 'cat'], stdin=p1.stdout, stdout=subprocess.PIPE) p2.communicate() select.poll() works on the os.pipes which are contained in the Popen objects.
ajross 15y ago
I'm unmasking myself as a curmudgeon, but my blood boils when I see python people trying to do this stuff. The subprocess API is just awful. Compare the clean shell syntax at the top of this document to the wordy, awful dreck at the bottom required to do the same stuff in python. Likewise compare it to the perl syntax that does the same thing (and happens to look at lot like the shell does, with good reason). Python is a great language, but sometimes the aesthetic taste of its adherents just boggles me. Edit: I see Pewpewarrows above pointing out that there's a third party module which fixes some of the nonsense in subprocess. And indeed, it looks fairly clean at first glance. So perhaps I should retract my blanket condemnation of its adherents' taste.
- jonathansizz 15y ago
  I agree. Scripting languages in general and Python in particular seem to be overused (and promoted) in domains that they're not well suited for. But we've been seeing a lot of random several-year-old python stuff on Hacker News for quite a while now. Can't someone just post a link to a comprehensive list of all python tutorials, rather than them all being submitted individually like this?
- krupan 15y ago
  I felt the same (well, not quite blood boiling, but annoyed) with regular expressions in python. Why so awkward? But now, I don't know, one thing the somewhat cumbersome subprocess and regular expression support helped me to learn was that I didn't want or need to spawn a subprocess or use a regex for every little thing. There are good python libraries for manipulating files and directories, globbing, and so forth. For strings, you can get a very long way using split, join, slicing, and the "in" operator. I like my programs more this way. They look like, you know, programs, instead of hacked together shell scripts.
  - kstrauser 15y ago
    Out of curiosity, what do you dislike in Python regexes? I switched from Perl to Python years ago and didn't notice much of a difference, other than that you have to write `value = re.sub(pattern, repl, value)` instead of `$value =~ s/pattern/repl/;`.
gaius 15y ago
Article was written 3 years ago; perhaps things are better now?

41 points by llambda 15y ago

In regards to "Accessing Return Values, poll() and wait()": Another easy way to get a returncode without polling is to call communicate(). returncode will be set when it returns from communicate().

Pewpewarrows 15y ago

As someone who to this day still cringes every time they need to work with the subprocess module, do yourself a favor and use Kenneth Reitz's excellent wrapper library called Envoy: https://github.com/kennethreitz/envoy https://github.com/kennethreitz/envoy For reference, he's the same guy that did the Requests module, which is pretty much the defacto standard for avoiding urllib hell.

shuzchen 15y ago
As someone who has used (and contributed code) to envoy, I must give warning that envoy is still very much a work in progress. Don't go in there expecting the same quality of the Requests module.
falcolas 15y ago
Envoy doesn't work if your data is larger than what you can safely store in memory (a few times over). I would guess that subprocess is so ugly just because it needs to be so flexible. Personally, I don't find it so offensive, mostly because I have to take advantage of its "warts" on almost a daily basis.

falcolas 15y ago

A few notes from a recent attempt to use subprocess in the middle of a few generators: p.communicate() only works if the output of the program can fit in memory. If it can't, python will gladly crash your box for you. The buffer size of the os pipes on Linux are 64k. If you don't do your reads and writes in this size, it's possible to deadlock yourself. Reads on the pipes has to be done in a non-blocking fashion, using os.read(), if you're dealing with streams of data that are larger than the memory you have available to yourself. You can chain multiple subprocesses together directly, if you feed the stdout pipe to stdin on the next process. p1 = subprocess.Popen(['cat', 'foo'], stdout=subprocess.PIPE) p2 = subprocess.Popen(['grep', 'cat'], stdin=p1.stdout, stdout=subprocess.PIPE) p2.communicate() select.poll() works on the os.pipes which are contained in the Popen objects.

ajross 15y ago

I'm unmasking myself as a curmudgeon, but my blood boils when I see python people trying to do this stuff. The subprocess API is just awful. Compare the clean shell syntax at the top of this document to the wordy, awful dreck at the bottom required to do the same stuff in python. Likewise compare it to the perl syntax that does the same thing (and happens to look at lot like the shell does, with good reason). Python is a great language, but sometimes the aesthetic taste of its adherents just boggles me. Edit: I see Pewpewarrows above pointing out that there's a third party module which fixes some of the nonsense in subprocess. And indeed, it looks fairly clean at first glance. So perhaps I should retract my blanket condemnation of its adherents' taste.

jonathansizz 15y ago
I agree. Scripting languages in general and Python in particular seem to be overused (and promoted) in domains that they're not well suited for. But we've been seeing a lot of random several-year-old python stuff on Hacker News for quite a while now. Can't someone just post a link to a comprehensive list of all python tutorials, rather than them all being submitted individually like this?
krupan 15y ago
I felt the same (well, not quite blood boiling, but annoyed) with regular expressions in python. Why so awkward? But now, I don't know, one thing the somewhat cumbersome subprocess and regular expression support helped me to learn was that I didn't want or need to spawn a subprocess or use a regex for every little thing. There are good python libraries for manipulating files and directories, globbing, and so forth. For strings, you can get a very long way using split, join, slicing, and the "in" operator. I like my programs more this way. They look like, you know, programs, instead of hacked together shell scripts.
- kstrauser 15y ago
  Out of curiosity, what do you dislike in Python regexes? I switched from Perl to Python years ago and didn't notice much of a difference, other than that you have to write `value = re.sub(pattern, repl, value)` instead of `$value =~ s/pattern/repl/;`.

gaius 15y ago

Article was written 3 years ago; perhaps things are better now?