3 ms·
Just to be clear, the issue is not in Python: the deadlock will happen if you won't consume the pipe whatever language you use to spawn a subprocess -- the buff
by d0mine 11y ago
Just to be clear, the issue is not in Python: the deadlock will happen if you won't consume the pipe whatever language you use to spawn a subprocess -- the buffer is not in your application -- it is the OS pipe buffer.
I imagine your broken code did something like:
#XXX WRONG, DON'T DO IT
p = Popen(cmd, stdout=PIPE)
p.wait() #XXX deadlock is possible if cmd generates enough output
output = p.stdout.read()
To avoid the deadlock, it is enough to swap the last two statements:
p = Popen(cmd, stdout=PIPE)
output = p.stdout.read()
p.wait()
that you would normally write as:
output = check_output(cmd) # 2.7+
If you need to consume more than one pipe separately then you should consume them concurrently and therefore you need threads or async.io.
`Popen.communicate()` can read from both pipes concurrently (as you've mentioned) and it is always available. It was available even in Python 2.4 that introduced the subprocess module.
- pjungwir 11y agoYep, I've hit this in Perl and Ruby and even once wrote a class to do something like Python's `communicate`. It's especially trappy if you want to run a command and both send stuff to its stdin and read stuff from its stdout. Bash makes it so easy, but it's a fraught with danger anywhere else!