4 ms·

Working with Python subprocess - Shells, Processes, Streams, Pipes, Redirects

60 points by timf 15y ago

malkia 15y ago
I had to use os.popen (Maya, MotionBuilder, and outside) (obsolete by now, but still available), and it seemed much more easier than subprocess.Popen because I simply wanted a file filter. Basically a coworker had ways to read/write custom ASCII 3D animation and model format, and another one introduced a binary form for it. All I did was to replace his "open" function with "open_model" and "open_anim" (no monkey patching), with something like this: def open_anim(name, mode, *args, **kwargs): if name.upper().endswith( ".BIN_ANIM" ): if 'r' in mode: return os.popen( "anim_bin2text <" + name, mode, *args, **kwargs ) if 'w' in mode: return os.popen( "anim_text2bin >" + name, mode, *args, **kwargs ) return open(name, mode, *args, **kwargs)
- VMG 15y ago
  popen might have been simple but don't forget popen2, popen3 and popen4, spawn, spawnvp, spawnlp. You've got to get rid of that cruft or you become php.
nzoschke 15y ago
Great article. Chaining pipes together in Python with subprocess can get a little wordy, but the power is there to do anything you need to do. It's especially excellent when compared to Ruby. Ruby has a bunch of popen calls, but each one has strange quirks and inflexibility to make it not good for building pipelines. http://whynotwiki.com/Ruby_/_Process_management#How_do_I_execute_an_external_program.3F http://whynotwiki.com/Ruby_/_Process_management#How_do_I_exe... I'm finding myself using a ton more shell (bash) for pipelines, as this is what it was built to do, but when I need to add more logic its Python and subprocess.
mturmon 15y ago
The python parts seemed OK. But the data flow diagram in the OP at: http://jimmyg.org/blog/2009/working-with-python-subprocess.html#what-happens-when-you-execute-a-command http://jimmyg.org/blog/2009/working-with-python-subprocess.h... (I.e., the first blue box in the article, "what happens when you execute a command") is wrong. It implies that the command (ls -l) is read by the terminal emulator, which gives it to bash, which gives it to the kernel. This is wrong in several ways. The terminal emulator starts bash (or whatever shell) and it is bash that directly reads the string "ls -l", figures out what it means, finds the ls program in PATH, and runs the ls binary. It is ls that calls kernel functions when it's doing its job. And of course, the kernel is not really of importance here. The selected command does whatever it does, which might not even access the kernel ("Hello, world"). And bash can and sometimes does directly call kernel functions without an intermediate binary (e.g., unlink(2) to overwrite a file for output redirection).
- thwarted 15y ago
  ...the data flow diagram in the OP..."what happens when you execute a command"...is wrong. It implies that the command (ls -l) is read by the terminal emulator, which gives it to bash, which gives it to the kernel. It's accurate, the author just left off where the command is coming from (implying it's the user sitting at a keyboard). You type into the terminal-emulator, which passes that input to the program it's running, the shell, which figures out what it means. A terminal (or something emulating a terminal) is always between the keyboard and the input of the program running. Even when you login on the console, the terminal driver is accepting your input, transforming it in some fashion, and giving it to the nxt level. A terminal doesn't exist between this echo command and the sh program: echo ls -l | sh And is, in fact, why things like expect(1) exist and that ssh can be configured to allocate a pseudo-terminal or not depending on how you want/need to interact with the remote command. Incidentally, there seems to be a bug in yum whereby it detects if its output is a terminal or not tries to format it, but it gets the check reversed and pretty-print word-wraps output that goes to a non-terminal. Which is why yum search something and yum search something | sort produce different output and the latter has blank lines scattered throughout. (yum 3.2.28 on Fedora 14). And of course, the kernel is not really of importance here. The selected command does whatever it does, which might not even access the kernel ("Hello, world"). Every Hello, World program I've seen displays text to the user (or sends it somewhere), which means a system call, which is calling "kernel functions". The kernel is the mediator between programs/processes, there are very few cases where it isn't directly involved. And bash can and sometimes does directly call kernel functions without an intermediate binary (e.g., unlink(2) to overwrite a file for output redirection). Not really a good example, but the idea is sound. Bash has builtin functionality that emulate the functionality of system calls or other binaries. For this case, bash opens files for redirection when it's going to overwrite with the O_TRUNC flag set. This can be changed with various shell options (clobber) to have it not implicitly overwrite. You can see this with strace. And technically, "overwriting" a file means truncation or seek-and-write, not unlink and recreate. The difference has implications for hardlinked files.
  - mturmon 15y ago
    OK, I see your point regarding the terminal emulator. There are some keys that are intercepted by the terminal emulator and processed by it alone (say, whatever does "print screen" or "close window"). The shell never sees these key presses. The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. And the presence of the kernel in the diagram is also confused. (You are right, system calls are all over, even printf() will bottom out in the kernel; I was trying to say that there is no requirement for the user program to call in to the kernel.) Here's a sentence in the OP that has the same confusion: "When you execute a program from Python you can choose to execute it directly with the kernel or via a shell." The kernel is not "executing the program directly". This is nonsense. What they want to say is that you can use a system call (execve) to directly load and run the program, or you can use an interpreter like sh. This same point of confusion is exhibited in their diagram.
squeed 15y ago
Python's subprocess module is great. It is definitely the cleanest wrapping of a system program of any of the scripting languages I've ever used. When you just want output, you call .communicate() and be done with it. When you want to interact, the file descriptors are there for you to play with. It's scripting heaven. There is one feature that he could have touched on more: shell=False is the only easy, safe way to put user-supplied input in a command line. This stuff is just too damned easy to get wrong. I've seen heroic code that tried to work around shell quoting and escaping vulnerabilities. One colleague was trying to write a wrapper around the ldapsearch binary, and decided that filtering all "|" would do it. He'd completely forgotten that ` also triggers arbitrary command execution. Don't be a hero. Keep bash away from your input.

60 points by timf 15y ago

I had to use os.popen (Maya, MotionBuilder, and outside) (obsolete by now, but still available), and it seemed much more easier than subprocess.Popen because I simply wanted a file filter. Basically a coworker had ways to read/write custom ASCII 3D animation and model format, and another one introduced a binary form for it. All I did was to replace his "open" function with "open_model" and "open_anim" (no monkey patching), with something like this: def open_anim(name, mode, *args, **kwargs): if name.upper().endswith( ".BIN_ANIM" ): if 'r' in mode: return os.popen( "anim_bin2text <" + name, mode, *args, **kwargs ) if 'w' in mode: return os.popen( "anim_text2bin >" + name, mode, *args, **kwargs ) return open(name, mode, *args, **kwargs)

VMG 15y ago
popen might have been simple but don't forget popen2, popen3 and popen4, spawn, spawnvp, spawnlp. You've got to get rid of that cruft or you become php.

nzoschke 15y ago

Great article. Chaining pipes together in Python with subprocess can get a little wordy, but the power is there to do anything you need to do. It's especially excellent when compared to Ruby. Ruby has a bunch of popen calls, but each one has strange quirks and inflexibility to make it not good for building pipelines. http://whynotwiki.com/Ruby_/_Process_management#How_do_I_execute_an_external_program.3F http://whynotwiki.com/Ruby_/_Process_management#How_do_I_exe... I'm finding myself using a ton more shell (bash) for pipelines, as this is what it was built to do, but when I need to add more logic its Python and subprocess.

mturmon 15y ago

The python parts seemed OK. But the data flow diagram in the OP at: http://jimmyg.org/blog/2009/working-with-python-subprocess.html#what-happens-when-you-execute-a-command http://jimmyg.org/blog/2009/working-with-python-subprocess.h... (I.e., the first blue box in the article, "what happens when you execute a command") is wrong. It implies that the command (ls -l) is read by the terminal emulator, which gives it to bash, which gives it to the kernel. This is wrong in several ways. The terminal emulator starts bash (or whatever shell) and it is bash that directly reads the string "ls -l", figures out what it means, finds the ls program in PATH, and runs the ls binary. It is ls that calls kernel functions when it's doing its job. And of course, the kernel is not really of importance here. The selected command does whatever it does, which might not even access the kernel ("Hello, world"). And bash can and sometimes does directly call kernel functions without an intermediate binary (e.g., unlink(2) to overwrite a file for output redirection).

thwarted 15y ago
...the data flow diagram in the OP..."what happens when you execute a command"...is wrong. It implies that the command (ls -l) is read by the terminal emulator, which gives it to bash, which gives it to the kernel. It's accurate, the author just left off where the command is coming from (implying it's the user sitting at a keyboard). You type into the terminal-emulator, which passes that input to the program it's running, the shell, which figures out what it means. A terminal (or something emulating a terminal) is always between the keyboard and the input of the program running. Even when you login on the console, the terminal driver is accepting your input, transforming it in some fashion, and giving it to the nxt level. A terminal doesn't exist between this echo command and the sh program: echo ls -l | sh And is, in fact, why things like expect(1) exist and that ssh can be configured to allocate a pseudo-terminal or not depending on how you want/need to interact with the remote command. Incidentally, there seems to be a bug in yum whereby it detects if its output is a terminal or not tries to format it, but it gets the check reversed and pretty-print word-wraps output that goes to a non-terminal. Which is why yum search something and yum search something | sort produce different output and the latter has blank lines scattered throughout. (yum 3.2.28 on Fedora 14). And of course, the kernel is not really of importance here. The selected command does whatever it does, which might not even access the kernel ("Hello, world"). Every Hello, World program I've seen displays text to the user (or sends it somewhere), which means a system call, which is calling "kernel functions". The kernel is the mediator between programs/processes, there are very few cases where it isn't directly involved. And bash can and sometimes does directly call kernel functions without an intermediate binary (e.g., unlink(2) to overwrite a file for output redirection). Not really a good example, but the idea is sound. Bash has builtin functionality that emulate the functionality of system calls or other binaries. For this case, bash opens files for redirection when it's going to overwrite with the O_TRUNC flag set. This can be changed with various shell options (clobber) to have it not implicitly overwrite. You can see this with strace. And technically, "overwriting" a file means truncation or seek-and-write, not unlink and recreate. The difference has implications for hardlinked files.
- mturmon 15y ago
  OK, I see your point regarding the terminal emulator. There are some keys that are intercepted by the terminal emulator and processed by it alone (say, whatever does "print screen" or "close window"). The shell never sees these key presses. The thing I didn't (and still don't) like about the diagram is it implies the whole "ls -l" is delivered by the terminal emulator to the shell. Which isn't how it works. The emulator is transparent to all normal keystrokes and the shell is doing all the work, character by character, even for stuff like ctrl-H which, years ago, could have been (and sometimes was) intercepted and handled by the terminal driver. And the presence of the kernel in the diagram is also confused. (You are right, system calls are all over, even printf() will bottom out in the kernel; I was trying to say that there is no requirement for the user program to call in to the kernel.) Here's a sentence in the OP that has the same confusion: "When you execute a program from Python you can choose to execute it directly with the kernel or via a shell." The kernel is not "executing the program directly". This is nonsense. What they want to say is that you can use a system call (execve) to directly load and run the program, or you can use an interpreter like sh. This same point of confusion is exhibited in their diagram.

squeed 15y ago

Python's subprocess module is great. It is definitely the cleanest wrapping of a system program of any of the scripting languages I've ever used. When you just want output, you call .communicate() and be done with it. When you want to interact, the file descriptors are there for you to play with. It's scripting heaven. There is one feature that he could have touched on more: shell=False is the only easy, safe way to put user-supplied input in a command line. This stuff is just too damned easy to get wrong. I've seen heroic code that tried to work around shell quoting and escaping vulnerabilities. One colleague was trying to write a wrapper around the ldapsearch binary, and decided that filtering all "|" would do it. He'd completely forgotten that ` also triggers arbitrary command execution. Don't be a hero. Keep bash away from your input.