4 ms·

> Python has os.spawnl, os.spawnv, etc., which fork()s, wait4()s, etc., without involving a shell. Good. How do you pipeline commands with these?

kragen 1y ago
These functions can't do it. In Python you have to use the subprocess module if you want to pipeline commands without the bugs introduced by the shell. From https://docs.python.org/3.7/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3.7/library/subprocess.html#replacin...: p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Of course, now, nobody has an hda, and dmesg is root-only. A more modern example is in http://canonical.org/~kragen/sw/dev3/whereroot.py http://canonical.org/~kragen/sw/dev3/whereroot.py: p1 = subprocess.Popen(["df"], stdout=subprocess.PIPE) p2 = subprocess.Popen(["grep", "/$"], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() return p2.communicate()[0] Note that the result here is a byte string, so if you want to print it out safely without the shell-like bugginess induced by Python's default character handling (what happens if the device name isn't valid UTF-8?), you have to do backflips with sys.stdout.buffer or UTF-8B. Python got a lot of things wrong, and it gets worse all the time, but for now spawning subprocesses is one of the things it got right. Although, unlike IIRC Tcl, it doesn't raise an exception by default if one of the commands fails. Apart from the semantics of the operations, you could of course desire a better notation for them. In Python you could maybe achieve something like (cmd(["df"]) | ["grep", "/$"]).output() but that is secondary to being able to safely handle arguments containing spaces and pipes and whatnot.
- oguz-ismail 1y ago
  Dunno, so much work to achieve so little. I'm even more inclined to stick with shell scripts now
  - kragen 1y ago
    The Bourne shell is definitely less work unless you want your code to correctly or reliably handle user input. Then it's more work.
    - oguz-ismail 1y ago
      Not in my experience. Any concrete examples off the top of your head, where it's more work than setting up pipes in python manually?
      - baobun 1y ago
        Fill in the blank to run a docker container which opens the file with user-provided path in (say) vim. docker run --rm -it ...? Now run a container doing the exact same thing ("docker-in-docker"). docker run --rm -it -v $DOCKER_HOST:/var/run/docker.sock ...?
        oguz-ismail 1y ago
        > Fill in the blank to run a docker container which opens the file with user-provided path in (say) vim. Never used docker before, but this seems to work: docker run --rm -it debian bash -c 'vim -- "$1"' _ "$user_provided_path"
        kragen 1y ago
        Looks relatively safe to me, though it doesn't seem to work because debian:latest doesn't have vim in it (so I'm skeptical of your implicit claim of having tried it), and, if $user_provided_path is empty, it defaults to browsing the filesystem. But there are a lot of characters there that are specifically there to avoid footguns; without them, it would seem to work, but it would fail when $user_provided_path contained special characters. The version I tested was docker run --rm -it debian bash -c 'apt update; apt install -y vim; vim -- "$1"' _ "$user_provided_path"

kragen 1y ago

These functions can't do it. In Python you have to use the subprocess module if you want to pipeline commands without the bugs introduced by the shell. From https://docs.python.org/3.7/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/3.7/library/subprocess.html#replacin...: p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Of course, now, nobody has an hda, and dmesg is root-only. A more modern example is in http://canonical.org/~kragen/sw/dev3/whereroot.py http://canonical.org/~kragen/sw/dev3/whereroot.py: p1 = subprocess.Popen(["df"], stdout=subprocess.PIPE) p2 = subprocess.Popen(["grep", "/$"], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() return p2.communicate()[0] Note that the result here is a byte string, so if you want to print it out safely without the shell-like bugginess induced by Python's default character handling (what happens if the device name isn't valid UTF-8?), you have to do backflips with sys.stdout.buffer or UTF-8B. Python got a lot of things wrong, and it gets worse all the time, but for now spawning subprocesses is one of the things it got right. Although, unlike IIRC Tcl, it doesn't raise an exception by default if one of the commands fails. Apart from the semantics of the operations, you could of course desire a better notation for them. In Python you could maybe achieve something like (cmd(["df"]) | ["grep", "/$"]).output() but that is secondary to being able to safely handle arguments containing spaces and pipes and whatnot.

oguz-ismail 1y ago
Dunno, so much work to achieve so little. I'm even more inclined to stick with shell scripts now
- kragen 1y ago
  The Bourne shell is definitely less work unless you want your code to correctly or reliably handle user input. Then it's more work.
  - oguz-ismail 1y ago
    Not in my experience. Any concrete examples off the top of your head, where it's more work than setting up pipes in python manually?
    - baobun 1y ago
      Fill in the blank to run a docker container which opens the file with user-provided path in (say) vim. docker run --rm -it ...? Now run a container doing the exact same thing ("docker-in-docker"). docker run --rm -it -v $DOCKER_HOST:/var/run/docker.sock ...?
      - oguz-ismail 1y ago
        > Fill in the blank to run a docker container which opens the file with user-provided path in (say) vim. Never used docker before, but this seems to work: docker run --rm -it debian bash -c 'vim -- "$1"' _ "$user_provided_path"
        kragen 1y ago
        Looks relatively safe to me, though it doesn't seem to work because debian:latest doesn't have vim in it (so I'm skeptical of your implicit claim of having tried it), and, if $user_provided_path is empty, it defaults to browsing the filesystem. But there are a lot of characters there that are specifically there to avoid footguns; without them, it would seem to work, but it would fail when $user_provided_path contained special characters. The version I tested was docker run --rm -it debian bash -c 'apt update; apt install -y vim; vim -- "$1"' _ "$user_provided_path"