3 ms·
> Just freakin learn perl or python! For calculations? Sure; but that's not really what bash is for. Shells are excellent at invoking and managing subprocesse
by chriswarbo 8y ago
> Just freakin learn perl or python!
For calculations? Sure; but that's not really what bash is for.
Shells are excellent at invoking and managing subprocesses and piping. Python (not sure about Perl, I've not really used it) is terrible at those things.
Take a simple, very common piece of bash code like `foo | bar`. How might we do this in Python? Maybe we reach for the builtin `subprocess` module:
import subprocess
foo_output = subprocess.check_call(['foo'])
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
bar.communicate(foo_output)
Except that this isn't a pipe: it will run `foo` to completion, storing all of the output in memory, then call `bar` on this data, e.g. `foo_output=$(foo); echo "$foo_output" | bar`. This is unsuitable for long-lived processes (e.g. if `foo` is long lived and `bar` is meant to be logging its output), or if there is a lot of data (e.g. `foo` is generating GBs of text and `bar` is summarising it, like `wc -l`).
OK, maybe instead of `check_call` (which is blocking) we make `foo` asynchronous. Note that we can't use `foo.communicate` to get its output, since that would also block. What if we just shuttle data between the two manually, line by line (urgh)?
import subprocess
foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE)
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
while foo.poll() is None:
bar.stdin.write(foo.stdout.readline())
bar.stdin.close()
bar.wait()
This appears to work, especially when testing with small amounts of data. Yet it's actually a timebomb, since it will deadlock when the subprocesses' pipe buffers fill up (this is why the documentation tells us to use `communicate`, except that we can't since that's blocking https://docs.python.org/3.7/library/subprocess.html https://docs.python.org/3.7/library/subprocess.html ).
It's at this point that we move the data shuttling into a separate thread:
import subprocess
import threading
foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE)
bar = subprocess.Popen(['bar'], stdin=subprocess.PIPE)
def shuttle():
while foo.poll() is not None:
bar.stdin.write(foo.stdout.readline())
foo.stdout.close()
bar.stdin.close()
thread = threading.Thread(target=shuttle)
thread.daemon = True
thread.start()
thread.join()
bar.wait()
Now we've got a pile of code mixing multiprocessing with multithreading, in a domain known to have deadlocks, with hand-written hard-coded line buffering.
At this point, I'd say just freakin learn bash!
- kamaal 8y ago>>not sure about Perl, I've not really used it Perl is great at these things. In fact you talk shell native tongue in Perl. use strict; use warnings; my @output = `your_command | your_another_command`; foreach my $line (@output) { chomp; #removes new line $line = $_; if ($line =~ /your regex goes here/) { #if match, use $1, $2... to get the groups matched #your buisness logic goes here } }
- tomc1985 8y agoI agree, subprocess is a pain in the ass. But there are other ways at invoking things, like with grave quotes. Personally I think control structures in bash are too convoluted and easy to get lost in. Also if you are running long-lived high-output processes and you want your program to be interactive, then of course you have to do multithreading shenanigans. At that point you're going to need a proper event queue or something at the very least. The first use case I can think of (logging dmesg -w) would definitely not work using a naive approach
- Too 8y agoI haven't run it but I'm quite sure you can do this to get direct pipe without having to run foo to completion before starting bar: foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE) bar_output = subprocess.check_output(['bar'], stdin=foo.stdout) Though the idea with python is that you shouldn't have to do as much piping as you do in bash, no need for xargs, cut, find, etc. Just process the data inside python code. So the pain of having to use two lines instead of a | isn't as big as many people claim.