3 ms·
One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily b
by comex 5y ago
One thing parallel can do better than xargs is collect output.
If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough).
In contrast, `parallel` by default doesn't mix together output from different commands at all, instead buffering the entire output until the command exits and then printing it.
With `--line-buffer` the unit of atomicity can be weakened from an entire command output to individual lines of output, reducing latency.
Alternately, with `--keep-order`, `parallel` can ensure the outputs are printed in the same order as the corresponding inputs, which makes the output deterministic if the program is deterministic. Without that you'll get results in an arbitrary order.
These aren't technically things that xargs and shell can't do; you could reimplement the same behavior by hand with the shell. But by the same token, there isn't anything xargs can do that the shell can't do alone; you could always use the shell to manually split up the input and invoke subprocesses. It's just a question of how much you want to reimplement by hand.
- chubot 5y agoOK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. You even have some notion of progress by checking the file size with ls -l. I tend to use a pattern where each task also outputs a metadata file: the exit status, along with the data from "time" (rusage, etc.) But I admit that this is annoying to rewrite in every script that uses xargs! It does make sense to have this functionality in a tool. But I think that tool should be a LANGUAGE like Oil, not a weirdo interface like GNU parallel :) But thanks for the explanation (and thanks to everyone in this subthread) -- I learned a bunch and this is why I write blog posts :)
- Godel_unicode 5y agoThank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python. For what it's worth, I consider oil to be closer to a unixy PowerShell rather than a more powerful bash. Note that this is not a slight, PowerShell is sweet for what it is. It (oil) really takes a hard left from the POSIX philosophy of focusing on one thing and doing it well. I'm also bitter that, if it's going to veer so far away from POSIX, that it didn't go the whole hundred and become a function language with comprehensions and such. For what it's worth, everything you mentioned above about your approach can be done with parallel.
- cben 5y agoThe point of oil that there are really basic things like safe quoting that shells should do well, yet none of the posix shell do! Functional: there are interesting shells like Elvish. But it really goes PowerShell by adding internal rich data pipelines that dont have a unixy stream-of-bytes representation. Oil does NOT go that way; it works on stuff like QSN to make pure unix interconnects more robust.
- scottlamb 5y agoA lot of this comes down to familiarity. I tend to use "make -j 100" for what you're describing. If I write the Makefile carefully [1], it will handle resuming a half-finished job. I just looked and GNU parallel has a --resume argument which probably does something similar, and maybe with less hassle. But I don't do this often enough—and/or GNU parallel isn't "better enough"—that I'm likely to ever invest the time to learn GNU parallel. btw, oil looks very cool. I hate how many footguns are in common shells. [1] eg writing to a tempfile and atomically renaming into place: "task_with_stdout > $dir/task_id.txt.tmp && mv $dir/task_id.txt{.tmp,}"
- MainJane 5y agoYour `do_one`: * does not buffer stderr * does not check if the disk is full for a period of time during a task (thus risking incomplete output) * does not clean up, if killed * does not work correctly if task_with_stdout is a composed command Given that GNU Parallel is a drop-in replacement for xargs, I am curious why you find it a 'weirdo interface'.