3 ms·

(author here) Hm I don't see either of these points because: GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I shoul

by chubot 5y ago

(author here) Hm I don't see either of these points because: GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post) xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds. -- edit: Logging can also be done with by recursively invoking shell functions that log with the $0 Dispatch Pattern, explained in the post. I don't see a need for another tool; this is the Unix philosophy and compositionality of shell at work :)

deleted 5y ago
[deleted]
jeffbee 5y ago
Parallel's killer feature is how it spools subprocess output, ensuring that it doesn't get jumbled together. xargs can't do that. I use parallel for things like shelling out to 10000 hosts and getting some statistics. If I use xargs the output stomps all over itself.
- chubot 5y ago
  Ah OK thanks, I responded to this here: https://news.ycombinator.com/item?id=28259473 https://news.ycombinator.com/item?id=28259473
Godel_unicode 5y ago
As far as I'm aware, xargs still has the problem of multiple jobs being able to write to stdout at the same time, potentially causing their output streams to be intermingled. Compare this with parallels --group. Also parallels can run some of those threads on remote machines. I don't believe xargs has an equivalent job management function.
MainJane 5y ago
In your examples you fail to put 'xargs -P' in the middle of a pipeline: You only put it at the end. In other words: some command | xargs -P other command | third command This is useful if 'other command' is slow. If you buffer on disk, you need to clean up after each task: Maybe there is not enough free disk space to buffer the output of all tasks. UNIX is great in that you can pipe commands together, but due to the interleaving issue 'xargs -P' fails here. It does not live up to the UNIX philosophy. Which is probably why you unconsciously only use it at the end of a pipeline. You can find a different counterexample on https://unix.stackexchange.com/questions/405552/using-xargs-instead-of-gnu-parallel https://unix.stackexchange.com/questions/405552/using-xargs-... I will be impressed if you can implement that using xargs. Especially if you can make it more clean than the paralel version.

by chubot 5y ago

deleted 5y ago

[deleted]

jeffbee 5y ago

Parallel's killer feature is how it spools subprocess output, ensuring that it doesn't get jumbled together. xargs can't do that. I use parallel for things like shelling out to 10000 hosts and getting some statistics. If I use xargs the output stomps all over itself.

chubot 5y ago
Ah OK thanks, I responded to this here: https://news.ycombinator.com/item?id=28259473 https://news.ycombinator.com/item?id=28259473

Godel_unicode 5y ago

As far as I'm aware, xargs still has the problem of multiple jobs being able to write to stdout at the same time, potentially causing their output streams to be intermingled. Compare this with parallels --group. Also parallels can run some of those threads on remote machines. I don't believe xargs has an equivalent job management function.

MainJane 5y ago

In your examples you fail to put 'xargs -P' in the middle of a pipeline: You only put it at the end. In other words: some command | xargs -P other command | third command This is useful if 'other command' is slow. If you buffer on disk, you need to clean up after each task: Maybe there is not enough free disk space to buffer the output of all tasks. UNIX is great in that you can pipe commands together, but due to the interleaving issue 'xargs -P' fails here. It does not live up to the UNIX philosophy. Which is probably why you unconsciously only use it at the end of a pipeline. You can find a different counterexample on https://unix.stackexchange.com/questions/405552/using-xargs-instead-of-gnu-parallel https://unix.stackexchange.com/questions/405552/using-xargs-... I will be impressed if you can implement that using xargs. Especially if you can make it more clean than the paralel version.