8 ms·

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

chubot 5y ago
I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-automatically-parallelizes-tasks https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)
- leephillips 5y ago
  Remote execution.
  - chubot 5y ago
    I'd like to see a demo of it! I will try rewriting it with the $0 Dispatch Pattern and ssh :)
    - xmcqdpt2 5y ago
      Good luck balancing node usage! Here is an example of how it works, https://docs.computecanada.ca/mediawiki/index.php?title=GNU_Parallel&mobileaction=toggle_view_desktop#Running_on_Multiple_Nodes https://docs.computecanada.ca/mediawiki/index.php?title=GNU_... This + restart capabilities make gnu parallel very well suited to running 1000s of compute-heavy jobs on HPC clusters.
      - figomore 5y ago
        I used Parallel to distribute the rendering of a little Blender animation It worked very well. https://github.com/tfmoraes/blender_gnu_parallel_render/blob/master/render_blender_parallel.sh https://github.com/tfmoraes/blender_gnu_parallel_render/blob...
- comex 5y ago
  One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at all, instead buffering the entire output until the command exits and then printing it. With `--line-buffer` the unit of atomicity can be weakened from an entire command output to individual lines of output, reducing latency. Alternately, with `--keep-order`, `parallel` can ensure the outputs are printed in the same order as the corresponding inputs, which makes the output deterministic if the program is deterministic. Without that you'll get results in an arbitrary order. These aren't technically things that xargs and shell can't do; you could reimplement the same behavior by hand with the shell. But by the same token, there isn't anything xargs can do that the shell can't do alone; you could always use the shell to manually split up the input and invoke subprocesses. It's just a question of how much you want to reimplement by hand.
  - chubot 5y ago
    OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. You even have some notion of progress by checking the file size with ls -l. I tend to use a pattern where each task also outputs a metadata file: the exit status, along with the data from "time" (rusage, etc.) But I admit that this is annoying to rewrite in every script that uses xargs! It does make sense to have this functionality in a tool. But I think that tool should be a LANGUAGE like Oil, not a weirdo interface like GNU parallel :) But thanks for the explanation (and thanks to everyone in this subthread) -- I learned a bunch and this is why I write blog posts :)
    - Godel_unicode 5y ago
      Thank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python. For what it's worth, I consider oil to be closer to a unixy PowerShell rather than a more powerful bash. Note that this is not a slight, PowerShell is sweet for what it is. It (oil) really takes a hard left from the POSIX philosophy of focusing on one thing and doing it well. I'm also bitter that, if it's going to veer so far away from POSIX, that it didn't go the whole hundred and become a function language with comprehensions and such. For what it's worth, everything you mentioned above about your approach can be done with parallel.
- bsmithers 5y ago
  Not to be pedantic, but that's a bit of a non-argument. _Of course_ you can do it with xargs and shell, but imho parallel is generally more convenient, especially for remote execution. It provides a higher level of abstraction for such tasks.
- lacksconfidence 5y ago
  i don't know if xargs cant, but i use gnu parallel to split an input pipe into N parallel pipes processing slices of the input stream. Edit: To clarify, xargs usually wants to spin up a process per task. I have parallel spin up N processes and then continuously feed them.
- orf 5y ago
  Resumption, error reporting and much better progress monitoring.
  - vhold 5y ago
    Oh I didn't know about resumption.. parallel has so many features packed into its CLI it's kind of ridiculous. For others that didn't know about it, see the examples here: https://www.gnu.org/software/parallel/parallel_tutorial.html#progress-information https://www.gnu.org/software/parallel/parallel_tutorial.html... Here's another surprising feature: https://www.gnu.org/software/parallel/parallel_tutorial.html#limiting-the-resources https://www.gnu.org/software/parallel/parallel_tutorial.html...
    - O_H_E 5y ago
      Wow!, That is surprising and potentially very useful.
- bloopernova 5y ago
  GNU Parallel can be sourced into a bash session from a plain text file and used as a function. I've used it to get around overly-restrictive build environments. (overly restrictive because the team that manages the build image wasn't open to modifying their image for my use case)
- xmcqdpt2 5y ago
  Restart capability and remote executions make gnu parallel the tool if choice for HPC. For example, you might very well use gnu parallel to run 1000s of cpu-hours of numerical simulation using patterns such as these ones, https://docs.computecanada.ca/mediawiki/index.php?title=GNU_Parallel https://docs.computecanada.ca/mediawiki/index.php?title=GNU_... Using xargs for this kind of work is euhm... not a good idea.
- orhmeh09 5y ago
  Issue complaint prompts to promote the author, for one.
- MainJane 5y ago
  It is documented in the GNU Parallel documentation: https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... If you seriously believe you can implement everything using xargs, then this (contrived) example is for you: https://unix.stackexchange.com/questions/405552/using-xargs-instead-of-gnu-parallel https://unix.stackexchange.com/questions/405552/using-xargs-... Newer versions include 'parset' which can set shell variables in parallel, which is useful if you want to 'map' values from one array to another.
- enriquto 5y ago
  > What does it do that xargs and shell can't? (honest question) For me, an essential feature of GNU parallel is that it is semantically equivalent to "sh". Imagine that you write a file that contains a long list of commands. You can pipe that file to "sh" to run the commands, or pipe it to "parallel" to do the same, but faster. If you are building the list of commands on the fly, then you can use xargs with a slightly different syntax. But somehow using "sh" or "parallel" gives a certain peace of mind due to its straightforward semantics. I never used any argument of GNU parallel apart from -j My usage pattern: to build a list of commands explicitly then run it (possibly teeing the list into a temporary file to inspect it): for i in one two three; do printf "echo $i\n" done |sh # or |parallel
- oofabz 5y ago
  I use GNU Parallel for long-running jobs for its --eta option. If a job will take days or longer it's useful to know that early in the process. You might want to cancel it and try something else, and if you want to proceed with the long job you can make plans around when your data will be ready.
orhmeh09 5y ago
The nagware prompts of parallel are so objectionable that I will do a lot of things to avoid using it at all. So pretentious!
- grawlinson 5y ago
  Seems like some distributions patch out the nagware. I know Arch Linux does[0]. [0]: https://github.com/archlinux/svntogit-community/tree/packages/parallel/trunk https://github.com/archlinux/svntogit-community/tree/package...
- queuebert 5y ago
  It's also written in Perl!
  - orhmeh09 5y ago
    Veering off course here, after experiencing how incredibly long it took to install Sqitch, I will go out of my way to avoid anything that is more than a single script, certainly anything requiring CPAN too. I don’t think there’s anything technically wrong with these programs or with Perl, they’re just presented in ways that are unique hassles in this day and age.
    - MainJane 5y ago
      > anything that is more than a single script ... which is exactly what GNU Parallel is. Your concern is even mentioned in the design documentation: https://www.gnu.org/software/parallel/parallel_design.html https://www.gnu.org/software/parallel/parallel_design.html
    - queuebert 5y ago
      I remember when Perl was the coolest thing ever. I even wrote numerical simulations in it, just to try. Only with the invention of Python and Ruby did we realize how much better things could be. Of course that the Perl inventor was an IOCCC winner should've been a red flag.
- 5y ago
cormacrelf 5y ago
If you need more visibility into long running processes, pueue is another alternative. You can of course use `xargs -P1 pueue add ./process_file.sh` to add the jobs in the first place. Sends a job to pueued, returns immediately. Great for re-encoding dozens of videos. For jobs that aren’t already multi-core, set the queue parallelism with pueue, after you’ve seen your cpu is under-utilised. Obviously downside to the visibility and dynamism is that it redirects stdout. You can read it back later, in order. But it’s not there for continued processing immediately.

chubot 5y ago

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-automatically-parallelizes-tasks https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

leephillips 5y ago
Remote execution.
- chubot 5y ago
  I'd like to see a demo of it! I will try rewriting it with the $0 Dispatch Pattern and ssh :)
  - xmcqdpt2 5y ago
    Good luck balancing node usage! Here is an example of how it works, https://docs.computecanada.ca/mediawiki/index.php?title=GNU_Parallel&mobileaction=toggle_view_desktop#Running_on_Multiple_Nodes https://docs.computecanada.ca/mediawiki/index.php?title=GNU_... This + restart capabilities make gnu parallel very well suited to running 1000s of compute-heavy jobs on HPC clusters.
    - figomore 5y ago
      I used Parallel to distribute the rendering of a little Blender animation It worked very well. https://github.com/tfmoraes/blender_gnu_parallel_render/blob/master/render_blender_parallel.sh https://github.com/tfmoraes/blender_gnu_parallel_render/blob...
comex 5y ago
One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at all, instead buffering the entire output until the command exits and then printing it. With `--line-buffer` the unit of atomicity can be weakened from an entire command output to individual lines of output, reducing latency. Alternately, with `--keep-order`, `parallel` can ensure the outputs are printed in the same order as the corresponding inputs, which makes the output deterministic if the program is deterministic. Without that you'll get results in an arbitrary order. These aren't technically things that xargs and shell can't do; you could reimplement the same behavior by hand with the shell. But by the same token, there isn't anything xargs can do that the shell can't do alone; you could always use the shell to manually split up the input and invoke subprocesses. It's just a question of how much you want to reimplement by hand.
- chubot 5y ago
  OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. You even have some notion of progress by checking the file size with ls -l. I tend to use a pattern where each task also outputs a metadata file: the exit status, along with the data from "time" (rusage, etc.) But I admit that this is annoying to rewrite in every script that uses xargs! It does make sense to have this functionality in a tool. But I think that tool should be a LANGUAGE like Oil, not a weirdo interface like GNU parallel :) But thanks for the explanation (and thanks to everyone in this subthread) -- I learned a bunch and this is why I write blog posts :)
  - Godel_unicode 5y ago
    Thank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python. For what it's worth, I consider oil to be closer to a unixy PowerShell rather than a more powerful bash. Note that this is not a slight, PowerShell is sweet for what it is. It (oil) really takes a hard left from the POSIX philosophy of focusing on one thing and doing it well. I'm also bitter that, if it's going to veer so far away from POSIX, that it didn't go the whole hundred and become a function language with comprehensions and such. For what it's worth, everything you mentioned above about your approach can be done with parallel.
bsmithers 5y ago
Not to be pedantic, but that's a bit of a non-argument. _Of course_ you can do it with xargs and shell, but imho parallel is generally more convenient, especially for remote execution. It provides a higher level of abstraction for such tasks.
lacksconfidence 5y ago
i don't know if xargs cant, but i use gnu parallel to split an input pipe into N parallel pipes processing slices of the input stream. Edit: To clarify, xargs usually wants to spin up a process per task. I have parallel spin up N processes and then continuously feed them.
orf 5y ago
Resumption, error reporting and much better progress monitoring.
- vhold 5y ago
  Oh I didn't know about resumption.. parallel has so many features packed into its CLI it's kind of ridiculous. For others that didn't know about it, see the examples here: https://www.gnu.org/software/parallel/parallel_tutorial.html#progress-information https://www.gnu.org/software/parallel/parallel_tutorial.html... Here's another surprising feature: https://www.gnu.org/software/parallel/parallel_tutorial.html#limiting-the-resources https://www.gnu.org/software/parallel/parallel_tutorial.html...
  - O_H_E 5y ago
    Wow!, That is surprising and potentially very useful.
bloopernova 5y ago
GNU Parallel can be sourced into a bash session from a plain text file and used as a function. I've used it to get around overly-restrictive build environments. (overly restrictive because the team that manages the build image wasn't open to modifying their image for my use case)
xmcqdpt2 5y ago
Restart capability and remote executions make gnu parallel the tool if choice for HPC. For example, you might very well use gnu parallel to run 1000s of cpu-hours of numerical simulation using patterns such as these ones, https://docs.computecanada.ca/mediawiki/index.php?title=GNU_Parallel https://docs.computecanada.ca/mediawiki/index.php?title=GNU_... Using xargs for this kind of work is euhm... not a good idea.
orhmeh09 5y ago
Issue complaint prompts to promote the author, for one.
MainJane 5y ago
It is documented in the GNU Parallel documentation: https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... If you seriously believe you can implement everything using xargs, then this (contrived) example is for you: https://unix.stackexchange.com/questions/405552/using-xargs-instead-of-gnu-parallel https://unix.stackexchange.com/questions/405552/using-xargs-... Newer versions include 'parset' which can set shell variables in parallel, which is useful if you want to 'map' values from one array to another.
enriquto 5y ago
> What does it do that xargs and shell can't? (honest question) For me, an essential feature of GNU parallel is that it is semantically equivalent to "sh". Imagine that you write a file that contains a long list of commands. You can pipe that file to "sh" to run the commands, or pipe it to "parallel" to do the same, but faster. If you are building the list of commands on the fly, then you can use xargs with a slightly different syntax. But somehow using "sh" or "parallel" gives a certain peace of mind due to its straightforward semantics. I never used any argument of GNU parallel apart from -j My usage pattern: to build a list of commands explicitly then run it (possibly teeing the list into a temporary file to inspect it): for i in one two three; do printf "echo $i\n" done |sh # or |parallel
oofabz 5y ago
I use GNU Parallel for long-running jobs for its --eta option. If a job will take days or longer it's useful to know that early in the process. You might want to cancel it and try something else, and if you want to proceed with the long job you can make plans around when your data will be ready.

orhmeh09 5y ago

The nagware prompts of parallel are so objectionable that I will do a lot of things to avoid using it at all. So pretentious!

grawlinson 5y ago
Seems like some distributions patch out the nagware. I know Arch Linux does[0]. [0]: https://github.com/archlinux/svntogit-community/tree/packages/parallel/trunk https://github.com/archlinux/svntogit-community/tree/package...
queuebert 5y ago
It's also written in Perl!
- orhmeh09 5y ago
  Veering off course here, after experiencing how incredibly long it took to install Sqitch, I will go out of my way to avoid anything that is more than a single script, certainly anything requiring CPAN too. I don’t think there’s anything technically wrong with these programs or with Perl, they’re just presented in ways that are unique hassles in this day and age.
  - MainJane 5y ago
    > anything that is more than a single script ... which is exactly what GNU Parallel is. Your concern is even mentioned in the design documentation: https://www.gnu.org/software/parallel/parallel_design.html https://www.gnu.org/software/parallel/parallel_design.html
  - queuebert 5y ago
    I remember when Perl was the coolest thing ever. I even wrote numerical simulations in it, just to try. Only with the invention of Python and Ruby did we realize how much better things could be. Of course that the Perl inventor was an IOCCC winner should've been a red flag.
5y ago

cormacrelf 5y ago

If you need more visibility into long running processes, pueue is another alternative. You can of course use `xargs -P1 pueue add ./process_file.sh` to add the jobs in the first place. Sends a job to pueued, returns immediately. Great for re-encoding dozens of videos. For jobs that aren’t already multi-core, set the queue parallelism with pueue, after you’ve seen your cpu is under-utilised. Obviously downside to the visibility and dynamism is that it redirects stdout. You can read it back later, in order. But it’s not there for continued processing immediately.