5 ms·

Or, good enough for 95% of the use cases, simply use 'xargs' which is part of findutils and installed by default on all linux distributions. See the '-P/--max-p

by zevv 10y ago

Or, good enough for 95% of the use cases, simply use 'xargs' which is part of findutils and installed by default on all linux distributions. See the '-P/--max-procs' option in the man page for more details.

deleted 10y ago
[deleted]
bandrami 10y ago
Right, but it's those remaining 5% of situations where you want parallel execution on multiple remote hosts that make this cool (xargs itself can be replaced 95% of the time with backticks, for that matter). Combining parallel, pies[1], and shepherd[2] makes for some very cool possibilities, which you can check out with the GNU System Distribution[3], which I've recently migrated all my personal systems to. [1] http://www.gnu.org.ua/software/pies/ http://www.gnu.org.ua/software/pies/ [2] https://www.gnu.org/software/shepherd/ https://www.gnu.org/software/shepherd/ [3] https://www.gnu.org/software/guix/ https://www.gnu.org/software/guix/
- sigil 10y ago
  You can use xargs for parallel remote execution too. In the simplest case: list-hosts | xargs \ -P8 \ -n1 \ -I% \ ssh % some-command
  - bandrami 10y ago
    Sure, and you can also run an ssh command inside backticks too. There's something of a Turing tar pit here.
adrianN 10y ago
With xargs I always have the problem that lines get mangled when you have concurrent output, so you have to work around that. GNU parallel prevents this by some magic.
- sigil 10y ago
  The "magic" is simply output buffering inside GNU parallel...something you should be aware of if the concurrent programs are producing large amounts of output.
  - xorcist 10y ago
    If you find yourself depending on the order of output lines, your are probably better off using something like Python Fabric instead.
  - adrianN 10y ago
    Reproducing that behavior with just bash and xargs is too difficult for me.
  - ole_tange 10y ago
    "inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#Buffering-on-disk https://www.gnu.org/software/parallel/parallel_design.html#B...

by zevv 10y ago

deleted 10y ago

[deleted]

bandrami 10y ago

Right, but it's those remaining 5% of situations where you want parallel execution on multiple remote hosts that make this cool (xargs itself can be replaced 95% of the time with backticks, for that matter). Combining parallel, pies[1], and shepherd[2] makes for some very cool possibilities, which you can check out with the GNU System Distribution[3], which I've recently migrated all my personal systems to. [1] http://www.gnu.org.ua/software/pies/ http://www.gnu.org.ua/software/pies/ [2] https://www.gnu.org/software/shepherd/ https://www.gnu.org/software/shepherd/ [3] https://www.gnu.org/software/guix/ https://www.gnu.org/software/guix/

sigil 10y ago
You can use xargs for parallel remote execution too. In the simplest case: list-hosts | xargs \ -P8 \ -n1 \ -I% \ ssh % some-command
- bandrami 10y ago
  Sure, and you can also run an ssh command inside backticks too. There's something of a Turing tar pit here.

adrianN 10y ago

With xargs I always have the problem that lines get mangled when you have concurrent output, so you have to work around that. GNU parallel prevents this by some magic.

sigil 10y ago
The "magic" is simply output buffering inside GNU parallel...something you should be aware of if the concurrent programs are producing large amounts of output.
- xorcist 10y ago
  If you find yourself depending on the order of output lines, your are probably better off using something like Python Fabric instead.
- adrianN 10y ago
  Reproducing that behavior with just bash and xargs is too difficult for me.
- ole_tange 10y ago
  "inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#Buffering-on-disk https://www.gnu.org/software/parallel/parallel_design.html#B...