3 ms·
You can have remote workers with parallel (not sure if you can do that with xargs or not). parallel --progress --wd '.' -S '4/fourworkers@blehworker.org,8/
by perturbation 10y ago
You can have remote workers with parallel (not sure if you can do that with xargs or not).
parallel --progress --wd '.' -S '4/fourworkers@blehworker.org,8/eightworkers@eightworkers.org,2/:' "./complicated_command --param1={1} --param2={2} ::: $(some_shellcommand_to_generate_param1_values.sh) ::: $(some_shellcommand_to_generate_param2_values.sh)
It's very flexible, become my go-to for speeding up parallelizable work. Next step up would be Celery or Spark or something.