5 ms·
I always tended to autogenerate a makefile with fake goals like: g1: job1 g2: job2 and then one goal to depend on them all and I used
by thealistra 10y ago
I always tended to autogenerate a makefile with fake goals like:
g1:
job1
g2:
job2
and then one goal to depend on them all and I used make -j 4
- kstrauser 10y agoparallel is awesome for when you want to want to run a number of jobs unknown in advance, such as using it as an xargs replacement: find . -type f -name | parallel --jobs 5 process_file That will run "process_file foo" for ever file in a directory with 5 processes running in parallel.
- coredog64 10y agoNot being a jerk, but what does parallel bring to the table that xargs doesn't? find . -type f -name | xargs -I {} -n 1 -P 5 process_file {}
- perturbation 10y agoYou can have remote workers with parallel (not sure if you can do that with xargs or not). parallel --progress --wd '.' -S '4/fourworkers@blehworker.org,8/eightworkers@eightworkers.org,2/:' "./complicated_command --param1={1} --param2={2} ::: $(some_shellcommand_to_generate_param1_values.sh) ::: $(some_shellcommand_to_generate_param2_values.sh) It's very flexible, become my go-to for speeding up parallelizable work. Next step up would be Celery or Spark or something.
- ole_tange 10y agoBigger list: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
- ole_tange 10y agoFrom: https://www.gnu.org/software/parallel/history.html https://www.gnu.org/software/parallel/history.html > parallel dates back to around the same time. It was originally a wrapper that generated a makefile and used make -j to do the parallelization.