3 ms·

What about & and wait? Could it have been an adequate alternative?

eisbaw 3y ago
No, that is more messy and can easily leave lingering processes. But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2e160988a https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2... GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.
- untilted 3y ago
  GNU parallel supports bash functions, provided you "export -f" them beforehand
capableweb 3y ago
Probably for very simple use cases, but the real power in parallel really comes from the myriad of switches that enables so much more than what "&" and "wait" could do. Here are a bunch of examples: https://www.gnu.org/software/parallel/parallel_examples.html https://www.gnu.org/software/parallel/parallel_examples.html A fun one I end up using ~monthly or so for various things (usually with more switches added as needed): GNU Parallel as queue system/batch manager # start queue true >jobqueue; tail -n+0 -f jobqueue | parallel # add job echo my_command my_arg >> jobqueue # to start queue for remote execution true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..
agumonkey 3y ago
You just taught me something
klyrs 3y ago
When I'm using parallel, it's usually because I have thousands of jobs. Worse, they have nontrivial memory requirements. When you background processes with &, the system starts timeslicing. Each process gets to allocate its memory before being paused to make time for the next process. Your system will almost immediately crumple under load. Hopefully, the oom killer will target your backgrounded jobs... but the script spawning them will go untouched because it isn't the thing hogging memory. Before I learned of parallel, I tried a hack where I'd manually assemble jobs into batches, and wait on the batches before starting the next. It achieved very low system utilization, because inevitably, one job each the batch takes much longer than the rest. A slight improvement (still not good), is to use `split` to chop your jobs file into $num_cores chunks, and background each chunk. But still, this gets low utilization. Problem being that you aren't using a thread/worker pool. Parallel (or, TIL, xargs) can maintain 100% system utilization, until the very last $num_cores jobs.

No, that is more messy and can easily leave lingering processes. But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2e160988a https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2... GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.

untilted 3y ago
GNU parallel supports bash functions, provided you "export -f" them beforehand

capableweb 3y ago

Probably for very simple use cases, but the real power in parallel really comes from the myriad of switches that enables so much more than what "&" and "wait" could do. Here are a bunch of examples: https://www.gnu.org/software/parallel/parallel_examples.html https://www.gnu.org/software/parallel/parallel_examples.html A fun one I end up using ~monthly or so for various things (usually with more switches added as needed): GNU Parallel as queue system/batch manager # start queue true >jobqueue; tail -n+0 -f jobqueue | parallel # add job echo my_command my_arg >> jobqueue # to start queue for remote execution true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..

agumonkey 3y ago

You just taught me something

klyrs 3y ago

When I'm using parallel, it's usually because I have thousands of jobs. Worse, they have nontrivial memory requirements. When you background processes with &, the system starts timeslicing. Each process gets to allocate its memory before being paused to make time for the next process. Your system will almost immediately crumple under load. Hopefully, the oom killer will target your backgrounded jobs... but the script spawning them will go untouched because it isn't the thing hogging memory. Before I learned of parallel, I tried a hack where I'd manually assemble jobs into batches, and wait on the batches before starting the next. It achieved very low system utilization, because inevitably, one job each the batch takes much longer than the rest. A slight improvement (still not good), is to use `split` to chop your jobs file into $num_cores chunks, and background each chunk. But still, this gets low utilization. Problem being that you aren't using a thread/worker pool. Parallel (or, TIL, xargs) can maintain 100% system utilization, until the very last $num_cores jobs.