4 ms·
I can run a set # of jobs at a time, working through the list until the set is exhausted - a simple pool of 'workers'. It buffers child process output and emit
by kyleburton 17y ago
I can run a set # of jobs at a time, working through the list until the set is exhausted - a simple pool of 'workers'.
It buffers child process output and emits it when the child completes - this keeps concurrent process output from being as confusing.
Those were features I wanted at the time and '&' with wait didn't provide them.
- RiderOfGiraffes 17y agoAh, simply saying "parallelize" put me on the wrong track. I didn't realize you wanted to limit the number running simultaneously. Cool you discovered that xargs can do it - I didn't know that. I have, however, done this before. I have a "load_wait" command which waits for machine load to drop below a certain level before launching the command. A simple "sleep 1" between instances of that then lets me launch thousands of commands without killing the machine. sleep 1 ; load_wait 3 <command1> & sleep 1 ; load_wait 3 <command2> & sleep 1 ; load_wait 3 <command3> & sleep 1 ; load_wait 3 <command4> & ... I then launch those via "system" from inside awk and I'm done. The commands tend to be in muscle memory.