2 ms·
You can do this with xargs -P (with GNU xargs at least). I often use the pattern: do_one_thing() { local arg=$1 ... do stuff with $arg }
by chubot 10y ago
You can do this with xargs -P (with GNU xargs at least). I often use the pattern:
do_one_thing() {
local arg=$1
... do stuff with $arg
}
do_with_limited_parallelism() {
cat things-to-do.txt | xargs -P 10 -n 1 -- $0 do_one_thing "$@"
}
"$@"
That limits the number of concurrent processes, which is almost always good enough for limiting the number of concurrent downloads. If you have really tiny downloads, then the overhead of starting a connection per process may matter. In that case it's easy to do more than one download per process with say -n 50 instead of -n 1.
- gpderetta 10y agoxargs -P is great and I have used it often for quick parallelization. The problem I had was that the crawler would recursively invoke itself while crawling the (possibly unbalanced) tree so xargs wouldn't have worked.
- chubot 10y agoOK, I see. Your implementation sounds a little like how "make -j" is actually implemented (in C). This basically sounds like implementing a semaphore with a pipe. http://make.mad-scientist.net/papers/jobserver-implementation/ http://make.mad-scientist.net/papers/jobserver-implementatio... """ Rather than using complex schemes such as a central server, shared memory, sockets, etc. for synchronizing the various instances of make, he suggested we use the simplest, most basic UNIX construct: a pipe. Not only is this easy, but it’s also a relatively portable concept. Most operating systems these days implement some sort of pipe feature, because that paradigm is so useful. The idea is ingeniously simple: the initial, top-level make creates a pipe and writes N one-byte tokens into the pipe. That pipe is shared between that make and all submakes, and any time any make wants to run a job it first has to read a token from the pipe. Once the job is complete, it writes the token back to the pipe. Since there are only N tokens, you know that you will never invoke more than N jobs. """
- gpderetta 10y agoYes, except that gmake implementation is brilliant as it uses the pipe itself as a semaphore, while my version had an explicit process keeping a count and listening to V and P messages. Unfortunately I didn't know how make -j was implemented at that time.
- Hello71 10y agomkfifo queue run_crawler root > queue & parallel -a queue run_crawler > queue add --group or -k to taste.
- gpderetta 10y agoNice. I do need to add parallel to the my set of bash tricks. So, is it just me that keep refining bash 'one liners' well past the point they should be implemented on a real language?