3 ms·
> "process these urls but only have 8 requests in flight at a time." What's the state of the art solution to this in node.js?
by davedx 8y ago
> "process these urls but only have 8 requests in flight at a time."
What's the state of the art solution to this in node.js?
- cloverich 8y agoThere's many ways to do this, but at its simplest because Node is single threaded, you can have a shared array of URL's and then a pool of workers who .pop() a URL off and stop when there are none left. Each worker's http request runs async, so with the `async` keyword at the start of their signature, and `await fetch(url)` inside, the code reads (and in some ways behaves) like regular synchronous iteration but runs (mostly) concurrently.
- always_good 8y agoconst results = await Promise.map(urls, url => process(url), { concurrency: 8 }) You can find a "map with concurrency limit" implementation in Bluebird (shown) or standalone on NPM. For example, off the top of my head this would probably take me 40 lines of ungeneral wait group code in Go.