3 ms·
> Multi-threading is a must, when scraping at scale. The more concurrent requests your spiders can make the better your performance - simple. Intuitively I wou
by blattimwind 8y ago
> Multi-threading is a must, when scraping at scale. The more concurrent requests your spiders can make the better your performance - simple.
Intuitively I would think that this sort of problem would profit from using asynchronous ingestion at the edge pushing unprocessed contents to a multi-threaded/multi-process backend. (Because I'd expect that network latencies mean you need lots of threads to saturate I/O, which I'd expect would conflict with effectively using the available CPU power to do the actual document processing).
- skate22 8y agoI would expect that you would use both async requests & multithreading. I might restrict each thread to operate on an exclusive set of domains to throttle the request rate to any given domain at a point in time
- mgliwka 8y agoThat's been exactly my experience. Most time is spent connecting or waiting for the server response (TTFB). Using an async I/O event loop approach in combination with EPOLL/KQUEUE you can handle thousands of concurrent connections. You then push the response to your worker nodes, which process the data in a multi-threaded fashion. Stream Processing Frameworks like Apache Spark or Storm work great for that.