17 ms·
> I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine. Scraping is an embarrassin
by kortilla 6y ago
> I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine.
Scraping is an embarrassingly perfect scenario for coroutines. Most asynchronous frameworks even use scraping as one of the examples.
In short, it would probably be done in 15 minutes, assuming you don’t get throttled quickly. If the tool wasn’t already async capable, another 15 minutes to wrap some scraping in gevent/eventlet.
- nijave 6y agoEven without async it's pretty easy to slap a concurrent.futures ThreadPool on something normally single threaded and get massive performance gains.