3 ms·
I wanted to do some larger distributed scraping jobs recently and although it was easy to get everything running on one machine (with different tools including
by tluyben2 6y ago
I wanted to do some larger distributed scraping jobs recently and although it was easy to get everything running on one machine (with different tools including Scrapy), I was surprised how hard it was to do at scale. The open source ones I could find was hard/impossible to get working, overly complex, badly documented etc.
The services I found to be reasonably priced for small jobs, but at scale they quickly become vastly more expensive than setting this up yourself. Especially when you need to run these jobs every month or so. Even if you have to write some code to make the open source solutions actually work.
- nr2x 6y agoIt gets more complicated when you need to leverage real browser engines (eg Chrome). I've got jobs spread across ~ 20 machines/ 140 concurrent browser instances, it's non-trivial.
- turtlebits 6y agoAWS Lambdas are an easy way to get scheduled scraping jobs running. I use their Python-based chalice framework (https://github.com/aws/chalice https://github.com/aws/chalice) which allows you to add a decorator to a method for a schedule, @app.schedule(Rate(30, unit=Rate.MINUTES)) It's also a breeze to deploy. chalice deploy