I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapyli

I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapylib/blob/master/scrapylib/deltafetch.py https://github.com/scrapinghub/scrapylib/blob/master/scrapyl... and lots of support and help from the community. Admittedly writing this from scratch requires learning the framework, but being able to share and reuse code is a huge win (disclaimer: I'm a Scrapy contributor).