I've been working on this: https://github.com/schollz/crawdad https://github.com/schollz/crawdad. Its a redis-backed distributed scraper that's easy to continu

I've been working on this: https://github.com/schollz/crawdad https://github.com/schollz/crawdad. Its a redis-backed distributed scraper that's easy to continue after interruptions.