Yes, Scrapy is quite a good scraper technology for some features, especially caching, but for some websites it's like doing things the hard way... The easiest

Yes, Scrapy is quite a good scraper technology for some features, especially caching, but for some websites it's like doing things the hard way... The easiest scraper with a proxy rotator I've found is in my current fave web-automator, scraper scripter and scheduler: Rtila [1] Created by an indy/solo developer-on-fire cranking out user-requested features quite quickly... check the releases page [2] I have used (or at least trialled) the vast majority of scraper-tech and written hundreds of scrapers since my first VB5 controlling IE then dumping to SQLserver in the 90's and then moving to various php and python libs/frameworks and a handful of windows apps like ubot and imacros (both of which were useful to me at some point but I never use those nowadays) A recent release of Rtila allows creating standalone bots you can run using it's built-in local Node.js server (which also has it's own locally hosted server API you can program anything else against using any language you like) [1] https://www.rtila.net https://www.rtila.net [2] https://github.com/IKAJIAN/rtila-releases/releases https://github.com/IKAJIAN/rtila-releases/releases