We released estela for this and other purposes, check it out, maybe it will suit your needs: https://github.com/bitmakerla/estela https://github.com/bitmakerla

We released estela for this and other purposes, check it out, maybe it will suit your needs: https://github.com/bitmakerla/estela https://github.com/bitmakerla/estela Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. It would be great to have feedback to consider it when prioritizing some over others :-)