So much this! I work in E-Commerce and (needless to say) we scrape a lot of websites. Due to our growth and the increase in scrapers we require, I’ve been writ

So much this! I work in E-Commerce and (needless to say) we scrape a lot of websites. Due to our growth and the increase in scrapers we require, I’ve been writing a proposal to a higher up to talk to our biggest competitors to all set up a public API that batches the data for a smaller amount of requests. It would save everyone quite some traffic and effort.