2 ms·
I encourage developers thinking of doing this to check that they aren't required to have their website be scraper-friendly first. The company I work for does a
by danpalmer 9y ago
I encourage developers thinking of doing this to check that they aren't required to have their website be scraper-friendly first.
The company I work for does a large amount of scraping of partner websites, with whom we have contracts that allow us to do it and that someone in their company signed off, but we still get blocked and throttled by tech teams who think they are helping by blocking bots. If we can't scrape a site we just turn off the partner, and that means lost business for them.