3 ms·
Congratulations on your success, Kevin and team. I couldn't find this on your website - does Scrapingbee respect robots.txt directives, or is there any other m
by niel 5y ago
Congratulations on your success, Kevin and team.
I couldn't find this on your website - does Scrapingbee respect robots.txt directives, or is there any other method for a website owner to limit or even just slow down your scraping?
- babelfish 5y agoThey almost certainly don't
- blep-arsh 5y agoWould be surprised if they do. My service is on the receiving end of these scrapers and it gets annoying when some "growth hacker" decides to flood the service with meaningless CPU-heavy requests with a bunch of typos in parameter values. The scraping service gets a $.0001 for every $1 of damage they help inflict, so yay, bootstrapped business.