There are many legitimate reasons for web scraping and I think it’s fair that you set out your desires for scraping in your robots.txt file. You can block other

There are many legitimate reasons for web scraping and I think it’s fair that you set out your desires for scraping in your robots.txt file. You can block other bots while allowing Googlebot. Alternatively, you can also use a WAF that restricts bot access. But I do have a problem with scraping other sites to detract traffic away from them. Perhaps if it’s for data analysis. Also, I would be very careful as some governments have very strict privacy laws (EU, for example) and you never know what you are scraping. But one of the other problems is that web scrapers can be used for very nefarious purposes – see this article on web scraping attacks: https://www.incapsula.com/web-application-security/web-scraping-attack.html https://www.incapsula.com/web-application-security/web-scrap...