A lot of websites are using cloudflare which does make scraping quite difficult (just by default I think). Spoofing your user agent is a must if you need to do

A lot of websites are using cloudflare which does make scraping quite difficult (just by default I think). Spoofing your user agent is a must if you need to do anything nowadays. To your second point that same would be applicable to Google and Bing and any other search engine. Even if you follow robots.txt and consume an equal or lesser bandwidth it does not matter much if you aren't an established player.