This is not really about Google. Websites block crawlers because they get abused / crashed by Crawlers. In the early days (2000-2010) Google not only got bann

This is not really about Google. Websites block crawlers because they get abused / crashed by Crawlers. In the early days (2000-2010) Google not only got banned by some websites, it even got DNS-banned for abusing some DNS domains. You see, Google already has already built the "megacrawlers" described in this article, it can melt any website on the Internet, even Facebook - the largest, and they paid a high price for letting the early Google crawlers run free. Google today has a rate-limit for every single website and DNS sub-domain on the internet. For small websites the default is a handful of web pages every few seconds. Google has a very slow (days) algorithm to increase its crawl rate, and a very fast (1d) algorithm to cut the rate limit if it's getting any of the errors likely due to website overload. To summarize, Google has several layers of congestion control custom-designed into the crawl application. Most small web crawlers have zero. None of these other crawlers have figured this out, so they abuse websites, causing all small-scale crawlers to get banned. - ex-Google Crawl SRE