I think there are plenty of other people crawling the web. There's Common Crawl, there's the Wayback machine... it's not just Google. Then there is a very lon

I think there are plenty of other people crawling the web. There's Common Crawl, there's the Wayback machine... it's not just Google. Then there is a very long tail of crawlers that show up in the logs for my small-potatoes personal website. Whatever they're doing, they seem to be existing in peace, at the very least. To some extent, I agree with this site that people are nicer to Google than other crawlers. That's because the crawl consumes their resources but provides benefits -- you show up on Google, the only search engine people actually use. But at the same time, they are happy to drag Google in front of Congress for some general abuse, so... maybe there is actually a little bit of balance there.