That’s fascinating. But assuming you aren’t one of the top two or three search engines, wouldn’t it hurt you as much if not more to stop crawling a major site l

That’s fascinating. But assuming you aren’t one of the top two or three search engines, wouldn’t it hurt you as much if not more to stop crawling a major site like Nike which visitors surely expect to be crawled? If you are one of the top two or three, I would like to think their crawlers are amazingly complex and robust but maybe I’m just naive.