3 ms·

Agreed, Bing is my vote too...if only they could increase their coverage a bit!

• _delirium 16y ago
  I'm not sure how widespread it is, but a handful of Bing blindspots I've run across are something they can't do anything about without violating robots.txt: a decent number of sites block all crawlers by default, but whitelist GoogleBot, presumably from a "we don't want to be crawled, but we do want to be googleable" line of thought.
  • bad_user 16y ago
    http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must...
    • _delirium 16y ago
      Yeah, MSNbot isn't very friendly, so they somewhat bring it on themselves. Both it and Yahoo's bot are much worse than Google at predicting likely page updates, as well--- on one of my sites that has a blank robots.txt, MSNbot accounts for >10x as many hits as GoogleBot, yet Google still manages to keep its index just as up to date. Some people have had success with crawl-delay, though there are occasional reports of MSNbot not honoring that either.