3 ms·
http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must...
by bad_user 16y ago
http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must...
- _delirium 16y agoYeah, MSNbot isn't very friendly, so they somewhat bring it on themselves. Both it and Yahoo's bot are much worse than Google at predicting likely page updates, as well--- on one of my sites that has a blank robots.txt, MSNbot accounts for >10x as many hits as GoogleBot, yet Google still manages to keep its index just as up to date. Some people have had success with crawl-delay, though there are occasional reports of MSNbot not honoring that either.