3 ms·
no. by end of 2010 (tentative).
by jayp 17y ago
no. by end of 2010 (tentative).
- danudey 17y agoI wish they'd hurry up, Yahoo's Slurp is the worst crawler I've seen. The other day it had 113 connections open to one of our sites. Jeff Atwood has also blocked them from StackOverflow because they were taking up a disproportionate amount of bandwidth and generally behaving like tools. I love flickr, but Yahoo Search needs to die.
- eli 17y agoMSN is far worse. On several of my sites, it routinely requests URLs that are 1) blocked in robots.txt and 2) have been 404 for years