4 ms·

Another Yahoo! bot behaving badly? Between them and MSN/Bing's bot, what the hell is going on with search engine spiders lately?

• jrockway 17y ago
  Imagine you have to write a computer program to visit every page of every website on the entire Internet. Sometimes you are going to make a mistake, and someone on the Internet is going to notice.
  • timdorr 17y ago
    Except this has been going on for months, has happened a number of times, and occurs in violation of standards, like robots.txt, that were created to prevent it from happening. I'd expect it from an individual or startup, but not a large company that's been doing spidering for a long time.