That was very interesting. It's amusing to me how Google's crawler is more efficient than the competition. Maybe Google isn't a search business as much as a cra

That was very interesting. It's amusing to me how Google's crawler is more efficient than the competition. Maybe Google isn't a search business as much as a crawler business now! I've thought about updating my robots.txt but I really don't see the point. It's a cat and mouse game and there is no 'delete previous fetch' in robots.txt. I seriously doubt that they would update their index to remove contents if the robots.txt changed over time. Besides, it's going to be death by a thousand paper cuts, keeping up with all these robots. Most of the young ones certainly wouldn't abide by robots.txt anyway as they're just in alpha, right?... The cat's out of the bag now... The only real purpose of having a restrictive robots.txt is to reduce traffic, imho. Or to help these robots find content that they wouldn't normally dig "that deep".