3 ms·

But whats the point of it downloading the robots.txt file if it isn't going to honor it?

• robtoo 15y ago
  Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.