4 ms·

I don't think it's ignoring robots.txt, it's probably just cached at another thread/machine/datacenter. It will pick up the new robots.txt eventually. This beha

by zorked 15y ago

I don't think it's ignoring robots.txt, it's probably just cached at another thread/machine/datacenter. It will pick up the new robots.txt eventually. This behavior is well-known.

afhof 15y ago
But whats the point of it downloading the robots.txt file if it isn't going to honor it?
- robtoo 15y ago
  Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.