4 ms·
He doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into
by puzzle 8y ago
He doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into the ground.
It's not clear if he was getting hit by GoogleBot or e.g. FeedFetcher, which does NOT obey robots.txt, because it's triggered by human actions: https://support.google.com/webmasters/answer/178852?hl=en https://support.google.com/webmasters/answer/178852?hl=en
There's also the possibility, as you say, that traffic was spoofed. Google publishes its netblocks over DNS, so it's easy to double check that.