4 ms·
What a weird... rant. There are so many questions here: - Is he sure it's Googlebot and not somebody spoofing headers? - Is he sure Google is generating the q
by crazygringo 8y ago
What a weird... rant. There are so many questions here:
- Is he sure it's Googlebot and not somebody spoofing headers?
- Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere?
- Is he sure it's the search crawler, and not some browser service trying to cache autocomplete combinations or results?
- Has he tried simple potential fixes like changing GET to POST for searches, or robots.txt to set a crawl interval to something like 30s?
It just seems so weird. Google crawlers have finite resources so this seems like a misunderstanding or really bizarre edge case or both.
- puzzle 8y agoHe doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into the ground. It's not clear if he was getting hit by GoogleBot or e.g. FeedFetcher, which does NOT obey robots.txt, because it's triggered by human actions: https://support.google.com/webmasters/answer/178852?hl=en https://support.google.com/webmasters/answer/178852?hl=en There's also the possibility, as you say, that traffic was spoofed. Google publishes its netblocks over DNS, so it's easy to double check that.
- mintplant 8y ago> Is he sure it's Googlebot and not somebody spoofing headers? In the author's previous post on the subject [0], he verified that the request were coming from Google IPs. > Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere? From the URLs quoted in the article, it looks more like some sort of (machine learning-y?) system trying to infer his site's canonical URL scheme. [0] https://www.hackerfactor.com/blog/index.php?/archives/484-Google-Abuse.html https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
- emayljames 8y agoIs it possible that it is Google Cloud (console) IP?.