3 ms·
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed B
by ChillyCapy 2mo ago
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites.
Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.
Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.
- herbst 2mo ago[dead]
- dewey 2mo agoFor Google it's pretty straight forward to throw away fake crawlers by just only allowing their published list of crawler IPs so you don't accidentally allow someone from a random GCP IP to crawl you if unwanted (https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests https://developers.google.com/crawling/docs/crawlers-fetcher...).
- inigyou 2mo agoAlso Googlebot is an AI training crawler so if you block AI training crawlers you should just block it.
- Jskewel 2mo agoWith Cloudflare you can set a rule to block traffic that identifies as Googlebot but is not a "verified bot", ie is not from the proper IP range.