4 ms·
This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s reall
by judge2020 5y ago
This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it.
https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
- Scoundreller 5y agoHow does Google get around cloaking? Don't they need to visit a site from time to time without coming in as Googlebot to make sure they're getting presented the same page as Googlebot? Or is that always done via manual review?
- judge2020 5y agoThey might pretend to be on other networks they own when they do things like ad/policy reviews (eg. the Google Fi or Google Fiber ASNs[0]), but I don't know of anyone confirming that this happens. 0: https://bgp.he.net/AS16591 https://bgp.he.net/AS16591