4 ms·

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was th

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "Googlebot" in my user agent the frequent errors vanished. So sometimes it's not about using the right IP addresses, but just the right keywords in your user agent. ;-)

beebeepka 5y ago
Changing user agent to Google it or whatever also lets you past quite a few paywalls too. Or at least it used to. These days I don't even bother
judge2020 5y ago
This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it. https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
- Scoundreller 5y ago
  How does Google get around cloaking? Don't they need to visit a site from time to time without coming in as Googlebot to make sure they're getting presented the same page as Googlebot? Or is that always done via manual review?
  - judge2020 5y ago
    They might pretend to be on other networks they own when they do things like ad/policy reviews (eg. the Google Fi or Google Fiber ASNs[0]), but I don't know of anyone confirming that this happens. 0: https://bgp.he.net/AS16591 https://bgp.he.net/AS16591