3 ms·

Can't duckduckbot just present itself to the webserver as GoogleBot?

• Aldo_MX 13y ago
  In our company we do a forward-confirmed DNS to verify whenever a bot is who it claims to be. Oddly enough we have blocked legit googlebot/bing/baidu servers, because they fail to properly configure their servers...
  • nly 13y ago
    Their servers are probably configured fine. They likely have a pool of servers with no reverse DNS to try and catch servers issuing different content to Googlebot
    • Aldo_MX 13y ago
      I'm pretty confident that the server 1.2.3.4 which returns crawl-5-6-7-8-googlebot.com it's a badly configured one.