3 ms·
>They need to build their own crawlers like gigablast. Unfortunately the opportunity for that is pretty meek. Many webmasters block crawlers that aren't the to
by hueving 13y ago
>They need to build their own crawlers like gigablast.
Unfortunately the opportunity for that is pretty meek. Many webmasters block crawlers that aren't the top search engines. :-(
- mverwijs 13y agoCan't duckduckbot just present itself to the webserver as GoogleBot?
- Aldo_MX 13y agoIn our company we do a forward-confirmed DNS to verify whenever a bot is who it claims to be. Oddly enough we have blocked legit googlebot/bing/baidu servers, because they fail to properly configure their servers...
- nly 13y agoTheir servers are probably configured fine. They likely have a pool of servers with no reverse DNS to try and catch servers issuing different content to Googlebot
- Aldo_MX 13y agoI'm pretty confident that the server 1.2.3.4 which returns crawl-5-6-7-8-googlebot.com it's a badly configured one.