5 ms·
I applaud the DDG spirit but without their own crawling data their future is doomed. If their data sources ever cut them off, it's over. They need to build th
by ck2 13y ago
I applaud the DDG spirit but without their own crawling data their future is doomed.
If their data sources ever cut them off, it's over.
They need to build their own crawlers like gigablast.
- ithkuil 13y agohttps://duckduckgo.com/duckduckbot https://duckduckgo.com/duckduckbot ?
- ck2 13y agoThey aren't collecting much if anything.
- hueving 13y ago>They need to build their own crawlers like gigablast. Unfortunately the opportunity for that is pretty meek. Many webmasters block crawlers that aren't the top search engines. :-(
- mverwijs 13y agoCan't duckduckbot just present itself to the webserver as GoogleBot?
- Aldo_MX 13y agoIn our company we do a forward-confirmed DNS to verify whenever a bot is who it claims to be. Oddly enough we have blocked legit googlebot/bing/baidu servers, because they fail to properly configure their servers...
- nly 13y agoTheir servers are probably configured fine. They likely have a pool of servers with no reverse DNS to try and catch servers issuing different content to Googlebot
- Aldo_MX 13y agoI'm pretty confident that the server 1.2.3.4 which returns crawl-5-6-7-8-googlebot.com it's a badly configured one.