2 ms·
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
by ck2 14y ago
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
- boyter 14y agoFollow up to this (its been a week so probably nobody will see this) http://techzinglive.com/page/1028/179-tz-interview-gabriel-weinberg-blowing-up-duckduckgo http://techzinglive.com/page/1028/179-tz-interview-gabriel-w... around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.
- metalruler 14y agoI checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image. Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) I'm assuming this is the crawler because it does not fetch anything besides text/html.
- Matt_Cutts 14y agoThat's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
- samuellb 14y agoNo, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).