3 ms·

Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain i

by boyter 14y ago

Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain it crawls Quora, as via this tweet https://twitter.com/yegg/status/33693491838066688 https://twitter.com/yegg/status/33693491838066688

ck2 14y ago
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
- boyter 14y ago
  Follow up to this (its been a week so probably nobody will see this) http://techzinglive.com/page/1028/179-tz-interview-gabriel-weinberg-blowing-up-duckduckgo http://techzinglive.com/page/1028/179-tz-interview-gabriel-w... around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.
- metalruler 14y ago
  I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image. Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) I'm assuming this is the crawler because it does not fetch anything besides text/html.
  - Matt_Cutts 14y ago
    That's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
- samuellb 14y ago
  No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).

3 ms·

Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain i

by boyter 14y ago

ck2 14y ago
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
- boyter 14y ago
  Follow up to this (its been a week so probably nobody will see this) http://techzinglive.com/page/1028/179-tz-interview-gabriel-weinberg-blowing-up-duckduckgo http://techzinglive.com/page/1028/179-tz-interview-gabriel-w... around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.
- metalruler 14y ago
  I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image. Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) I'm assuming this is the crawler because it does not fetch anything besides text/html.
  - Matt_Cutts 14y ago
    That's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
- samuellb 14y ago
  No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).