4 ms·

According to DuckDuckGo's FAQ you are wrong http://help.duckduckgo.com/customer/portal/articles/216399-sources http://help.duckduckgo.com/customer/portal/articl

According to DuckDuckGo's FAQ you are wrong http://help.duckduckgo.com/customer/portal/articles/216399-sources http://help.duckduckgo.com/customer/portal/articles/216399-s... The DuckDuckBot crawls and indexes the web. http://duckduckgo.com/duckduckbot.html http://duckduckgo.com/duckduckbot.html

ck2 14y ago
I suspect it's trivial at best. Every test search I've ever done on DDG shows near identical results to Bing. I'd like to see a search that uses it's own data, examples? Gigablast was the last serious third-party backend that had a chance for independent data. It's like old-school Google. Gabriel should try to buy Gigablast and merge it with DDG so he has his own independent dataset.
- lubujackson 14y ago
  Blekko has millions behind it and does a full index of the web.
  - ck2 14y ago
    Ah now that is an interesting engine. Checking to see if it's hit any of our sites. 3k pages, not bad. Data is kinda stale though.
- boyter 14y ago
  Gigablasts creator has made this now http://procog.com/ http://procog.com/ the results are better then gigablast.
boyter 14y ago
Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain it crawls Quora, as via this tweet https://twitter.com/yegg/status/33693491838066688 https://twitter.com/yegg/status/33693491838066688
- ck2 14y ago
  It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
  - boyter 14y ago
    Follow up to this (its been a week so probably nobody will see this) http://techzinglive.com/page/1028/179-tz-interview-gabriel-weinberg-blowing-up-duckduckgo http://techzinglive.com/page/1028/179-tz-interview-gabriel-w... around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.
  - metalruler 14y ago
    I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image. Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) I'm assuming this is the crawler because it does not fetch anything besides text/html.
    - Matt_Cutts 14y ago
      That's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
  - samuellb 14y ago
    No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).
elssar 14y ago
Try searching for something using their api, and compare with websearch. There is a world of difference between the search results. IIRC they aren't allowed to share results from some of their other sources via their api, so you can get a good idea of how much they take from other sources and how much from their own index. But that is a non-issue as far as I'm concerned. As long as the results are relevant and they got them legally, then who cares where or how they came from?

ck2 14y ago

I suspect it's trivial at best. Every test search I've ever done on DDG shows near identical results to Bing. I'd like to see a search that uses it's own data, examples? Gigablast was the last serious third-party backend that had a chance for independent data. It's like old-school Google. Gabriel should try to buy Gigablast and merge it with DDG so he has his own independent dataset.

lubujackson 14y ago
Blekko has millions behind it and does a full index of the web.
- ck2 14y ago
  Ah now that is an interesting engine. Checking to see if it's hit any of our sites. 3k pages, not bad. Data is kinda stale though.
boyter 14y ago
Gigablasts creator has made this now http://procog.com/ http://procog.com/ the results are better then gigablast.

boyter 14y ago

Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain it crawls Quora, as via this tweet https://twitter.com/yegg/status/33693491838066688 https://twitter.com/yegg/status/33693491838066688

ck2 14y ago
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
- boyter 14y ago
  Follow up to this (its been a week so probably nobody will see this) http://techzinglive.com/page/1028/179-tz-interview-gabriel-weinberg-blowing-up-duckduckgo http://techzinglive.com/page/1028/179-tz-interview-gabriel-w... around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.
- metalruler 14y ago
  I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image. Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) I'm assuming this is the crawler because it does not fetch anything besides text/html.
  - Matt_Cutts 14y ago
    That's interesting. Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
- samuellb 14y ago
  No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).

elssar 14y ago

Try searching for something using their api, and compare with websearch. There is a world of difference between the search results. IIRC they aren't allowed to share results from some of their other sources via their api, so you can get a good idea of how much they take from other sources and how much from their own index. But that is a non-issue as far as I'm concerned. As long as the results are relevant and they got them legally, then who cares where or how they came from?