4 ms·

I have some skepticism in those regards, at least unless they make use of their own crawlers. Bootstrapping themselves with different information sources as th

I have some skepticism in those regards, at least unless they make use of their own crawlers. Bootstrapping themselves with different information sources as they've been doing is good for the very beginning as then they serve as a privacy-respecting aggregator to those sources, but as the tank man incident proved, you run the risk you'll be made complicit to censorship, whether intentionally or not.

yegg 5y ago
Search has changed a lot in the past decade or so. Now most results pages are full of instant answers, e.g., knowledge graph responses (which we do completely ourselves) and local results (for which we have our own index + Apple Maps and local providers like TripAdvisor). In other words, it isn’t just one index, but about 20 that really matter in terms of adoption. Additionally, people engage with the first thing on the page about twice as much as the second thing, and that second thing about twice as much as third thing, and so on. That means if instant answers are near the top (increasingly the case) it means that those sub-indexes are increasingly more important as they are engaged with relatively more than the traditional “10 blue links." A lot of that traditional web index is also used for just navigational purposes, which we can easily do ourselves. We are already crawling for our tracker blocking data set, encryption data set, and several other things. In other words, we do make use of our own crawlers, and also have the ability to change some things around for indexes we use, which we do routinely. However, you need all of these indexes to make search work well, and have them all come up at the right times (a lot of what we do in search ourselves). In this context, I think people elevate the long-tail “web index” too much. Should we have satellites in space to make good maps? It is also necessary for search. Should we be collecting sports and stocks data ourselves? Also necessary for search, but no one really calls on us to do that. Our approach has been to partner with the best data providers in each vertical so we can provide the best alternative to Google search. And when nothing exists or it isn’t good enough, we do it ourselves or add layers on top. To that end, it isn’t binary — there is a lot in between and in fact that’s the way it works today.
- kemonocode 5y ago
  Thank you for taking the time to explain things at least. I don't personally rely that much on "instant answers"- at least for the things that I'm most likely to have to look up, but I can see why they're so useful. EDIT: Also, back to the point: any plans when it comes to only relying on Bing for images? That was the issue in the end- least how I understood it.
  - yegg 5y ago
    See https://twitter.com/yegg/status/1401216879293874185 https://twitter.com/yegg/status/1401216879293874185
- azalemeth 5y ago
  Also: Thank you very much for making DDG. Thank you for trying to help -- and being the springboard to others making the choice. I used to be a solid "advanced" googler, but made the switch ~3-4 years ago and haven't looked back. I'm glad to see your adoption has grown hugely. Some specific things that would help me a bit (I don't know if they exist!) -- these may well be pipe dreams, but... -- A decent search engine for mathematics in the form of LaTeX would be _amazing_... -- I'd enjoy being able to pass parameters to bangs specifically. An example would be Google Scholar (alas still the best and the last part of their ecosystem I use -- doesn't seem to be as tied in to ads as the rest though) such as 'date from' and 'date to' parameters, with a sensible calling syntax, e.g. `!gsc;f:2015;t:- Convolutional Neural Nets` -- Likewise, a customisable, ideally API-able scientific/technical search interface would be very useful (for me). -- No big provider has tried to index the onionweb (I don't know if you're feeling that brave) -- I'd be interested to know Finally, are you considering being a privacy-first cloud provider in your own right, out of curiosity?
- freediver 5y ago
  Respectfully, vast majority of searches in DDG will still end up with no instant answer and ten (Bing) blue links. Stocks, weather and calculator.. get you only so far (and are not "searches"). Why did DDG shutdown the community effort to extend IA when you say yourself that IAs are DDGs leverage?
analyte123 5y ago
It wasn't widely reported, but DuckDuckGo recently bought Yippy, which was a semi-decent search engine with its own crawler and index. edit: DuckDuckGo did not buy Yippy. Yippy.com seems to be defunct and just redirects to DuckDuckGo.com
- yegg 5y ago
  This is actually a false rumor. We did not buy them, and we actually have had no relationship with Yippy.
  - analyte123 5y ago
    Wow! At first when I saw the Yippy.com redirect to DDG, because it was so "sloppy" I had assumed the truth, which was they just shut it down and redirected. But this was falsely reported in a few places and even Wikipedia has an uncited claim that they were bought by you guys. It's kind of sad that they shut it down. They actually had a decent index and a unique "category" thing on the sidebar that you could use to narrow down search results.

yegg 5y ago

Search has changed a lot in the past decade or so. Now most results pages are full of instant answers, e.g., knowledge graph responses (which we do completely ourselves) and local results (for which we have our own index + Apple Maps and local providers like TripAdvisor). In other words, it isn’t just one index, but about 20 that really matter in terms of adoption. Additionally, people engage with the first thing on the page about twice as much as the second thing, and that second thing about twice as much as third thing, and so on. That means if instant answers are near the top (increasingly the case) it means that those sub-indexes are increasingly more important as they are engaged with relatively more than the traditional “10 blue links." A lot of that traditional web index is also used for just navigational purposes, which we can easily do ourselves. We are already crawling for our tracker blocking data set, encryption data set, and several other things. In other words, we do make use of our own crawlers, and also have the ability to change some things around for indexes we use, which we do routinely. However, you need all of these indexes to make search work well, and have them all come up at the right times (a lot of what we do in search ourselves). In this context, I think people elevate the long-tail “web index” too much. Should we have satellites in space to make good maps? It is also necessary for search. Should we be collecting sports and stocks data ourselves? Also necessary for search, but no one really calls on us to do that. Our approach has been to partner with the best data providers in each vertical so we can provide the best alternative to Google search. And when nothing exists or it isn’t good enough, we do it ourselves or add layers on top. To that end, it isn’t binary — there is a lot in between and in fact that’s the way it works today.

kemonocode 5y ago
Thank you for taking the time to explain things at least. I don't personally rely that much on "instant answers"- at least for the things that I'm most likely to have to look up, but I can see why they're so useful. EDIT: Also, back to the point: any plans when it comes to only relying on Bing for images? That was the issue in the end- least how I understood it.
- yegg 5y ago
  See https://twitter.com/yegg/status/1401216879293874185 https://twitter.com/yegg/status/1401216879293874185
azalemeth 5y ago
Also: Thank you very much for making DDG. Thank you for trying to help -- and being the springboard to others making the choice. I used to be a solid "advanced" googler, but made the switch ~3-4 years ago and haven't looked back. I'm glad to see your adoption has grown hugely. Some specific things that would help me a bit (I don't know if they exist!) -- these may well be pipe dreams, but... -- A decent search engine for mathematics in the form of LaTeX would be _amazing_... -- I'd enjoy being able to pass parameters to bangs specifically. An example would be Google Scholar (alas still the best and the last part of their ecosystem I use -- doesn't seem to be as tied in to ads as the rest though) such as 'date from' and 'date to' parameters, with a sensible calling syntax, e.g. `!gsc;f:2015;t:- Convolutional Neural Nets` -- Likewise, a customisable, ideally API-able scientific/technical search interface would be very useful (for me). -- No big provider has tried to index the onionweb (I don't know if you're feeling that brave) -- I'd be interested to know Finally, are you considering being a privacy-first cloud provider in your own right, out of curiosity?
freediver 5y ago
Respectfully, vast majority of searches in DDG will still end up with no instant answer and ten (Bing) blue links. Stocks, weather and calculator.. get you only so far (and are not "searches"). Why did DDG shutdown the community effort to extend IA when you say yourself that IAs are DDGs leverage?

analyte123 5y ago

It wasn't widely reported, but DuckDuckGo recently bought Yippy, which was a semi-decent search engine with its own crawler and index. edit: DuckDuckGo did not buy Yippy. Yippy.com seems to be defunct and just redirects to DuckDuckGo.com

yegg 5y ago
This is actually a false rumor. We did not buy them, and we actually have had no relationship with Yippy.
- analyte123 5y ago
  Wow! At first when I saw the Yippy.com redirect to DDG, because it was so "sloppy" I had assumed the truth, which was they just shut it down and redirected. But this was falsely reported in a few places and even Wikipedia has an uncited claim that they were bought by you guys. It's kind of sad that they shut it down. They actually had a decent index and a unique "category" thing on the sidebar that you could use to narrow down search results.