4 ms·

DDG does operate their own crawler[1], though they also do still rely on third parties[2]. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/duckdu

DDG does operate their own crawler[1], though they also do still rely on third parties[2]. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ https://help.duckduckgo.com/duckduckgo-help-pages/results/du... [2] https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so...

mekster 6y ago
Author didn't even DDG to find this out?
- eqv 6y ago
  Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998 https://news.ycombinator.com/item?id=24023998
  - Eeems 6y ago
    Just don't give him IRC ops and then get into a private argument with him. https://www.omnimaga.org/news/omnomirc-moved-to-new-server/msg236376/#msg236376 https://www.omnimaga.org/news/omnomirc-moved-to-new-server/m...
    - djsumdog 6y ago
      Wow. I'm honestly not surprised. That's ... that's pretty shitty.
      - Eeems 6y ago
        Knowing a bit of his personal history I can kind of understand why he acts the way he does, and has the opinions he does. Doesn't excuse some of it, but at least I kinda get why. I just wish his name would stop coming up for me tied to opinion pieces like this. I'd rather just see things about how some project he's working on is doing great and being widely adopted.
    - dang 6y ago
      Quite apart from this type of attack breaking the site guidelines and not being allowed on HN (about which see https://news.ycombinator.com/item?id=25130908 https://news.ycombinator.com/item?id=25130908)... that was 9 years ago. Imagine being publicly shamed for the worst thing you've done in the past decade. I don't think anyone is going to pass that test. Is this the kind of world you (or any of us) really want to be part of? Surely not. Therefore please don't help to create it here.
      - mekster 6y ago
        I think we also have the right to know as I wasn't aware of such past but now I can't see the page.
      - Eeems 6y ago
        If I had done something like that so publicly in the past, and hadn't reached out to the people I hurt to reconcile. I would want people to publicly shame me for it. That way I can attempt to resolve it so everyone can have proper closure. That said, message heard. I will refrain from this in the future here.
  - djsumdog 6y ago
    I don't like to criticize the author. We all have good takes and bad takes and really for a single post, you should address the argument. Digging up the past is part of what's making the world worse. That being said, I do see a valid reason for bringing up his history of bad takes. I use to respect Devault. He banned me on the Fediverse because he disagreed with me being against defunding the police and against critical race theory. I find some of stuff interesting, and I agree with more AGPL and more real open source development. I'd even say I'm jealous that he can actually fund himself off of his FOSS projects and do what he loves. But I do agree, he does have a lot of questionable takes. He seems to love Go and hate Rust, hate threads for some reason, and has a lot of RMS style takes. Not all of them are bad, and hardcore people can help you think. As far as this post goes, I do think search is pretty broken. I think a better solution is more specialized search. Have a web tool just for tech searching that does StackExchange sites, github, blogs, forums, bug trackers and other things specialized to development. Another idea would be an index that just did blogs, do you can look up any topic and see what people are writing about long form for the current month. Add features to easily see what people were saying 5 or 10 years ago too. There is a ton of specialized work there, in filtering blog spam, making sure you get topics from all sides (including "banned" blogs), etc. You use to have to go to Lycos, Yahoo, Hotbot, Excite and you'd get different results and find lots of different helpful things. We need that back. It will take some good, specialized tools, to break people from Google search.
  - dang 6y ago
    No personal attacks on HN, please. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Digging up past internet history as ammunition in an argument isn't cool in general. It's not that such details are necessarily wrong or irrelevant, but doing this has a systemically degrading effect and we don't want to be that sort of community. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=ammunition%20personal%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
- Dahoon 6y ago
  Clearly neither did you as he is correct. DDGs crawler is not a crawler like googlebot.
Kiro 6y ago
Their own crawler is only used to fetch things for the widgets, not the search index.
- andreareina 6y ago
  Ahh, that does seem like a more correct reading of the page. Do we have a source that's unambiguous? Seems strange to have a bot that's able to parse pages for their instant answers but then not use those same results for regular search.
  - anaganisk 6y ago
    They literally user Bing api for search results which is well known
    - andreareina 6y ago
      It's well-known that they use Bing. They also say that they use other sources. In this case I'm looking for an explicit disambiguation of what sources they use for what; my first read led me to interpret it as them also using their crawler to return search links (as opposed to just being used for their instant answers).
      - Kiro 6y ago
        "We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)." This refers to the actual search results and if they used their bot for that I don't see why they would say "multiple partners" instead of "multiple partners and our own crawler". The fact that they don't have their own index is such a common "complaint", and this page is often referred, so if they really used their own bot they should have added that a long time ago. And it's not just a legacy page they have forgot to update. It keeps being updated. In 2019 it said: "We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing." It's interesting to note that back in 2014 the page looked like this: https://web.archive.org/web/20131202065705/https://duck.co/help/results/sources https://web.archive.org/web/20131202065705/https://duck.co/h... Here they talk about their own indexes getting bigger but at the same time admitting that "it seems silly to compete on crawling and, besides, we do not have the money to do so". Completely understandable but also interesting that the current page doesn't mention their own index at all. Maybe they used to have a goal to build their own independent index that has now been dropped? All in all, I think it's safe to presume that their own crawler is only used for Instant Answers etc since that's the part of the sources where it's mentioned. Or at the very least used to such a small extent in the actual search results that it would be disingenuous to even mention it as a source.

mekster 6y ago

Author didn't even DDG to find this out?

eqv 6y ago
Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998 https://news.ycombinator.com/item?id=24023998
- Eeems 6y ago
  Just don't give him IRC ops and then get into a private argument with him. https://www.omnimaga.org/news/omnomirc-moved-to-new-server/msg236376/#msg236376 https://www.omnimaga.org/news/omnomirc-moved-to-new-server/m...
  - djsumdog 6y ago
    Wow. I'm honestly not surprised. That's ... that's pretty shitty.
    - Eeems 6y ago
      Knowing a bit of his personal history I can kind of understand why he acts the way he does, and has the opinions he does. Doesn't excuse some of it, but at least I kinda get why. I just wish his name would stop coming up for me tied to opinion pieces like this. I'd rather just see things about how some project he's working on is doing great and being widely adopted.
  - dang 6y ago
    Quite apart from this type of attack breaking the site guidelines and not being allowed on HN (about which see https://news.ycombinator.com/item?id=25130908 https://news.ycombinator.com/item?id=25130908)... that was 9 years ago. Imagine being publicly shamed for the worst thing you've done in the past decade. I don't think anyone is going to pass that test. Is this the kind of world you (or any of us) really want to be part of? Surely not. Therefore please don't help to create it here.
    - mekster 6y ago
      I think we also have the right to know as I wasn't aware of such past but now I can't see the page.
    - Eeems 6y ago
      If I had done something like that so publicly in the past, and hadn't reached out to the people I hurt to reconcile. I would want people to publicly shame me for it. That way I can attempt to resolve it so everyone can have proper closure. That said, message heard. I will refrain from this in the future here.
- djsumdog 6y ago
  I don't like to criticize the author. We all have good takes and bad takes and really for a single post, you should address the argument. Digging up the past is part of what's making the world worse. That being said, I do see a valid reason for bringing up his history of bad takes. I use to respect Devault. He banned me on the Fediverse because he disagreed with me being against defunding the police and against critical race theory. I find some of stuff interesting, and I agree with more AGPL and more real open source development. I'd even say I'm jealous that he can actually fund himself off of his FOSS projects and do what he loves. But I do agree, he does have a lot of questionable takes. He seems to love Go and hate Rust, hate threads for some reason, and has a lot of RMS style takes. Not all of them are bad, and hardcore people can help you think. As far as this post goes, I do think search is pretty broken. I think a better solution is more specialized search. Have a web tool just for tech searching that does StackExchange sites, github, blogs, forums, bug trackers and other things specialized to development. Another idea would be an index that just did blogs, do you can look up any topic and see what people are writing about long form for the current month. Add features to easily see what people were saying 5 or 10 years ago too. There is a ton of specialized work there, in filtering blog spam, making sure you get topics from all sides (including "banned" blogs), etc. You use to have to go to Lycos, Yahoo, Hotbot, Excite and you'd get different results and find lots of different helpful things. We need that back. It will take some good, specialized tools, to break people from Google search.
- dang 6y ago
  No personal attacks on HN, please. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Digging up past internet history as ammunition in an argument isn't cool in general. It's not that such details are necessarily wrong or irrelevant, but doing this has a systemically degrading effect and we don't want to be that sort of community. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=ammunition%20personal%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
Dahoon 6y ago
Clearly neither did you as he is correct. DDGs crawler is not a crawler like googlebot.

Kiro 6y ago

Their own crawler is only used to fetch things for the widgets, not the search index.

andreareina 6y ago
Ahh, that does seem like a more correct reading of the page. Do we have a source that's unambiguous? Seems strange to have a bot that's able to parse pages for their instant answers but then not use those same results for regular search.
- anaganisk 6y ago
  They literally user Bing api for search results which is well known
  - andreareina 6y ago
    It's well-known that they use Bing. They also say that they use other sources. In this case I'm looking for an explicit disambiguation of what sources they use for what; my first read led me to interpret it as them also using their crawler to return search links (as opposed to just being used for their instant answers).
    - Kiro 6y ago
      "We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)." This refers to the actual search results and if they used their bot for that I don't see why they would say "multiple partners" instead of "multiple partners and our own crawler". The fact that they don't have their own index is such a common "complaint", and this page is often referred, so if they really used their own bot they should have added that a long time ago. And it's not just a legacy page they have forgot to update. It keeps being updated. In 2019 it said: "We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing." It's interesting to note that back in 2014 the page looked like this: https://web.archive.org/web/20131202065705/https://duck.co/help/results/sources https://web.archive.org/web/20131202065705/https://duck.co/h... Here they talk about their own indexes getting bigger but at the same time admitting that "it seems silly to compete on crawling and, besides, we do not have the money to do so". Completely understandable but also interesting that the current page doesn't mention their own index at all. Maybe they used to have a goal to build their own independent index that has now been dropped? All in all, I think it's safe to presume that their own crawler is only used for Instant Answers etc since that's the part of the sources where it's mentioned. Or at the very least used to such a small extent in the actual search results that it would be disingenuous to even mention it as a source.