26 ms·

We can do better than DuckDuckGo

• mlinksva 6y ago
  DuckDuckGo Sourcehut Philadelphia No on point search results from anyone, as far as I can tell. I suppose DuckDuckGo is in a suburb, called Paoli. devault weinberg philadelphia Nothing useful for that either.
• programmarchy 6y ago
  > If SourceHut eventually grows in revenue — at least 5-10× its present revenue — I intend to sponsor this as a public benefit project, with no plans for generating revenue. I am not aware of any monetization approach for a search engine which squares with my ethics and doesn’t fundamentally undermine the mission. So, if no one else has figured it out by the time we have the resources to take it on, we’ll do it. Now _that_ is putting your money where your mouth is! Glad to see a technology leader taking this important issue head-on.
• suff 6y ago
  No you can't.
• suff 6y ago
  I take that back. Aside from loads of money and boundless dev time, you've got it all figured out :-)
• andreareina 6y ago
  DDG does operate their own crawler[1], though they also do still rely on third parties[2]. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ https://help.duckduckgo.com/duckduckgo-help-pages/results/du... [2] https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
  • mekster 6y ago
    Author didn't even DDG to find this out?
    • eqv 6y ago
      Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998 https://news.ycombinator.com/item?id=24023998
      • Eeems 6y ago
        Just don't give him IRC ops and then get into a private argument with him. https://www.omnimaga.org/news/omnomirc-moved-to-new-server/msg236376/#msg236376 https://www.omnimaga.org/news/omnomirc-moved-to-new-server/m...
        • djsumdog 6y ago
          Wow. I'm honestly not surprised. That's ... that's pretty shitty.
          • Eeems 6y ago
            Knowing a bit of his personal history I can kind of understand why he acts the way he does, and has the opinions he does. Doesn't excuse some of it, but at least I kinda get why. I just wish his name would stop coming up for me tied to opinion pieces like this. I'd rather just see things about how some project he's working on is doing great and being widely adopted.
        • dang 6y ago
          Quite apart from this type of attack breaking the site guidelines and not being allowed on HN (about which see https://news.ycombinator.com/item?id=25130908 https://news.ycombinator.com/item?id=25130908)... that was 9 years ago. Imagine being publicly shamed for the worst thing you've done in the past decade. I don't think anyone is going to pass that test. Is this the kind of world you (or any of us) really want to be part of? Surely not. Therefore please don't help to create it here.
  • Kiro 6y ago
    Their own crawler is only used to fetch things for the widgets, not the search index.
    • andreareina 6y ago
      Ahh, that does seem like a more correct reading of the page. Do we have a source that's unambiguous? Seems strange to have a bot that's able to parse pages for their instant answers but then not use those same results for regular search.
      • anaganisk 6y ago
        They literally user Bing api for search results which is well known
        • andreareina 6y ago
          It's well-known that they use Bing. They also say that they use other sources. In this case I'm looking for an explicit disambiguation of what sources they use for what; my first read led me to interpret it as them also using their crawler to return search links (as opposed to just being used for their instant answers).
          • Kiro 6y ago
            "We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)." This refers to the actual search results and if they used their bot for that I don't see why they would say "multiple partners" instead of "multiple partners and our own crawler". The fact that they don't have their own index is such a common "complaint", and this page is often referred, so if they really used their own bot they should have added that a long time ago. And it's not just a legacy page they have forgot to update. It keeps being updated. In 2019 it said: "We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing." It's interesting to note that back in 2014 the page looked like this: https://web.archive.org/web/20131202065705/https://duck.co/help/results/sources https://web.archive.org/web/20131202065705/https://duck.co/h... Here they talk about their own indexes getting bigger but at the same time admitting that "it seems silly to compete on crawling and, besides, we do not have the money to do so". Completely understandable but also interesting that the current page doesn't mention their own index at all. Maybe they used to have a goal to build their own independent index that has now been dropped? All in all, I think it's safe to presume that their own crawler is only used for Instant Answers etc since that's the part of the sources where it's mentioned. Or at the very least used to such a small extent in the actual search results that it would be disingenuous to even mention it as a source.
• unnouinceput 6y ago
  Quote: "They don’t operate a crawler for their search results, and are not independent." They have their own crawler too, beside aggregating results from Bingo and others.
• nikivi 6y ago
  I'd love a truly open source world class search engine. Curious how both the crawler and the search index / search is done by the likes of Google/Bing/DDG. Eventually someone will make an oss version of it that can compete. The beauty of such oss solution maybe the custom heuristics that can be created based off the crawled data.
  • ddorian43 6y ago
    Good luck with that mate. Check out https://commoncrawl.org/ https://commoncrawl.org/
  • Mediterraneo10 6y ago
    The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.
    • creese 6y ago
      How would anyone block a crawler? A crawler is just a headless browser.
      • tleb_ 6y ago
        robots.txt https://www.robotstxt.org/ https://www.robotstxt.org/ https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
        • Xylakant 6y ago
          Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identifying them is hard.
• metroholografix 6y ago
  DuckDuckGo is a mirage and should not be used by privacy-conscious folks. Take a look at its terms of service, information collected section: "We also save searches, but again, not in a personally identifiable way, as we do not store IP addresses or unique User agent strings. We use aggregate, non-personal search data to improve things like misspellings." So they save your web searches and claim that they do so in an non-personally identifiable way. The privacy problems with this claim are many, even if one accepts it at face value (good luck verifying that this is the case).
  • WA9ACE 6y ago
    Do you have a search engine that you prefer to use that claims not to store said information that I might try?
    • metroholografix 6y ago
      The only solution I see is fully distributed/decentralized search. Run your own crawler or be part of a network that distributes this out to each participating node. Every centralized search engine has immensely hard-to-resist and powerful incentives to play "The Eye of Sauron" with your data. Additionally, they offer single points of compromise to other, far more powerful actors. Whatever guarantees DuckDuckGo gives you -and right now they don't give any- don't mean much, if they've been thoroughly (willingly or unwillingly) compromised. Which doesn't mean one should always steer well clear just that one should at least be aware of the tradeoffs one makes when using a centralized search engine. And with DuckDuckGo's misleading marketing, I feel that this point is lost on significant chunks of its userbase.
      • burnthrow 6y ago
        "Run your own crawler" is not a solution. Cool my comments are immediately downvoted like that Italian guy's.
      • ravenstine 6y ago
        Such search engines have been around for many years, and they suck donkey balls. Pardon my French. Install YaCy and tell me how you like it. It wouldn't matter anyway, because decentralization doesn't really solve privacy any better than centralized search, besides the fact that it could theoretically provide more choices. No matter what you use, privacy ultimately depends on trust. The reason that I have more trust for DDG than I do Google is, unlike Google, its primary audience is privacy-minded folks. If it came out that DDG was tracking users and selling that data, DDG would be immediately done as a brand. They at least have some incentive to do what they say. Decentralization provides no such benefit because a search "node" is unlikely to have any sort of meaningful brand to keep up. > And with DuckDuckGo's misleading marketing, I feel that this point is lost on significant chunks of its userbase. How is it misleading? My understanding from their marketing is that they don't create profiles of their users based on searches. Until we have evidence to the contrary, it's not outrageous to assume they are being truthful.
• pmoriarty 6y ago
  My main problem with DDG is that there's no way to be sure they actually respect their users' privacy as they claim to. Ideally, services like theirs would be continuously audited by respectable, trusted organizations like the EFF.. multiple such organizations even. Then I'd have at least some reason to believe their claims of not collecting data about me. As it stands, I only have their word for it.. which in this day and age is pretty worthless. That said, I'd still much rather use DDG, who at least pay lip service to privacy, than sites like Google or Facebook, who are openly contemptuous of it. At the very least it sends a message to these organizations that privacy is still valued, and they'd lose out by not trying to accommodate the privacy needs of their users to some extent.
  • dangus 6y ago
    I don’t even care about the privacy. (Well, I do, but in this context I have no reasonable way to ensure it) What I do care about is trust-building and monopolistic practices. That, to me, is a great reason to use DDG instead of Google or even Bing.
    • cpeterso 6y ago
      I also prefer DDG's user interface over Google's. And DDG's !bang search shortcuts. DDG has been my default search engine for years and its results are good enough for me 95% of the time. I only need to use Google as a fallback when searching for niche technical information or "needles in haystacks".
      • jschwartzi 6y ago
        Even then, the Google results are usually terrible. I haven't used Google as a fallback in about a year because every time I tried it they couldn't find what I was looking for either. Or they did something atrocious like changing my search terms for me.
  • spinach 6y ago
    Facebook and Google are huge, global companies where their main product is free, and yet they aren't a charity. The only way to be mega-rich and offer something free is to be shady and manipulative with user's data. Exploiting privacy is their business model. They aren't gonna respect it. Being super financially successful off free products and services is not a recipe for an honest, citizen respecting company.
• mixologic 6y ago
  How would anybody ever know what the server is running and/or doing with the data you send it, regardless of if it is running open or closed source code? A service, running on somebody else's machine, is essntially closed. I think the only way to have an 'open' service is to have it managed like a co-op, where the users all have access to deployment logs or other such transparency. Even then, it requires implicit trust in whomever has the authorization to access the servers.
  • deleted 6y ago
    [deleted]
  • joosters 6y ago
    In theory, this is the kind of thing that the GPL v3 was trying to address: roughly speaking, if you host & run a service that is derived from GPL-v3'd software, you are obliged to publish your modifications. But, I agree with you - and I don't think the author had really thought through what they were demanding, they made no mention of licensing other than singing happy praises of FOSS as if that would magically mean you could trust what a search engine was doing.
    • lixtra 6y ago
      > In theory, this is the kind of thing that the GPL v3 was trying to address: roughly speaking, if you host & run a service that is derived from GPL-v3'd software, you are obliged to publish your modifications. You mean AGPL https://en.m.wikipedia.org/wiki/Affero_General_Public_License https://en.m.wikipedia.org/wiki/Affero_General_Public_Licens...
      • joosters 6y ago
        You're right... I'm misremembering the GPL, wikipedia says that it was only 'Early drafts of GPLv3 also let licensors add an Affero-like requirement that would have plugged the ASP loophole in the GPL' - I hadn't realised it never made it into the final version.
    • jedimastert 6y ago
• ufo 6y ago
  One thing I always wished for is if there were a way to use duckduckgo bang searches in my browser without sending them through DDG. But apparently it's harder to implement than it sounds.
  • iuqiddis 6y ago
    In Firefox you can right click on a search field and add a keyword bookmark. Once saved, you can type 'kw search query', where kw is your defined key word, in the address bar to directly search the relevant site
    • wldcordeiro 6y ago
      Love this feature. I've got basically all the bang keywords but instead of say `!g query` it just becomes `g <query>`.
    • ufo 6y ago
      I'm aware of that. The problem is that you have to manually add all the keywords yourself. AFAIK, there isn't an easy way to import a large list of curated keywords like the DDG bang list.
      • detaro 6y ago
        They are bookmarks you can export/import from Firefox, so someone could easily make a Firefox bookmark file for a large set of them.
  • takeda 6y ago
    You absolutely can, at least in Firefox you can right click on search field, select "Add a Keyword for this search..." Then save it as bookmark and enter the keyword (you don't have to use !, but it is an option if you chose so). You can also create such bookmark manually and use %s in the url as a placeholder where search query should be placed. The manual configuration can be useful when there's no direct search field. For example freshports.org allows querying freebsd.org. I can add a bookmark with search keyword "fp" to point to https://freshports.org/%S https://freshports.org/%S After that I can type in address bar: fp lang/python39 to land on https://freshports.org/lang/python39 https://freshports.org/lang/python39 (the capital %S doesn't escape special characters like /)
• benmller313 6y ago
  I think this person actually means "We can imagine doing better than DuckDuckGo".
  • timClicks 6y ago
    Well, ideas are much easier than implementations.
    • TedDoesntTalk 6y ago
      Cliqz in Germany was one such implementation, funded in part by Mozilla but completely independent. They wrote their own search engine. They closed shop earlier this year.
    • ikiris 6y ago
      It's kind of amazing how many people think an idea is the biggest part of a viable product.
      • 6510 6y ago
        So you want to build a team and organize finances first? That doesn't seem like a bad idea... wait...
  • 6510 6y ago
    The right question is: How to do search using open source tools? If your goal is "to make something better than the Duck" and you succeed, the Duck dies... what is your goal now?
• josefresco 6y ago
  Just do it.
  • MaxBarraclough 6y ago
    I suspect Drew has his hands full with the SourceHut project.
    • mekster 6y ago
      Perhaps it was better for him to say, "There's a better way to do it than DDG" than "We can do better than DDG" as if he's about to do it when in fact he's waiting for his revenue to go up.
  • eznzt 6y ago
    The last thing we need is a search engine with pictures of anime girls.
• bscphil 6y ago
  > they’ve demonstrated gross incompetence in privacy Not sure I buy the example that is given here. 1. It's an issue in their browser app, not their search service. 2. It's not completely indefensible: it allows fetching favicons (potentially) much faster, since they're cached, and they promise that the favicon service is 100% anonymous anyway. 3. They responded to user feedback and switched to fetching favicons locally, so this is no longer an issue. https://github.com/duckduckgo/Android/issues/527#issuecomment-653210967 https://github.com/duckduckgo/Android/issues/527#issuecommen... > The search results suck! The authoritative sources for anything I want to find are almost always buried beneath 2-5 results from content scrapers and blogspam. This is also true of other search engines like Google. This part is kinda funny because "DuckDuckGo sucks, it's just as bad as Google" is ... not the sort of complaint you normally hear about an alternative search engine, nor does it really connect with any of the normal reasons people consider alternative search engines. That said, I agree with this point. Both DDG and Google seem to be losing the spam war, from what I can tell. And the diagnosis is a good one too: the problem with modern search engines is that they're not opinionated / biased enough! > Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers. This is, obviously, very different from the modern search engine paradigm where domains are treated neutrally at the outset, and then they "learn" weights from how often they get linked and so on. (I'm not sure whether it's possible to make these opinionated decisions in an open source way, but it seems like obviously the right way to go for higher quality results.) Some kind of logic like "For Python programming queries, docs.python.org and then StackExchange are the tier 1 sources" seems to be the kind of hard-coded information that would vastly improve my experience trying to look things up on DuckDuckGo.
  • jbay808 6y ago
    Maybe instead of hard-coding these preferences in the search engine, or having it try to guess for you based on your search history, you can opt-in to download and apply such lists of ranking modifiers to your user profile. Those lists would be maintained by 3rd parties and users, just like eg. adblock blacklists and whitelists. For example, Python devs might maintain a list of search terms and associated urls that get boosted, including stack exchange and their own docs. "Learn python" tutorials would recommend you set up your search preferences for efficient python work, just like they recommend you set up the rest of your workflow. Japanese python devs might have their own list that boosts the official python docs and also whatever the popular local equivalent of stackexchange is in Japan, which gets recommended by the Japanese tutorials. People really into 3D printing can compile their own list for 3D printing hobbyists. You can apply and remove any number of these to your profile at a time.
• lambda_obrien 6y ago
  Why couldn't several coordinating specialized search engines share their data via something like "charge the downloader" S3 buckets? Then you get an org like StackExchange who could provide indexed data from their site and the algorithms to search the data the most efficiently, GitHub can do the same for their specific zone of speciality, Amazon, etc. Then anyone who wants to use the data can either copy it to their own S3 buckets to pay just once, or can use it with some sort of pay-as-you-go method. Anyone who runs a search engine can use the algorithms as a guide for the specific searches they are interested in for their site, or can just make their own. You could trust the other indexers not to give you bad data, because you'd have some sort of legal agreement and technical standards that would ensure that they couldn't/wouldn't "poison the well" somehow with the data they provide. Further, if a bad actor was providing faulty data, the other actors would notice and kick them out of the group or just stop using their data. It would have to be fully open source, I agree with the other parts of Drew's essay here, but I think we could share the index/data somehow if we got together and tried to think about it. We just need a standard for how we share the data.
  • cptskippy 6y ago
    So you're proposing Snowflake for search?
    • lambda_obrien 6y ago
      It appears to be the case in a technical workflow sense, from the little I just read of Snowflake, but my proposal would be a much more open system than one under control of a single vendor. It'd be more like a set of standards for interoperation and a common data center so that the data is accessible under one roof. Maybe each entity could do specialized search as a service and the search aggregators would pay by providing some infrastructure to run the crawlers. I don't personally think any system specification is impossible unless it goes against since mathematical law, so a really fast, distributed query system where there are a few hundred specialized providers for a single query is feasible. Imagine the aggregator does initial analysis to determine the category of search, like programming, news, or restaurant reviews, then sends the users query to a set of specialized providers that supply an index for that category, then fuse the results with some further analysis of the metadata returned. Then the user can also include or exclude the specialized providers at will. You could also eliminate the aggregator as a service as simply make it a user application on the desktop, allowing for even more user control and maybe caching or something.
• Siira 6y ago
  Searx is a partially viable FOSS meta-search engine.
• corytheboyd 6y ago
  You need money and dedicated resources to run and manage the service, which at some point is just going to require trust. Trusting nobody is smart, but expecting a service to compete and win the long game without trusting it is pointless.
• cbsks 6y ago
  > Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers. I like this idea. It would be interesting to see the domains of every search query that I have clicked on and see what the distributions is like. I suspect there would be a long tail but I wonder how many domains actually need to be indexed for 99% of my personal search needs. Does anyone have data like this?
• jedimastert 6y ago
  > Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustworthy. Does it get vetted? by whom? Also, who's definition of trustworthy are we trusting? If I want my blog to show up on your search engine, do I have to get it linked by one of those sites, or can I register with you? Will I be tier 1, or
  • mekster 6y ago
    But the email is already like this. It's the inbox providers who choose what domain is legit and new domains start from negative rating. Treating the web the same way doesn't sound too unnatural. It would be bad if those in the positions profit by "authorizing" who is good though.
    • buzzerbetrayed 6y ago
      I'm not sure why email should be an example of the correct way to do it. And with email I can check my spam folder and see exactly what has been rejected. So unless the search engine has a list of sites that aren't deemed worthy included with every search (which probably wouldn't happen), I think this solution has some pretty big flaws. It should be noted that the current system also has these flaws, as Google and DDG can show you whatever they want base don whatever criteria they see fit.
      • 6510 6y ago
        I like this idea! Have the usual official results... then have an option to go to level 2, level 3, level 4 etc (lvl 1 is not included in lvl 2) You can have really biased technically terrible filters that for example put a site on level 4 because it is to new, to small and any number of other dumb SEO nonsense arguments. (The topic was not in the url! There was poor choice of text color!) I think wikipedia has a lot of research to offer on what to do but also what not to do. Try getting to tier 2 edits on a popular article? It would take days to sort out the edits and construct by hand a tier 2 article.
• jesuscyborg 6y ago
  The way I'd code a better search engine is I'd design an ML model that's trained to recognize handwritten HTML like this, and only add those to the index. It'd be cheap to crawl probably only needing a single computer to run the whole search engine. It'd resurrect The Old Web, that still exists, but just got buried beneath the spammy SEO optimized grifter web over the years as normies flooded the scene.
  • buzzerbetrayed 6y ago
    I hope to never use your search engine. I love hand written HTML as much as the next guy, but search engine's are made to find things. And useful information exists on web sites that use generated and/or minified HTML.
    • jesuscyborg 6y ago
      Thanks for that buzzkill. I guess the lesson is if you can't do everything Google does, don't even try.
      • buzzerbetrayed 6y ago
        > I guess the lesson is if you can't do everything Google does, don't even try I don't agree with that at all. But if your goal is to make "a better search engine" as you said, it does actually have to be "better" and not just different.
• jron 6y ago
  SEO is crushing the utility of Google. It is pretty telling when you need to add things like site:reddit.com to get anything of value. Harnessing real user experiences (blogs, etc) is the key to a better search engine. This model unfortunately crumbles under walled gardens which is increasingly the preferred location of user activity.
  • djsumdog 6y ago
    > Harnessing real user experiences (blogs, etc) This is what we need more than anything. More independent blogs. The ability to search events now, or 10 years ago, mass indexing of RSS feeds, etc. A general search engine is kinda way out of the ballpark for now. But you could specialize for long form blogs, from all sides, hard-left, hard-right, women in tech, white supremacists, all the extremes and moderates. I've love to have an interface to search a topic and see what all kinds of people have posted long form, without commentary or Twitter/Facebook bullshit "Fact checking" notices. I what to see what real writers are seeing across the spectrum on a given topic for the week or month.
    • ant6n 6y ago
      Its hard to get readership writing blogs these days. Thats pretty demotivating.
      • CameronNemo 6y ago
        Also difficult to distinguish a blog from a content farm if you are just crawling the web. Any content pattern you select for would likely be quickly adopted by SEOs.
        • jdm63 6y ago
          You could use machine learning instead of a hard-coded heuristic.
        • spc476 6y ago
          I've found a direct correlation between the chance of a content farm and the number of ads on the blog. With 0 ads, the likelyhook of a content farm is 0%.
• merlinscholz 6y ago
  I miss Cliqz. It was a new search engine, with its own crawler, almost completely from scratch. It even had a dev blog where they wrote articles on how to build your own search engine: https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...
• flas9sd 6y ago
  is somebody aware of a project where the end-user Browser acts as a Crawler? it already spent the energy to render the content. Readability.js extracts page section, does some processing for keywords, hashes anchor links, signs it and sends it off. Cache-Control response headers indicate if the page is public or private. Of course, where it is sending to will have an electricity bill to pay to index the submissions.
  • jedimastert 6y ago
    That's an interesting point...I wouldn't trust the `Cache-Control`, unfortunately, but a distributed indexing model might be interesting... I know there have been talks of set-ups that essentially take a web archive of your entire history to search back through...
  • 1MachineElf 6y ago
    The idealist in me fantasizes this is possible with a browser-based P2P zettelkasten.
  • marc_abonce 6y ago
    Yes. Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having their add-on bundled by default in some Firefox installations[5] which was obviously very controversial because of its privacy and user consent implications. I still think the idea has potential, though, even if it's in a more limited scope. [1] https://github.com/PeARSearch/PeARS-orchard https://github.com/PeARSearch/PeARS-orchard [2] https://cliqz.com/en/whycliqz/human-web https://cliqz.com/en/whycliqz/human-web [3] https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-385000-to-open-source-projects-as-part-of-moss-mission-partners-program/ https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-3... [4] https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-strategic-investment-in-cliqz-to-enable-privacy-focused-search-innovation/ https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-s... [5] https://www.zdnet.com/article/firefox-tests-cliqz-engine-which-slurps-user-browsing-data/ https://www.zdnet.com/article/firefox-tests-cliqz-engine-whi...
• messo 6y ago
  > If SourceHut eventually grows in revenue — at least 5-10× its present revenue — I intend to sponsor this as a public benefit project, with no plans for generating revenue. I like this attitude. Makes me happy to be a paying member of SourceHut.
• rjurney 6y ago
  Check out the serious difficulties the Common Crawl had with crawling 1% of the public internet on donated money and then get back to me with a plan. This is really, really hard to do for free. Maybe talk to Gates :)
  • opan 6y ago
    TFA specifically mentions not crawling the web, but using a curated list.
    • rjurney 6y ago
      I don't follow. A curated list of what? You need to crawl the entire web to have the content of the entire web to index and search.
• neurobashing 6y ago
  Am I the only person who just doesn't have problems with DDG search results? What am I doing wrong (or right), here? I put a thing in and find it. I just don't use Google any more. Genuinely curious why it's working for me and such garbage for everyone else.
  • dybber 6y ago
    I'm mostly getting Norwegian results, when searching for Danish subjects from a Danish IP address. It also seems it just hasn't indexed as many websites as Google.
  • Dahoon 6y ago
    Does Google search results work for you? If yes, then I'd say the reason is you don't see or agree with how bad results are today (as others have posted extensively about). I for one find DDG as the search engine that returns the worst results. Qwant is a better Bing-using engine IMO but it is still bad.
  • keithnz 6y ago
    for generic stuff DDG is mostly ok. But for local results, even though it has a switch for local results, it REALLY REALLY REALLY sucks bad and often doesn't get any of the expected places anywhere in the first few pages for New Zealand which makes it somewhat useless
  • Moru 6y ago
    I'm also using DDG exclusively since many years. I find what I need usually as the first couple of results or in the box on the right, that usually goes directly to the authorative source anyway.
  • djsumdog 6y ago
    I say about 50% I'm good with DDG. About 1/3 of the time I add !g, usually for weird error messages and tech stuff. Honestly we shouldn't be using Google for everything. Why not just search StackExchange or Github issues directly for known bug problems? If you need a movie, !imdb or !rt forward you to exactly where you want to really search on. If DDG or Google also included independent small blogs for movie results, I could see the value in that. I'd prefer someone's review on their own site or video channel, but it doesn't. We've kinda lost that part of the Internet.
• wolco2 6y ago
  Privacy or not I'm starting to find things on ddg that google has been filtering. I found out through comments on hn that 8chan was backup under a new name: 8kun Typing it into google I get articles about it but no link in the results. In duckduckgo first link. Made me think what else am I missing?
  • the_only_law 6y ago
    Googles filtering is so weird. I have a bad habit of buying old hardware without checking if the documentation had been made available on the web. Recently I found myself desperate for any information on a price of hardware i had gotten. I was swapping out all sorts of queries woth different keywords hoping to find a manual. I was able to find some marketing material which was helpful, albeit barely. Eventually I had exhausted the search results for most pf my queries, gave up and assumed that it was simply lost to time and I was out of luck. Eventually I went back to the sales paper I found. Going to the site it was hosted on, a Lithuanian reseller. I translated the page, eventually finding a direct link to a user manual on the exact same page as the sales paper I had found. The document was in English, contained important words from my queries (such as the product name, company, "user manual" etc. The document was at the same path as the sales paper too. I hace no idea why Google found the sales paper but not the manual. Unfortunately the manual still wasn't what I was looking for exactly but it was a hell of a lot better than what I could get from Google's results.
• vladmk 6y ago
  This post looks horrible on the bottom left for mobile fyi
• dumbfounder 6y ago
  Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entirely proprietary engine and you need to CONSTANTLY refine all the above until it isn't garbage. Maybe you could muster something passable for $1B over 5 years with a strong core team that attracts great talent. If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version.
  • kilroy123 6y ago
    I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.
    • ur-whale 6y ago
      >I welcome any competition at this point. Microsoft tried and failed to build a competitor and it's not like they have shallow pockets. They grossly underestimated a number of aspects: - The huge number of man-years invested in Google's search quality stack hand-tuning and what it would take to replicate it. - The fact that the machine learning field was simply not ready to tackle the search quality problem. - The infrastructure required to build a crawler / indexer stack as good as Google's I think in 2020, the second problem is within reach of many companies technically. It's mostly a matter of throwing enough money at optimized infrastructure. However, replicating the search quality stack is going to be very hard, unless someone makes a huge breakthrough in machine learning / language modeling / language understanding at a thousandth of the cost it currently takes to run something like GPT-3. The most likely candidate to execute properly on that last bit is - unfortunately - Google.
      • Krasnol 6y ago
        Sure but Microsoft has tried and failed to build a competitor to google not to DDG. I don't see it that hopeless. I feel it kinda is like starting Open Street Maps. It won't be perfect for a long time but there will be people who'd prefer it and help out.
• TheGrassyKnoll 6y ago
  I actually heard a DDG ad on the radio in the Los Angeles area (KNX 1070 24 hour news). Still love the bangs.
• mcqueenjordan 6y ago
  I've recently been /tinkering/ with exactly such an idea! In my case, it's even more specific and scoped: A search engine with only allow-listed domains about software engineering/tech/product blogs that I trust. https://github.com/jmqd/folklore.dev https://github.com/jmqd/folklore.dev It's not even really at the POC stage yet, but I hope to host it with a simple web frontend sometime soon. Primarily, this is just for myself... I just want a good way to search the sources that I myself trust.
• Analemma_ 6y ago
  I spent seven years working at Bing, and I can tell you that this guy is massively, hugely underestimating the difficulty of this problem. His repeated "it's easy! You just have to..." suggestions are absurd. This is typical HN content where someone with no domain expertise swaggers in and assumes everyone in the space must be idiots, and that only he can save the day. Trust me, there is not a ton of potential "just sitting on the floor" in web search.
  • ablanco 6y ago
    Given your experience, What's your opinion about ddg result quality?
• wcerfgba 6y ago
  I wonder if instead of another search engine we would benefit from a directory, like DMOZ, or perhaps something tag based or non-hierarchical. Sometimes I find better results by first finding a good website in space of my query, and then searching within that site, as opposed to applying a specific query over all websites. Once example would be recipes: if you search for "bean burger recipe" you will get lots of results across many website, but some may not be very good, whereas if you already know of recipe websites that you consider high-quality or match your preferences, then you'll find the best (subjectively) recipe by visiting that site and then searching for bean burgers.
  • SNosTrAnDbLe 6y ago
    Yeah. Exactly my thoughts. I really liked the concept of del.icio.us. where humans could bookmark and tag web sites. dmoz looks pretty great but the categories look limiting.
• blibble 6y ago
  as a recent ddg convert, I've noticed little difference from google (might be because google's results these days are so bad though... can't really tell)
• beefield 6y ago
  How about a crowdsourced search engine like wikipedia or stackoverflow? Like: When you search for "kittens" you get the links that are most upvoted by the community. If nobody has ever submitted links for search term "kittens" , you get a link to selected generic search engines. And "kittens" end up into a list of words someone has searched but nobody has yet added a good result link for.
  • Moru 6y ago
    I hate to be so negative but that's just another sort of SEO problem. Someone will pay a large group of people to sit and click upvotes for their clients nonstop.
    • beefield 6y ago
      Of course there are going to be some highly debated search terms. But I think that applieas also to weikipedia and they have managed to pull it off so that it works reasonably well. I mean, you could always put a big red badge on top of the results that says something in the line of "this search term seems to be troublesome. You may want to check qwant/ddg or maybe even google."
• dgudkov 6y ago
  >We need a real, working FOSS search engine, complete with its own crawler. How would an open-source search engine stand against abusive SEO-optimization? If anyone can understand how the ranking algorithm works then anyone can game it.
  • AsyncAwait 6y ago
    Not as much if you have user curated tier 1 sites. If these start to become spammy, they get removed.
• nynx 6y ago
  I'd love to participate in a project like this. Does anyone know how to contact drew devault?
  • enriquto 6y ago
    Dude has an email on his website.
• znpy 6y ago
  the tiering system is dumb, really really dumb. it would basically make already famous domains shine and dump lesser known domains into 20th page oblivion. it saddens me because google search results actually helped you discover new sites and new people, but it's been years since that has changed.
• RileyJames 6y ago
  I’m pro privacy, but I dont have a problem with AdWords, outside of googles implementation. If AdWords targeting was purely based on the search term, I don’t mind. The search engine has to generate revenue somehow, and the revenue generated on “Saas crm” with a single click is likely to be larger than any users annual subscription. (10 - 100+ per click) I’m unclear on the ethical / privacy concerns of “AdWords” style advertising.
  • ricardo81 6y ago
    FWIW it's no longer called Adwords, just Google Ads Agree, keyword (and location, a lot of searches are for X near me) for the most part offers a way of delivering relevant ads. Google are able to generate more income per search because of their critical mass of searches and advertisers, as well as having more data on searchers based on search history to maximise that revenue per search.
  • blendergeek 6y ago
    Here is the problem (and it isn't privacy): A search engine's job is to present you with the best possible results for any given query. A ad is either A) the best possible result or B) not the best possible result. If the ad is the best possible result, then the search engine must display it anyway in order to fulfill its mission. If it is not the best possible result, the search engine must violate its mission in order to display it. To put it bluntly, advertising is paying to decrease the quality of search results.
• wenbin 6y ago
  It's almost impossible to build a decent web search engine from scratch today (i.e., build your own index, fight SEO spams, tweak search result relevance...). The web is already so big and so complex. Otherwise Google won't need to hire so many people to work on search alone. If you didn't start at the very early stage of tiny web (e.g., Google in 1996 as a research project) and grew with the web over the past 20+ years, or you don't have super deep pocket (e.g., Microsoft Bing in mid 2000s), then it's almost impossible to build a decent web search engine within a few years. It's possible to build vertical search engines on far smaller scale, far less complex, far less lucrative things that Google/Microsoft has little interest today (e.g., recipes [2], podcasts [3], gifs [4]...) It's also possible to come up with a different discovery mechanism for web (or a small portion of web), other than a traditional complete web search engine. Essentially you don't cross moat to attack a huge castle (e.g., Google). Instead, you bypass the castle [1], as it becomes irrelevant. [1] https://twitter.com/benedictevans/status/1038538688232226817?lang=en https://twitter.com/benedictevans/status/1038538688232226817... [2] https://www.yummly.com/ https://www.yummly.com/ [3] https://www.listennotes.com/ https://www.listennotes.com/ [4] https://giphy.com/ https://giphy.com/
  • mongol 6y ago
    You are probably right. But still... the approach suggested makes kind of sense. A curated list of trusted sites as kind of seed. Not the entire web. This can be as small or as large as can be useful. It does not need to be about the entire web. How big is the "useful" blogosphere, for example? Cannot an opensource project that gathers momentum somehow create a curated list of let's say 10 000 trusted blogs and index those? Index all mailing lists that can be found, index all of reddit, index Hacker News, index Wikipedia, the 100 most well regarded news sites in each country, etc. Would not such an index be a good start and better than Google in many cases?
• meerita 6y ago
  I'm an old-time user of DDG. I agree with the 3 points. I feel like I'm using a shitty car in a world where everyone runs a ferrari. The one that strikes me most is the results. I feel like DDG doesn't search the entire internet, like there's zillions of pages there waiting to be indexed, even old websites, but the results i get are so poor. Even with this handicap, I still use over INSERT YOUR AD HERE Google Search.
• deleted 6y ago
  [deleted]
• x87678r 6y ago
  I dont want privacy, I want competition. Which is why I use Bing and honestly it works virtually all the time. Maybe there could be some pure anonymous ad-free search engine but it more realistic to have alternative commercial one. I really dont care that people are looking at my searches for how to resize an array or cheap hotels in Florida.
  • ablanco 6y ago
    This "nothing to hide" argument has been pretty analized, and in my opinion, it's really dangerous. https://spreadprivacy.com/three-reasons-why-the-nothing-to-hide-argument-is-flawed/ https://spreadprivacy.com/three-reasons-why-the-nothing-to-h...
• staunch 6y ago
  We have to make sure to include highly relevant advertisements in the search results, at least 50% of the results should be ads. So there needs to be a marketplace for buying/selling ads. We can't have a search engine that is only useful for finding the most relevant web pages for a given query. People love highly relevant advertisement in their search results.
• abalaji 6y ago
  This is why search is hard: 15% of Google searches are new each day. [1] And, with over 1.7+ billion web pages, [2] it would take a gargantuan open source effort to put something together like this. Not to mention the cost, not sure something like this could be sustained with a Wikipedia-esque "please donate $15" fundraising model. [1] https://searchengineland.com/google-reaffirms-15-searches-new-never-searched-273786 https://searchengineland.com/google-reaffirms-15-searches-ne... [2] https://www.weforum.org/agenda/2019/09/chart-of-the-day-how-many-websites-are-there https://www.weforum.org/agenda/2019/09/chart-of-the-day-how-...
  • effie 6y ago
    Why is the ratio of new queries relevant at all? The open alternative to Google doesn't need to have the same capacity for download and indexing.
• easymovet 6y ago
  Crawlers is top down approach, a distributed list that people pay digital money for listing will both incentivize nodes to be online and transforms sybil attacks into paid advertising.
• 6510 6y ago
  While I agree with its lack of organization I don't think YaCy being untolerably slow is necessarily an argument. If you are looking for a complete set of pages on a specific topic time is sort-of irrelevant. Google for example has alerts for new results. That these pages are not available sooner (before publication) is not intolerable. You can also throw hardware at YaCy and adjust the settings which improves it a lot. The challenge with a distributed approach is sorting the results. Other crawlers have the same problem but in a distributed system it is even harder. Running an instance for websites related to your occupation or hobby YaCy is quite wonderful. You don't want google removing a bunch of pages that might cover exactly the sub-topic you are looking for. Of course the smaller the number of pages in your niche the better it works.
• deleted 6y ago
  [deleted]
• katsura 6y ago
  My biggest pet peeve with DDG at the moment is that whenever I search for something on my phone the first two results are ads, and those two results actually take up my whole screen. I mean sure, those are probably not privacy invading, but I literally don't care as I wasn't looking for them.
• baggachipz 6y ago
  > The main problem is: who’s going to pay for it? Advertisements or paid results are not going to fly — conflict of interest. Private, paid access to search APIs or index internals is one opportunity, but it’s kind of shit and I think that preferring open data access and open APIs would be exceptionally valuable for the community. There's no reason you couldn't allow the first N number of api hits to be free, then charge for higher tiers of access.
• moonchild 6y ago
  Findx[1] was an attempt to make an opensource search engine. Today it's just another bing wrapper; but their code[2] is still available, waiting to be used as a starting point for another project. 1. https://www.findx.com/ https://www.findx.com/ 2. https://github.com/privacore/open-source-search-engine https://github.com/privacore/open-source-search-engine
• ecommerceguy 6y ago
  I'm surprised noone has rented Ahrefs database, whipped up an algorithm and called it a search engine. Besides google and microsoft, who has a bigger snapshot of the entire web (NSA not included)? Majestic maybe?
• claytoneast 6y ago
  I wonder if you could start small on something like this. Build a proof of concept, a search engine for programmers that indexes only programming sites/material. See if you can technically do it, & if you can figure out governance mechanisms for the project. Sort of like Amazon starting with just selling books.
• deleted 6y ago
  [deleted]
• keyle 6y ago
  I agree DDG isn't perfect or great but it's _good_ 80% of the time. I always start with DDG and revert back to Google if it doesn't help, or I feel "there's got to be a better way". That said, talk is cheap, show us your engine.
• Guest19023892 6y ago
  What about a curated search engine? Allow anyone to curate the results and define a custom list of allowed URLs. Then, others can use their list. For example, I decide Google is terrible when I'm searching for product reviews, and all I get are results to Amazon referral websites and spam blogs that never owned the products to begin with. So, I find 200 sites or forums that actually have quality reviews and I create a whitelist of those URLs, and I name it "John Doe's product reviews list". Other people visit the search engine and they can see my list, favorite it, and apply it to their results. I maintain the list, so they continue to get updates as it's refined. The idea is you visit the search engine, type your query, then select from a drop down one of your favorite curated lists to apply. Maybe you like to use "Mike's favorite free stock photo websites" when searching for free photos for your projects. Maybe you like to apply "Jane's vegan friendly results" when searching recipes or face creams. Maybe you want to buy local, so you use the "Handmade in X" list when searching for your next belt. Maybe you use a list that only shows results from forums, or another for tracking/ad free websites. Keep track of list changes. So, if someone gets paid off to allow certain sites on their popular list, others can easily fork a past version of the list.
  • fourtens 6y ago
    I really like your idea and had similar thoughts - curious how you would build this, and if you planned to? I saw in your comments you ran a website that sounded like an earlier version of Reddit - could you share the name of the site? My email is orangegummy@gmail.com if you aren't interested in publicly posting that info ( I would have emailed you but there's no contact info I could see)
• api 6y ago
  A major challenge with search in 2020 is that it's adversarial. Any open source search engine that gets popular is going to be analyzed by black hat SEO people and explicitly targeted by spam networks. Competently indexing and searching content is really only a small part of the problem now, with the adversarial "red queen's race" against black hat SEO and spam being the more significant issue.
• todd3834 6y ago
  Aren't the secrets of the algorithm what prevent people from gaming the results? While I love the idea of search becoming fully open source I'm skeptical it could be done. I hope I'm wrong and I'd love to dedicate time to an open source project with this goal if anyone presents a convincing plan.
  • effie 6y ago
    The algorithms, code and configuration can be public, if the ranking isn't done just by those, but instead by many participants in the project all over the world and also by personal preferences of each client. That would be hard to game.
• qwerty456127 6y ago
  The three problems mentioned are not the only existing. Most of the people are unaware of censorship DuckDuckGo also does, as do Google and other major search engines. For example: https://news.ycombinator.com/item?id=25093009 https://news.ycombinator.com/item?id=25093009
• AntiImperialist 6y ago
  Well, yes, we can. Just use Google with non-Chrome browser with built-in ad-blocking, like Edge or Brave.
• simonebrunozzi 6y ago
  I tried to build something like this in 2007, together with a small band of nerds and geeks and Linux enthusiasts. It was called Beeseek. [0] I knew close to nothing about building a company or a project, or how a proper business model would have helped it. I was the leader (SABDFL) of the group, and unfortunately I didn't lead it well enough to succeed. We had some good ideas, but ultimately we failed at building more than the initial prototype. The idea behind it was simple: WorkerBee nodes (users' computers) would crawl the web, and provide the computational power to run Beeseek. Users could upvote pages (using "trackers" that anonymously "spy" the user in order to find new pages - repeat: anonymously). The entire DB would be hosted across multiple nodes. Auth and other functionalities would be provided by "higher level" nodes (QueenBee nodes). Everything was going to be open source. Well, it didn't work. Thankfully, because of Beeseek, I met a few very smart people that I am in touch with to this day. Life is strange and beautiful in its own way. Weird, though, that today I still believe that Beeseek could have been the right thing to build. Who knows? [0]: https://launchpad.net/beeseek https://launchpad.net/beeseek
  • jrochkind1 6y ago
    In what ways does what OP describes remind you of your project? Just that it was an open source web search? One difference from what you describe is that the OP is specifically recommending against decentralization/federation, where it seems to have been the core differentiator of your effort. I don't think what OP is describing is quite what you are describing.
    • PaulDavisThe1st 6y ago
      I think that DDV was arguing against decentralization/federation for searching the index. Not necessarily related in any way to building the index (if the distributed nodes all just forward back results to central hub).
    • simonebrunozzi 6y ago
      As in, to build a different kind of search engine.
• jimnotgym 6y ago
  > The search results suck Do they really though, for normal people that is?. Some of my searches today below, can't remember the exact terms I used. Mix of DDG and Google. 1) Walt Whitman, I wanted a basic overview of his work to satisfy some idle curiosity. DDG gave me his wikipedia page. Bingo 2) EAN-13 check digit. First result wikipedia telling me how to calculate it. I see it is simple and I have a long list in Excel to check. I can't be bothered to think so... 3) EAN-13 Excel. First result has an example that I copied and pasted. 4) Timezone [niche cloud system]. Said system didn't do what we expected, seems to be timezone issue. First article is discussing this niche issue and offers solutions 5) Does Shopify support x payments. Yes it does 6) Coronavirus test. Got straight to government site. 7) MacOS version numbers. First hit... 8) How come my Microsoft x platform is showing as being at y level of service when my Buddies is not. Straight in Am I just a perfect search customer? I don't seem to be getting the problems Drew is?
  • nl 6y ago
    In consumer search there is a really long tail of questions (in 2017 15% of Google's daily queries have never been seen before[1]) and performance on this is very important. I just searched for "lockdown rules for SA" (I'm in South Australia and we just had a new 20 person cluster, so we are going back into lockdown). On DDG the first results was a Guardian article which was good, but then the rest were a mix of South African articles and blog spam. There were no SA Gov pages on the first page of results. On Google the first result was the South Australian gov site with the rules, the second was the Guardian article, then more SA Gov pages and at result 8 I got a South African result. https://searchengineland.com/google-reaffirms-15-searches-new-never-searched-273786 https://searchengineland.com/google-reaffirms-15-searches-ne...
    • ddingus 6y ago
      I got this: https://www.enca.com/news/sa-lockdown-strick-international-travel-rules-announced https://www.enca.com/news/sa-lockdown-strick-international-t...
      • 6y ago
• m-i-l 6y ago
  Its still pretty new and I'm working on it in my spare time, but my side-project https://searchmysite.net/ https://searchmysite.net/ seems pretty close to what the author is after: - "100% of the software would be free software, and third parties would be encouraged to set up their own installations" - I'm planning on open sourcing it under AGPL soon, once I've got documentation, testing etc. ready. Plus it's easy to set up your own installation (git clone; mkdirs for data; docker-compose up -d). - "I would not have it crawling the entire web from the outset" - That's one of the key features of my approach, only crawling submitted domains. I'm focussing on personal websites and independent websites at the moment, primarily because I don't currently have the money for infra to crawl big but useful sites like wikipedia, but there's nothing to stop people setting up their own instances for other types of site. - "who’s going to pay for it? Advertisements or paid results are not going to fly" - A tough anti-advert stance is another key differentiating feature to try to keep out spam, e.g. I detect adverts on indexed pages and make sure those pages are heavily downranked. Planning to pay running costs via a listing fee, which gives access to additional features like greater control over indexing (e.g. being able to trigger reindexing on demand).
• betaby 6y ago
  I'm carious what happened with the Cliqz https://en.wikipedia.org/wiki/Cliqz https://en.wikipedia.org/wiki/Cliqz search engine source code. Quality of their search was OK.
• olladecarne 6y ago
  Low quality information is more profitable to produce than high quality information (thanks to Google and their ads). So all the incentives for people currently are to produce content that is nearly indistinguishable from spam. It is much more profitable to take 1 hour to write generic content than to take weeks to really think through everything. This problem will continue to get worse with AI generated text, and I don't see how Google can fight that. This is why 90% of my time is now spent on Youtube instead, which has the advantage that focusing on quality has a much higher ROI with video than with text. That doesn't mean it's not filled with garbage though, just that it's easier to swift through it. Product search is also something I now do on Amazon because Google results are essentially spam. The results there are also not ideal, but at least I can distinguish the bad results faster. One way to make search better would be to embrace bias and give people what they want. Just accept that most information is biased and bring it to the forefront. Initially there would be some default domain whitelisting, but users can request sites to add or remove from their bubble. Maybe they can "share" these lists among each other and certain clusters would form, which you can then use to recommend more nodes. Users would also be clearly told that results are biased based on their preferences, and that they can choose to view results from other points of reference. It would also have different "modes" for things like products, information, and news. Maybe some users only want to see results from independent or ad-free sources, and they can choose those clusters. Maybe they want only far-right or far-left sources, and then they can choose those. At least people would be aware of their bias, which I think will help fight it more than acting like it doesn't exist. Essentially there would be an explorable graph so I can see different realities. It would be a combination of search and social media.
• mikalv 6y ago
  Yahoo did open source theirs, https://vespa.ai/ https://vespa.ai/
• motge 6y ago
  A few months ago Infinity Search was featured here on HN. https://infinitysearch.co https://infinitysearch.co
• MayeulC 6y ago
  Hmm, I have been thinking about this more lately. How to get quality results, and a sustainable, community-led search engine? === Contexts === A "search engine" such as Google is good at many things, and extremely bad at others. The main issue with it in my view is that it lacks context about what you are looking for. The main context you can ask for is "Videos", "Pictures", etc. * Specifying context takes time, so it's OK for long searchs Google sucks at (find this specific article I read a while back). Take your time while you specify language, exact/fuzzy match, publication date, background color, author name or any number of things you know about your search. * Some requests can be processed with instant answers, that's good news as it fits the open source model quite well. * Lastly, the other requests. Some are asked like a question and might require NLP to sort trough. Quite hard IMO, it might get better but will still require compute power if done server-side. It's mostly: parse the question to find the context, and perform a contextual keyword search/instant answer. * And those that aren't questions: "regular", keyword-based requests, that "just" require a big index and a big infrastructure to search it. === Hardware === Now, we are left with the cost centers: hardware. IMO, the only way to scale is to rely on the community and distribute things. * Databases: if this is a community project, and not too latency-sensitive, the community can help by distributing them over a p2p network, even with a single source of trust. * Queries, walking the database: delegating processing to untrusted third-parties is a bit more dangerous. Maybe allow each user to specify a list of trusted servers? Can be centralized and clients ask the network, though it might leak part of their search, depending on the index method. Could be client-side? * Processing the answers: client-side, or trough any number of frontends (like searx). * Crawlers: crawling the net isn't cheap. You could use one or multiple sources of trust. Domain-specific crawlers, like hinted at in Drew's post. Maybe crawl on demand or trough the user's computer (web extension that indexes as the user browses, and allows them to full-text search their history; share it or not). === Content === For some measure of quality, I find that websites that do not have advertisements offer better-quality content. That's likely due to conflicting interests. It would be great if the semantic web mandated disclosing revenue sources. You could downrank or avoid crawling sites with ads and/or Google Analytics, for instance. This could be abused to an extent if the service ever becomes popular, but heh https://xkcd.com/810/ https://xkcd.com/810/ Domain-specific crawlers would be nice as well. === Added value === To be adopted, the service needs to be better than the original in some ways. I think that a new search engine should not try to conquer the masses at first. Instead, find some people that are not satisfied with the current offering and court them. Currently, I think this isn't met by advanced search: exclude websites protected by recaptcha, only include websites that are less than X years old, no ads, etc. Allow users to create their own contexts and easily switch them: bangs, tabs, date/time/geoip, etc. Have them create contexts dedicated to their activities: programming is an obvious one, but so is cooking, gardening, encyclopedic search, language usage/dictionaries, etc. === Monetization === At that point, I am not sure it can ever turn a profit? EU grants? Consulting? Help webmaster set up search on their own website? Sell desktop indexing software? Well, I do have some ideas around content curation, but I am a tad reticent to share them here, and not sure they are more useful than the above.
• O_H_E 6y ago
  For me, one of the weakest parts of ddg/google is finding niche content. Getting any results to a non-mainstream blog from anything but a direct quote is extremely hard. I always have to type HN/reddit to get authentic recommendations or opinions from people who have actual experience with the subject matter. Otherwise 90% of the results are from SEO-optmisezd sites that barely introduces the subject matter.
• switch11 6y ago
  I agree 100% with the fact that we can do better than DuckDuckGo A) DDG is already better than Google for some search queries B) I use Epic Search and that is good, though it just uses Yahoo or Bing results (not sure which) C) We should have a lot of search engines It should not be Google paying $8 billion a year to Apple and $X a year to Firefox/Mozilla and leveraging Google Search as Default in Chrome and monopolizing search
• lukestateson 6y ago
  Drew in his blog post talking about DuckDuckGo privacy issues, but his commercial startup Sourcehut does not offer the Privacy basics: 1. Account deletion 2. GDPR data request 3. Option to unsubscribe from emails So right now his blog reminds me one famous US politician Twitter account. Never fix your own problems, just blame others more often.
  • ddevault 6y ago
    I normally would not reply to someone who equates my blog posts with the ravings of a megalomaniacal fachist, but I will at least clarify for the benefit of onlookers that all three of these points are false. I handle account deletion and GDPR requests all the time, and every email you get from sr.ht (1) is not a marketing email and (2) can be trivially unsubsribed from, with the exception of payment notifications - which is not only allowed per the GPDR, but a lot better than silently charging you a recurring payment forever.
    • deleted 6y ago
      [deleted]
    • henry_bone 6y ago
      Trump is not a fascist. He's not far-right. Mere nationalist tendencies do not make one a fascist. From wikipedia:- "Fascism (/ˈfæʃɪzəm/) is a form of far-right, authoritarian ultranationalism characterized by dictatorial power, forcible suppression of opposition and strong regimentation of society and of the economy which came to prominence in early 20th-century Europe. The first fascist movements emerged in Italy during World War I, before spreading to other European countries. Opposed to liberalism, Marxism, and anarchism, fascism is placed on the far right within the traditional left–right spectrum."
      • ddevault 6y ago
        Refusing to concede a fair election, using gerrymandering and electoral fraud to keep a conservative minority party in power, obstructing the political process, packing the courts as a political weapon, not to mention open racism, a cult of personality, fervent nationalism, and tacit endorsement of political violence? If you compare the rise of European facism to contemporary trends in US politics then the parallels are pretty stark. If it walks like a duck, and quacks like a duck... it is, at the least, a precursor to fascism. It doesn't happen all at once, and if we wait until it's plainly obvious to call it like it is, then it'll probably be too late. And even if you don't want to call it facist, it's out of line to compare this shit to my personal blog. That's a baseless character attack, which is a dick move, not to mention against the HN guidelines.
• 0xbadcafebee 6y ago
  Thinking you can design a better search engine by yourself is either egotism or ignorance. Assuming you based it on state of the art search engine research, and could somehow avoid patent encumbrance, it'd still take you 5 years to match Google's results (and even then not likely) sans all the SEO bullshit. Most people still believe that it's possible for one search engine to help anyone find anything without it knowing anything about them, which is just ridiculous. To get good search results you practically have to read someone's mind. Google basically does this (along with their e-mails, and voicemails, and texts, and web searches, and AMP links, and PageRanked crawls, and context-aware filters) and they still don't always get it right. There is no magic algorithm that replaces statistical analysis of a large corpus along with a massive database of customized rulesets. > We should also prepare the software to boldly lead the way on new internet standards. Crawling and indexing non-HTTP data sources (Gemini? Man pages? Linux distribution repositories?), supporting non-traditional network stacks (Tor? Yggdrasil? cjdns?) and third-party name systems (OpenNIC?), and anything else we could leverage our influence to give a leg up on. Oh, great, so become the Devil himself, then. Count me out.
• sfletcher 6y ago
  I don't fully understand something about the general tech industry discourse around search and would love to hear if I'm wrong. Here's my brief and slightly made up history of search engines: In the beginning of time, search engines took a Boolean query (duck AND pond) and found all the documents which contained both words using an inverted index and then returned them in something like descending date order. But for queries which had big result sets, this order wasn't very useful and so search engines began letting users enter more "natural language" queries (duck pond) and sorting documents based on the number of terms that overlap with the query. They came up with a bunch of relevance formulas - tfidf, BM25 - that tried to model the query overlap. But it turns out this is tricky because user intent is a really tricky problem and so modern day search engines just declare that relevance is whatever users click on. Specifically they just model the probability that you're going to click on a link (or something) using a DNN that uses things like the individual term overlap, the number of users that have clicked on this link, the probability it's spam, the PageRank etc. Some search engines like Google also include personalized features like the number of times you have clicked on this particular domain - because for instance as a programmer your query of (Java) might have different intent than your grandmother's. This score then gets used to sort the results into a ranked list. This is why search engines (DDG included) collect all this data - because it makes the relevance problem tractable at web scale. Maybe just my perspective but I just really don't understand why OP would want to build an index - it's hard boring expensive and doesn't violate data privacy - and I don't think people grasp that - at least to some extent - data privacy and relevance are in direct conflict?
• lsiebert 6y ago
  I think the only way to get an open search engine going is to specifically target a niche first, build profit from ad results, and then expand. My suggestion? Older folks don't have a well known search engine targeted at them, and there are features that could make a search engine helpful for them (High contrast mode or built in screen reading for the vision impaired, anti fraud results for common scams that target seniors based on search results, links to places to watch old shows when people search for character names), and they are a lucrative demographic, both in terms of having money to buy things but also for political ads. I'm sure a focus group of seniors would have more detailed thoughts.
• yyyk 6y ago
  The whitelist approach reminds me of Yahoo's internet directory or DMOZ. The internet directories all ended up closing. That approach will not scale in the general case, not even with the search engine following links from said sites. Too many areas of interest, too many languages, nowhere near enough people to categorize 'high-quality' sites all the time. It could work for domain-specific searches for expert communities. SourceHut should start with a good code search engine...
• sidcool 6y ago
  I am all for open source and free software, but would a truly open source and free search engine possible that is as good as Google?
• stevage 6y ago
  >We can do better than DuckDuckGo Proceeds to give many reasons why it would be very difficult to do better.
• vibrant_mclean 6y ago
  > Crawling and indexing non-HTTP data sources (Gemini? Man pages? Linux distribution repositories?) > ⇒ This article is also available on gemini. Does anyone know what is gemini? I couldn't find relevant results in any of the search engines.
  • flobosg 6y ago
    https://drewdevault.com/gemini.html https://drewdevault.com/gemini.html
• Const-me 6y ago
  > Here’s how I would design it. The harder question is "who and why is going to pay for development?" Open source means you can't sell ads, any sane person would search for a fork without ads.
• smonff 6y ago
  DDG is a search engine that respect your privacy. DDG doesn't put you in a « bubble » based on your location, search history, your browsing history, physical moves, time of day, country of residence, IP, and billions parameters I don't know about. Therefore, there are things that cannot be done, like Google or Bing do. Google is like an oracle because it exploit all of this. And it works, with a cost for your privacy (but maybe you are ok to pay it). DDG is more a "meta-search-engine" with limited capabilities. Bit you got the flexibility of accessing the search engines of thousand of websites. If you don't care for your privacy, don't use DDG and stay with Google.
• diebeforei485 6y ago
  I think DDG is mostly fine. It does sometimes give results for the wrong location or older results, so if you want specific new information for your location you should use Google (easy enough, just use !g in the query and it sends you to Google).
• 8bitsrule 6y ago
  For all it's charms, DDG seems to be tailored to answering the kinds of questions that are most likely to be asked. IME, it's not well-suited to detailed dives into a narrow (likely unpopular) topic. If the first page doesn't satisfy, just prefix the search terms with '!b ' and Bing usually nails it.
• ElijahLynn 6y ago
  > DuckDuckGo is not a search engine. It’s more aptly described as a search engine frontend. They do handle features like bangs and instant answers internally, but their actual search results come from third-parties like Bing. They don’t operate a crawler for their search results, and are not independent. I didn't know this. I tried to use DDG for a while after I switched to Brave, but the results were just not very good and completely missing results I was looking for at times. This would explain it if it was coming from Bing.
• franczesko 6y ago
  DDG will be as good as Bing is, as they're using their API to fetch and render results.
• swyx 6y ago
  OT but I love the simple, easy to read design of this site!
• surajs 6y ago
  I can sympathize with the author. DDG is a terrible choice for a search engine. They have managed to somehow underplay the importance of their cause, which is why nobody uses DDG.
• juncode 6y ago
  Maybe we just need a paid search engine, like there are paid email providers - as an alternative to paying with privacy.
• m0zg 6y ago
  This is severely hampered by the availability of Google. After a few false starts I ended up switching to DDG as my default search engine. My usage pattern is as follows: for things that are predictably easy to find (e.g. the queries "normal" people would issue), I use DDG. That's about 90-95% of all my queries. For obscure stuff, I add "!g" at the end of the query and go to Google. That's the remaining 5-10%, and I doubt anyone can do better than Google there. That way I get both the privacy and the long tail.
• XeroSpaceOmega 6y ago
  One of the biggest problems the article points out more than anything is "Who's Going To Pay For It?" You have one of two options. The crowd-funded approach would have to come with an understanding that you're trying to build a better, more private search engine that will leave the payer paying for all those who don't and the payer won't be able to have a say in anything as we'd like search results to be flat and even across the board. That means if I search for Coronavirus results, not only do I get the goverment and "Official" sources, I should get everything that I'm looking for and refine as needed. The second approach is obviously big money but if you have big money coming in, big money will give You one of two options; Do as they say or they withdraw funding leaving you back at option one and having to downsize. Rocks and hard places people. Not much else you can do there. Unless you take a Pilled.net approach.
• rkagerer 6y ago
  Great. Write a better one then come back here.
• tinus_hn 6y ago
  Great, so when do you start?
• darepublic 6y ago
  Am I wrong or there are simply not enough parties out there interested in revolutionizing search? Google is good enough despite the misgivings around privacy? All we get are skins of existing search engines, or proxies.
• samhh 6y ago
  I think it makes sense to separate the notion of bangs, what a lot of people use DDG for, from that of a search engine. I think projects like my own (very primitive!) bangin are a step in the right direction here: https://github.com/samhh/bangin https://github.com/samhh/bangin
• benlivengood 6y ago
  Making an open source search engine is asking for it to be spammed. Every spammer will have a test harness for the SE stack to ensure their spam is ranked as highly as possible.
  • Ancapistani 6y ago
    By that logic, result quality should be relatively stable, because it’s unlikely that one entity will discover an effective way to game the system. If it can be made to work, it should continue to work.
• tonfreed 6y ago
  The only place I've really found duckduckgo lacking is its meme game and maybe esoteric tech problems. Even with the techy stuff if I narrow my search to github and stackoverflow I usually find what I'm looking for.
• Sami_Lehtinen 6y ago
  Mojeek is a good example of independent search engine. But it isn't distributed nor open source.
• t00r 6y ago
  I prefer DDG over google not because of the search results, but they don't block the Tor IP-range like google do. So google is for me unusable (always revolve captchas is very annoying).