12 ms·

We Need Viable Search Engine Competition

117 points by nemesisj 13y ago

PLenz 13y ago
duckduckgo.com
- sleepyhead 13y ago
  I use it but for many queries it still isn't good enough.
  - hitchhiker999 13y ago
    DDG is just not good enough - it's sloppy. We need an open-source, or decentralised (i wish) solution. That is as close as you can get to a pipe-dream at the moment. Doesn't mean we won't get there eventually though. EDIT: early days open src p2p search: http://yacy.net/en/index.html http://yacy.net/en/index.html
  - username223 13y ago
    DDG is good enough most of the time: What's the formula for the volume of a tetrahedron? What's a "Cleveland Steamer"? When it's not, there's a Google link right at the bottom of the page, so you can turn on private browsing and click on that.
  - Derbasti 13y ago
    Funny. I use DDG as my default. Frankly, I think it's results are way better than Google's. Yes, they are more simple-minded and less "intelligent", but then they are much more predictable as well. As such, it took some getting used to, but now I feel like I search more precisely than I ever would with Google. Way too often, Google will try to be "smart", and search for some "intelligent" interpretation of my search, where no such thing was called for. Especially for technical, precise searches for "strange" strings, this can get really annoying. Also, those bang-searches are just genius. I regularly search Wikipedia, programming language docs, or maps using those. And yes, sometimes Google as well, mostly for fuzzy "how do I" searches or searches that I don't know precise keywords for.
    - stephen_g 13y ago
      > "Way too often, Google will try to be "smart", and search for some "intelligent" interpretation of my search, where no such thing was called for. Especially for technical, precise searches for "strange" strings, this can get really annoying." Wow, so it's not just me then - I was describing my exact same feeling to someone (perhaps less eloquently than you!) just three days ago! Even when I go and set it to 'Verbatim' search (which is well hidden), it still often gives me useless results for these kinds of technical queries...
  - CraigJPerry 13y ago
    If you're outside of the US the results are left wanting for day to day searches. That said, the little helper widgets are brilliant. I find them more useful more often than the Google result widgets. I've already switched back to G on pretty much all my machines.
  - Dewie 13y ago
    A cool thing is that you can prefix your query with "!g" in DDG and it will do an encrypted Google search.
- patdennis 13y ago
  I use duck duck go as my default in my search bar, but I rarely actually use it to search the web directly. Instead, I use the !bang features to get where I need to go faster without taking my hands off the keyboard. Some examples: ctrl-t !w cyclocross ctrl-t !g site:ohio.gov filetype:pdf economic development ctrl-t !image sad reaction gif
  - raganwald 13y ago
    !i does the same thing as !image for me.
hitchhiker999 13y ago
Last year i wrote this: https://medium.com/surveillance-state/32ba2b38c219 https://medium.com/surveillance-state/32ba2b38c219 - and got a proper dollop of hate from certain segments in the webmaster community. They claimed it was 'sour grapes' - I have 140k visitors a day, am doing great, better than great. The monopoly is bad for all of us, even though I'm benefiting as it is. This Google thing is WAY out of control.
Aqueous 13y ago
Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royalties or wait until it expires. I'm not saying that PageRank is the be all end all of search algorithms, and certainly someone somewhere could come up with a different method of ranking superior to Google's. Ranking pages by how many other pages cite them seems like a pretty fundamental insight, and where I would start with any new search engine.
- nemesisj 13y ago
  I didn't realise this. Is this really enforceable? One of the tools that Moz sells is their PR proxy, opensiteexplorer. I wonder how they're able to replicate PR without running afoul of this? Also, is PR really the best, only way to do this? Seems like there are all kinds of better (more modern) signals we could use other than links, which were kind of the only game in town 10ish years ago.
  - Aqueous 13y ago
    Google pays Mozilla hundreds of millions a year to keep its default status in the search bar. Perhaps being able to use PR was part of the deal, or perhaps Mozilla is rich enough to pay royalties to Stanford. Or perhaps Google doesn't mind because Mozilla doesn't seek to directly compete with Google on text search. Edit: Woops. So none of this is true.
    - mrnzc 13y ago
      Moz (formerly: SeoMoz) != Mozilla
      - deleted 13y ago
        [deleted]
      - 13y ago
lawl 13y ago
I wish more people would use and/or improve YaCy[0]. It's a decentralized search engine. Client is written in java. With the power of decentralisation hardware shouldn't really be an issue. It's just that the search result somewhat sucked the last time I tried it. But I guess now is a good time to install it again, help scraping the internet, and maybe hack at the code. [0] http://yacy.net/en/ http://yacy.net/en/
- rocky1138 13y ago
  I love the idea of this stuff but I'm so scared things like this install a backdoor to our PCs. Is there any way to mitigate this? Run YaCy in a VM on VirtualBox?
  - deleted 13y ago
    [deleted]
  - trit 13y ago
    Since it's open source, I doubt something like that would go unnoticed. You can check out the source here though: https://gitorious.org/yacy https://gitorious.org/yacy
gradys 13y ago
I would say that Bing already is a viable alternative. Having used it for several months now, there aren't too many queries that it doesn't handle well, and usually when I check Google's results for those queries, they are equally bad.
- kenjackson 13y ago
  Bing has finally gotten to the point where I don't fallback to Google queries anymore (I used to end up doing that about once a day in the past). I just use Bing now, and only use Google when I hear about some interesting Google art on their homepage.
- alextingle 13y ago
  It's certainly true that Google's search product has been getting worse and worse for years. You have to jump through hoops to get the real ("verbatim") search results, rather than "what we thought you might mean".
- teddyh 13y ago
  So instead of an entrenched monopolist, I’m supposed to choose… Microsoft? That just feels wrong.
  - romanovcode 13y ago
    It's 2014, times have changed.
    - teddyh 13y ago
      I don’t think Microsoft has changed enough to earn my trust or respect. I doubt they could.
  - namenotrequired 13y ago
    Personally, I will try gradys suggestion because I need to use several google products for work; I know Microsoft is just as bad as Google but at least it's not Google, so they'll both only have more partial data.
chestnut-tree 13y ago
My biggest concern with Google is the phenomenal amount of data they track and record about users. They have no self-restraint when it comes to tracking the online behaviour of users. Now they can track across devices (phone, desktop, tablet) giving them unprecedented knowledge of your online activity. Companies are collecting more data about online behaviour than ever before. And no-one has the same online reach as Google. From analytics, to apps, to fonts to jquery - there's barely a site that doesn't link in one form or another back to Google in some way. Google's digital fingerprints reach into every corner of the web. I've said this before, but I was hoping that 2014 would be the year we become more privacy-conscious, but I don't actually think that will be the case. Google get an incredibly easy ride on the subject of privacy and online tracking from the tech community. They're probably salivating at the prospect of capturing even more precise user behaviour through an OS (Chrome) that potentially captures everything you do online. Google aren't capturing this data anonymously either. The tech community's response to this seems largely to be - so what? For anyone who cares about privacy, that's pretty depressing.
- salient 13y ago
  If I were Larry Page, I would put homomorphic encryption research at a much higher priority than quantum computers, or even researching robots (unless they intend to make most of their money from robots within a few short years, and not rely as much on search money, but that seems pretty unlikely in the short term). If they plan on continuing to make money from tracking users, then they'd better figure out a way to do it very securely and without invading user privacy, otherwise they're going to feel an increasingly bigger pain in terms of public perception of Google over this, which could lead to them losing money in the long term, too. I would also forget about tracking "everything possible" until then, and encrypt end-to-end stuff like chatting and video-calls (maybe they can do this one less costly through P2P or a hybrid system). I very much doubt they see a ton of money as a return from tracking and data mining people's chats online. And the downside is quite huge, because those online chats can be abused by the governments. So why not secure them properly? A little cost to them, huge privacy benefit to their users. The goal should be to only track public, and not private information (at least in the short term, and then they should use homomorphic encryption for public information, too, as that will become increasingly more revealing, too, in the future). I would also pay a lot of attention to what the Dark Mail Alliance is doing with e-mail encryption, and I would at the very least implement their protocol as an option for people who want to talk securely with others, from inside Gmail. They don't necessarily have to make all of Gmail encrypted by default, although that would obviously be very nice, but probably not very practical until they figure out homomorphic encryption. There are also other things they could do to make e-mails a little more secure against abusive governments. In the US, ECPA allows the government to take the e-mail after 180 days without a warrant. So how about you ask people to give a password to G-mail, that's locally stored, and can automatically encrypt emails older than 179 days. If you need to access your own 6+ months old e-mails, then you're just asked to insert your password to access them. I don't see this as a huge issue for convenience, since the vast majority of 6+ months old e-mail is never accessed again by most people anyway. The memories of a privacy-focused Google are certainly getting blurry, but I seem to remember there was a time when Google really cared about user privacy, and didn't have the same mentality as NSA for "collecting it all", storing it forever, and using it forever with data-mining. Google has a lot of very smart people working for them. I'm sure they can come up with many more and much better solutions than even I proposed here. The problem is they have to want it. If it doesn't come as an objective from the top, then it's not going to happen. From Wired, last year: > Lloyd made his pitch, proposing a quantum version of Google’s search engine whereby users could make queries and receive results without Google knowing which questions were asked. The men were intrigued. But after conferring with their business manager the next day, Brin and Page informed Lloyd that his scheme went against their business plan. “They want to know everything about everybody who uses their products and services,” he joked. http://www.wired.com/wiredscience/2013/10/computers-big-data/all/ http://www.wired.com/wiredscience/2013/10/computers-big-data... Google needs to lose that attitude. Adapt or die, Google. And by adapt, I mean having their business incentives once again aligned with those of their users, or it's not going to end well for them.
mark_l_watson 13y ago
I agree with the article. I wrote last week (http://markwatson.com/blog/2013-12/practical-internet-security.html http://markwatson.com/blog/2013-12/practical-internet-securi...) about using two browsers. Chrome with default security settings: used only for gmail, google search, facebook, and twitter; Firefox with ghostery: all normal web browsing, search using duckduckgo and bing. My setup encourages me to at least use duckduckgo and/or bing when I am using firefox.
- andrenotgiant 13y ago
  FYI: I would keep tabs on Ghostery, there's a for-profit company behind it that works with Ad Networks - They don't do anything bad now but experience shows that a massive user-base that you make zero revenue off can be a tempting fruit when profits start falling. http://en.wikipedia.org/wiki/Ghostery#Criticism http://en.wikipedia.org/wiki/Ghostery#Criticism
  - mark_l_watson 13y ago
    Thanks Andre, I will watch out for that.
ceph_ 13y ago
>1. Google is making arbitrary rules on how sites should behave, because they have a monopoly. How are rules against paid linking scams or procedure generated content farms considered arbitrary? It's clearly trying to game the system, and the rules are explicitly laid out to tell you NOT to do it. >2. Google needs these rules, because Google’s rankings are apparently trivial to game. If this were true, you can make millions executing your plan for any number of websites. It's not.
- alextingle 13y ago
  > paid linking scams One man's advertising is another man's paid linking scam. Do you know where to draw the line? Google doesn't - or at least they can't create an algorithm that knows the difference, even after significant investment in the problem.
- bushido 13y ago
  > How are rules against paid linking scams or procedure generated content farms considered arbitrary? It's clearly trying to game the system, and the rules are explicitly laid out to tell you NOT to do it. The whole case with the delisting/penalization and subsequent(and extremely speedy) re-listing of RapGenius is a great example of Google's current arbitrary practices. I covered this is some details on another thread[0] in a couple of replies[1][2][3][4]. [0] https://news.ycombinator.com/item?id=7010997 https://news.ycombinator.com/item?id=7010997 [1] https://news.ycombinator.com/item?id=7011063 https://news.ycombinator.com/item?id=7011063 [2] https://news.ycombinator.com/item?id=7011111 https://news.ycombinator.com/item?id=7011111 [3] https://news.ycombinator.com/item?id=7011356 https://news.ycombinator.com/item?id=7011356 [4] https://news.ycombinator.com/item?id=7011597 https://news.ycombinator.com/item?id=7011597
  - ceph_ 13y ago
    The rule they broke is clearly laid out [1]. The practice of manual penalties is as well [2]. They outline a course of action on how to remove the offending links [3]. How is this arbitrary? [1]https://support.google.com/webmasters/answer/66356 https://support.google.com/webmasters/answer/66356 [2]https://support.google.com/webmasters/answer/2604824 https://support.google.com/webmasters/answer/2604824 [3]https://support.google.com/webmasters/answer/2700611 https://support.google.com/webmasters/answer/2700611
r0h1n 13y ago
Perfectly valid expectation with near impossible odds of coming true. Google has gotten so rich, entrenched and popular that IMHO no competitor can dislodge it. I say this as a thoroughly-disappointed user who's tried nearly all the alternatives to Google in the various segments that it operates in. I've managed to stop using nearly all Google services except Android (running CM) and Search. As others have pointed out in the thread, DDG is nowhere near as good, especially if you're not in the US (I'm in India). After having forced myself to use DDG for a month, I've now resigned myself to Google searches with a couple of extra steps: - all searches performed while logged out of all Google services - browser plugins to rewrite all Google tracking URLs from search results So sure, we need viable search engine competition. But don't wait up for it either.
- tsurantino 13y ago
  "Google has gotten so rich, entrenched and popular that IMHO no competitor can dislodge it." Isn't this the definition of a monopoly? And if so, isn't that reason enough to consider search as a public good or a publicly regulated means of accessing information?
  - aggronn 13y ago
    Sure, but what are you saying the implications of this be? Is handicapping the best options for search really whats best for consumers?
  - r0h1n 13y ago
    The key word there is "IMHO" Judging from the various anti-trust cases that have been brought on against Google around the world it's clear that proving that is nearly impossible too. More so because Google operates in a sector (Internet software) that is theoretically open to infinite competition and zero switching costs.
    - gress 13y ago
      That theory - infinite competition and zero switching costs - seems to be completely ungrounded in any kind of reality.
CombAce 13y ago
or: We need an inclusionist wikipedia - where every possible search term starts with an empty page/any possible search term would have an entry, even if it's just for redirecting a spelling mistake. Eventually every search term would grow its own little community of experts. Unfortunately, as pg said, "Deletionists rule Wikipedia."
- examsheart 13y ago
  Appreciate the lateral thinking. Several issues related to evil SEO-like behavior would need to be addressed as well: * Blow away my competitor's links on the topic I'm related to * Always put my links at the top on my topic page * Sneak my link into other pages not directly related to my topic (viagra spam on comp.lang.*)
- mrfusion 13y ago
  That's a really smart idea. I wonder how you'd get a list of search queries to launch it. Maybe you could require a tiny amount of bitcoin to edit a page to prevent spammers?
- pcloadletter 13y ago
  The growth would be exponential. Every day Google sees 15% of queries that have never been seen before. Every single day.
- weinzierl 13y ago
  That's what Wikia[1] initially tried to do. [1] http://en.wikipedia.org/wiki/Wikia http://en.wikipedia.org/wiki/Wikia
AnthonyMouse 13y ago
The complaint seems to boil down to how Google deals with spammers. The question is, how is competition going to make it better? If we instead had five search engines that each had 20% market share, would people stop trying to game them? Would false positives in efforts to thwart spammers be eliminated? How? The problem isn't even one that responds to market forces -- the "victims" (sites that should rank highly in organic results) are no one's customers. If you want to flip the script, they're the product. They have no leverage, and having more search engines leaves them with still no leverage. Only the users of the search engine have leverage because they can switch to another search engine -- but they can do that today. The problem is the alternatives are no better. What we need is a cost-effective accurate way to identify spammers and exclude them from high rankings in search results. Someone who could do that more effectively could challenge Google, because there is a market for spam-free search results. But if that was an easy problem to solve then why hasn't Google solved it? They have the right incentives. It's just not easy, because spammers adapt. Whenever a search engine does something to thwart spammers, the spammers do something different. Having more search engines doesn't make it easier, it makes it harder because each search engine has less resources to dedicate to it and they have to duplicate each other's work, and the cost to sites of legitimate optimization for a larger number of search engines increases which creates an even larger advantage for major institutions over small timers who can't afford the higher cost. It's silly to say "why doesn't Google do X" and list some abstract thing you think would solve it which they've already considered and declined to do. There is probably a reason. Maybe manually curating every website is too expensive. Maybe arbitration proceedings would be overrun with spammers trying to challenge legitimate removals of their spam. And if they're wrong, don't speculate about it on a blog, prove it by building a better search engine. So far no one has been able to do it.
- marcosdumay 13y ago
  Most of his complaints are reasonable (some aren't), what's happening is that Google is not evil enough for them not turning into actual damage. But if you distance yourself from the current context, you'll be able to see how the means Google use to deal with spammers are dangerous, and we are betting the Internet on Google not being evil (or somebody replacing them fast after they do become). Things would be better if we had 5 search engines with 20% market share. But then, I completely agree that just complaining about the issue is useless. Anybody that (thinks that he) can build a competitor for Google will try it or not based on his odds of success, tolerance to risk, etc, not because somebody is complaining.
avalaunch 13y ago
The internet is now littered with badly written content linking to more badly written content, all in an attempt to increase inbound links and ultimately rank higher in Google. And since Google can't tell the difference between badly written and well written content companies continue to pump this crap out. A lot of problems might be solved with some healthy competition but I don't think this particular problem would be. It would require a new way of ranking that severely devalued inbound links and that seems like a really, really hard nut to crack.
jpdelatorre 13y ago
Here's an idea... how about instead of using just one search engine website, browsers will have a functionality that allows you to search for something and have it query to 3 different search engine at the same time? Users can configure which 3 search engine he/she wants to use. That way, website owners won't be so affected when one search engine screwed him up.
- GoodIntentions 13y ago
  https://en.wikipedia.org/wiki/Dogpile https://en.wikipedia.org/wiki/Dogpile I haven't used dogpile in forever - actually thought they were defunct, but your comment brought it to mind. Seems like what you're after?
jorgecastillo 13y ago
I don't think anything can compete with Google when it comes to search in English. I almost feel like Google can read my mind. With the adequate query you can find anything you've ever read on the internet if it's still online. Yesterday night I wanted to re-read a blog post about a guy that made a few bucks with QNX, but I didn't remember much I tried a few queries and I got the expected blog post with this: https://www.google.com/search?q=netherlands+blog+qnx+a+few+bucks https://www.google.com/search?q=netherlands+blog+qnx+a+few+b... http://jacquesmattheij.com/how-to-make-a-quick-buck http://jacquesmattheij.com/how-to-make-a-quick-buck I just tried the same query with DDG and Bing and they don't even come close to giving me the link I want.
- throwawaykf03 13y ago
  Just tried it on Bing: http://www.bing.com/search?q=Netherlands+qnx+a+quick+buck http://www.bing.com/search?q=Netherlands+qnx+a+quick+buck Worked for me. Edit: Different query from yours, my bad. I mixed both your links up.
- srathi 13y ago
  I feel the same. I wanted to find a video about a zombie prank in a New York cafe and did a Google search for it. The first result was the actual video that I needed.[1] Needless to say, Bing fails here.[2] [1] https://www.google.com/search?q=telekinesis+prank https://www.google.com/search?q=telekinesis+prank [2] http://www.bing.com/search?q=telekinesis+prank http://www.bing.com/search?q=telekinesis+prank
  - throwawaykf03 13y ago
    The 2nd and 3rd results on Bing the same as the 1st one on Google.
sigil 13y ago
Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, and PageRank is only a tiny part of that. Try building a search engine sometime. To hackers inspired by this article: here be dragons. DuckDuckGo took the only sane approach and aggregated results from existing search engine apis, then gradually mixed in some secret sauce. Even then, is DDG viable competition for Google? Will it ever be? All the same, I join you in wishing for some Innovators Dilemma-style disruption here. A "toy" service comes along one day that's a substitute for Google search, but only within a tiny niche. Google doesn't take it seriously enough until it's too late...and we have a real ballgame on our hands again. Thought experiment: what could that disruptive niche be?
- Dewie 13y ago
  > Thought experiment: what could that disruptive niche be? Internal search engines could get a lot better. I don't know how people who make websites make them, what tech they use etc., if it's mostly incompetence that does this, but the fact that I'd rather use plain Google than an internal search most of the time tells me that Google is just too good.
- ForHackernews 13y ago
  > Even then, is DDG viable competition for Google? I'd say "yes". I've switched to DDG as my primary search engine, and I'd say they return a good result about 80-90% of the time. They're definitely not as good as Google, but you can always add "g!" to redirect to Google results. I'd say their best selling point is their focus on privacy, but oddly they don't seem to be touting it very loudly or trying to make hay from the recent NSA revelations.
- rogerbinns 13y ago
  > Even then, is DDG viable competition ... An irritation for me with DDG is them formatting result URLs incorrectly, and then ignoring any feedback about it. Two examples are them adding spurious www. prefixes to Google code results, and leaving out slashes when creating Apple developer URLs. DDG could distinguish themselves by playing bazaar to Google's cathedral, but they don't appear to. A search engine that uses crowd sourcing and feedback would be disruptive IMO. The examples. https://duckduckgo.com/?q=apsw https://duckduckgo.com/?q=apsw - note the second link is on google hosting and shows code.google.com/p/apsw/ but clicking on it gives page not found because it goes to WWW.code.google.com/p/apsw/. https://duckduckgo.com/?q=NSString https://duckduckgo.com/?q=NSString and note infobox at top which goes to Apple developer but clicking gives an error. The link should have a slash between Reference and NSString at the end.
drewvolpe 13y ago
This is one of PG's frighteningly ambitious problems and I believe it's an important (and really hard) problem. I think the answer is to focus on an important niche and grow from there. DDG has done a good job by focusing on privacy; they're currently at nearly 4m queries per day. I'm starting by building the search engine I want to use while writing code, which does parallel searches of different parts of the web dynamically based on the query. I have an initial (ugly and very limited) prototype here: http://gigglebang.com/ http://gigglebang.com/ I'm looking for co-founders. If this is an interesting problem to you, email me: gb@dewdrops.net
- greenyoda 13y ago
  Thanks, your prototype looks interesting; I'll try it out. The one-click redirection to DuckDuckGo, Python, etc. is very convenient. It would be nice if you could add a privacy policy to your "about" page that disclosed what kind of data you collect about your users' queries.
nilved 13y ago
DuckDuckGo provides better results than Google. We have viable alternatives.
- BTurkE 13y ago
  You know their web results are from Bing, right?
  - nilved 13y ago
    Bing is one of many places that they get results from, yes.
- alan_cx 13y ago
  I am sorry to say this, but I have been trying to move to DDG for a while now. Its currently my default search where ever I can make it a default search, and frankly, for me, it's pretty much useless. Most results point to US sites, and in most of my search cases, thats no good. Unless my search is incredibly simple, I end up back on google.co.uk. OK, ethics wise, I don't like google one little bit, but in the end, google gives me useful results. DDG rarely does. I simply do not know how people, and there is a lot of them, come to the conclusion that DDG results are in any way better than google, apart from the fact that DDG is not google. It would not totally surprise me if people were more making do with DDG because its not google to make a point, and as a result being over generous about its utility. Maybe for Americans, DDG does give better results, I cant say, but I'd like to know on what criteria that is based. But for me, a non American, like I say, DDG is regrettably mostly useless.
marc0 13y ago
http://www.yandex.com http://www.yandex.com is a real competitor
- gregholmberg 13y ago
  On a whim, I asked yandex.com: 'k-d tree'. Nine results were about the space-partitioning data structure. Also, there is a tree-trimming service on Long Island called K&D Tree Masters. Surprisingly, pleasantly -- all ten items are useful to me.
gesman 13y ago
One of the viable venues to compete with Google is not by developing "new, better search engine", but rather "better vertical platform". What's the best search engine where buyers go? Not google.com and never really been google, it's amazon.com and ebay.com. What's the best search engine to find answers on technical questions? Stackoverflow.com is approaching google very fast. So the answer would be for sharp minds to develop sharper and better vertical portals/platforms and pick google apart piece by piece.
- weinzierl 13y ago
  Not to forget: What's the best search engine to find people? facebook.com
- lubujackson 13y ago
  That's exactly what my startup is trying to address - how do you help users navigate 100, 1000 or even a million search engines? Ultimately, you need a way to browse search engines by topic and bundle them into custom lists for your specific needs. It's early going, but check Nuggety out if you're interested: http://nuggety.com/ http://nuggety.com/
  - gesman 13y ago
    There is no need to navigate million of search engines. There aren't too many to speak of. Aggregating services are successful only where there are millions of equal service providers (like hotels or restaurants) and the one needs a one-stop place (aggregator) to search. When Google (and Bing and others are far secondary) aggregator model couldn't really do much.
  - namenotrequired 13y ago
    I'm interested, but I've checked the front page and am still confused what / who it's for, could you clarify?
    - lubujackson 13y ago
      Sure. I know the site needs some more clarity and that's what I'm working on now. The basic idea is a Pinterest-type community where you can build a list of sites on any topic and share lists with others. But each list lets you search each website directly from the same search box (results come up in a new tab, like Kayak). So what is it good for? Any subject where you want depth or breadth over surface results. So collectors and researchers can build a portal for themselves. And there are some interesting possibilities for specific needs, like a search list for many places to find a cached URL: http://nuggety.com/u/nuggety/cached-webpages http://nuggety.com/u/nuggety/cached-webpages or a search list to search any of Google's 190 international websites: http://nuggety.com/u/nuggety/international-google http://nuggety.com/u/nuggety/international-google
antics 13y ago
The comments here about building search engines that viably compete with Google in terms of quality are hilariously wrong. Hilariously wrong. The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem. Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might be able to make up a small margin, but our search relevance staff -- good though they are -- cannot compete with a brilliant workforce that is 5 times larger than them. If we are going to actually compete with Google, the problem is not hiring smart people, it's hiring 5 times the current number of smart people employed for this task at MS. Never mind buying such a team. MS can afford that. How do you even find that many people in 2-3 years? When you consider the rest of the data, the case for building something like Google really begins to look grim. For example, how much does it really cost to build a search engine? We've poured at least tens of millions of dollars into just search relevance (I'm not even counting infrastructure). That's good mileage, considering that this way is littered with the corpses of companies like Cuil, who made investments in this area and failed miserably. Still, while this has been a great deal for us, we're still not there yet, and it's not clear we will be in the very near future. And so it's worth wondering: if MS can't buy something like that, realistically, who can? DDG? lol no. In the end, this is the true dominating cost of building a search engine: people capital. Other bottlenecks, like engineering debt, politics, etc. pale against the sheer, awe-inspiring investment of Google in people capital.
- ChuckMcM 13y ago
  This. When you do build a competitive search engine and you see how much money Microsoft has put behind theirs to essentially no avail (how many times have angry shareholders tried to get them to kill it off?) you start to realize a key fact. If you want search engine competition you have to take Google's Ad business away. Its weird I know but Google's search ads pay $80 - $100 RPMs and other guys ads pay $30 RPMs. If Microsoft could use Google's ad network they would be a solid contributor to the bottom line of Microsoft (which is why they can't of course). If you could magically peel that Ad network/agency into its own entity and require it to give non-discriminatory terms to everyone, I believe we would have a pretty vibrant search space. My reasoning there is that the money associated with search advertising would fall into buckets that were much more closely aligned with market share, as opposed to today where someone like Microsoft can have a large share of the search 'eyeballs' but only a fraction of the revenue because their Ads don't have the RPM numbers.
digisth 13y ago
Is there any new information regarding the future plans for Bing? This article from a couple of months ago had it looking ominous: http://www.bloomberg.com/news/2013-11-08/microsoft-ceo-candidate-elop-said-to-mull-windows-shift.html http://www.bloomberg.com/news/2013-11-08/microsoft-ceo-candi... "Besides emphasizing Office, Elop would be prepared to sell or shut down major businesses to sharpen the company’s focus, the people said. He would consider ending Microsoft’s costly effort to take on Google with its Bing search engine, and would also consider selling healthy businesses such as the Xbox game console if he determined they weren’t critical to the company’s strategy, the people said."
thedawn 13y ago
You are so right. I agree with every word. Competition is very important and a real helper to the us the users.
yodha77 13y ago
as lot of people have said.. u cannot beat google at search. but, there is a way.. push information that users are looking for before they even think about it (similar to Google NOW). search is still a sophisticated operation for regular users. by showing the right information at the right time, users don't even have to search.
arikrak 13y ago
Yes, search is hard. Look how most websites that build a search for their own site still do a worse job than Google's site search. And its not like people haven't tried to take on Google. Google always faces the threat of companies succeeding in smaller areas of search, but so far it's dealt with those pretty well too. The other threat it faces is a (big) company developing an alternative approach, e.g IBM Watson.
namenotrequired 13y ago
Alternatives: https://www.blippex.org/ https://www.blippex.org/ http://www.yandex.com/ http://www.yandex.com/ https://duckduckgo.com/ https://duckduckgo.com/ And of course: http://www.bing.com/ http://www.bing.com/ http://www.yahoo.com/ http://www.yahoo.com/
- namenotrequired 13y ago
  Oh and also: https://epicsearch.in/ https://epicsearch.in/
adamseabrook 13y ago
We (meanpath) released[1] some crawl data which may be of interest to those of you hacking away on a search engine project. [1] http://blog.meanpath.com/meanpath-january-2014-index/ http://blog.meanpath.com/meanpath-january-2014-index/

117 points by nemesisj 13y ago

PLenz 13y ago

duckduckgo.com

sleepyhead 13y ago
I use it but for many queries it still isn't good enough.
- hitchhiker999 13y ago
  DDG is just not good enough - it's sloppy. We need an open-source, or decentralised (i wish) solution. That is as close as you can get to a pipe-dream at the moment. Doesn't mean we won't get there eventually though. EDIT: early days open src p2p search: http://yacy.net/en/index.html http://yacy.net/en/index.html
- username223 13y ago
  DDG is good enough most of the time: What's the formula for the volume of a tetrahedron? What's a "Cleveland Steamer"? When it's not, there's a Google link right at the bottom of the page, so you can turn on private browsing and click on that.
- Derbasti 13y ago
  Funny. I use DDG as my default. Frankly, I think it's results are way better than Google's. Yes, they are more simple-minded and less "intelligent", but then they are much more predictable as well. As such, it took some getting used to, but now I feel like I search more precisely than I ever would with Google. Way too often, Google will try to be "smart", and search for some "intelligent" interpretation of my search, where no such thing was called for. Especially for technical, precise searches for "strange" strings, this can get really annoying. Also, those bang-searches are just genius. I regularly search Wikipedia, programming language docs, or maps using those. And yes, sometimes Google as well, mostly for fuzzy "how do I" searches or searches that I don't know precise keywords for.
  - stephen_g 13y ago
    > "Way too often, Google will try to be "smart", and search for some "intelligent" interpretation of my search, where no such thing was called for. Especially for technical, precise searches for "strange" strings, this can get really annoying." Wow, so it's not just me then - I was describing my exact same feeling to someone (perhaps less eloquently than you!) just three days ago! Even when I go and set it to 'Verbatim' search (which is well hidden), it still often gives me useless results for these kinds of technical queries...
- CraigJPerry 13y ago
  If you're outside of the US the results are left wanting for day to day searches. That said, the little helper widgets are brilliant. I find them more useful more often than the Google result widgets. I've already switched back to G on pretty much all my machines.
- Dewie 13y ago
  A cool thing is that you can prefix your query with "!g" in DDG and it will do an encrypted Google search.
patdennis 13y ago
I use duck duck go as my default in my search bar, but I rarely actually use it to search the web directly. Instead, I use the !bang features to get where I need to go faster without taking my hands off the keyboard. Some examples: ctrl-t !w cyclocross ctrl-t !g site:ohio.gov filetype:pdf economic development ctrl-t !image sad reaction gif
- raganwald 13y ago
  !i does the same thing as !image for me.

hitchhiker999 13y ago

Last year i wrote this: https://medium.com/surveillance-state/32ba2b38c219 https://medium.com/surveillance-state/32ba2b38c219 - and got a proper dollop of hate from certain segments in the webmaster community. They claimed it was 'sour grapes' - I have 140k visitors a day, am doing great, better than great. The monopoly is bad for all of us, even though I'm benefiting as it is. This Google thing is WAY out of control.

Aqueous 13y ago

Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royalties or wait until it expires. I'm not saying that PageRank is the be all end all of search algorithms, and certainly someone somewhere could come up with a different method of ranking superior to Google's. Ranking pages by how many other pages cite them seems like a pretty fundamental insight, and where I would start with any new search engine.

nemesisj 13y ago
I didn't realise this. Is this really enforceable? One of the tools that Moz sells is their PR proxy, opensiteexplorer. I wonder how they're able to replicate PR without running afoul of this? Also, is PR really the best, only way to do this? Seems like there are all kinds of better (more modern) signals we could use other than links, which were kind of the only game in town 10ish years ago.
- Aqueous 13y ago
  Google pays Mozilla hundreds of millions a year to keep its default status in the search bar. Perhaps being able to use PR was part of the deal, or perhaps Mozilla is rich enough to pay royalties to Stanford. Or perhaps Google doesn't mind because Mozilla doesn't seek to directly compete with Google on text search. Edit: Woops. So none of this is true.
  - mrnzc 13y ago
    Moz (formerly: SeoMoz) != Mozilla
    - deleted 13y ago
      [deleted]
    - 13y ago

lawl 13y ago

I wish more people would use and/or improve YaCy[0]. It's a decentralized search engine. Client is written in java. With the power of decentralisation hardware shouldn't really be an issue. It's just that the search result somewhat sucked the last time I tried it. But I guess now is a good time to install it again, help scraping the internet, and maybe hack at the code. [0] http://yacy.net/en/ http://yacy.net/en/

rocky1138 13y ago
I love the idea of this stuff but I'm so scared things like this install a backdoor to our PCs. Is there any way to mitigate this? Run YaCy in a VM on VirtualBox?
- deleted 13y ago
  [deleted]
- trit 13y ago
  Since it's open source, I doubt something like that would go unnoticed. You can check out the source here though: https://gitorious.org/yacy https://gitorious.org/yacy

gradys 13y ago

I would say that Bing already is a viable alternative. Having used it for several months now, there aren't too many queries that it doesn't handle well, and usually when I check Google's results for those queries, they are equally bad.

kenjackson 13y ago
Bing has finally gotten to the point where I don't fallback to Google queries anymore (I used to end up doing that about once a day in the past). I just use Bing now, and only use Google when I hear about some interesting Google art on their homepage.
alextingle 13y ago
It's certainly true that Google's search product has been getting worse and worse for years. You have to jump through hoops to get the real ("verbatim") search results, rather than "what we thought you might mean".
teddyh 13y ago
So instead of an entrenched monopolist, I’m supposed to choose… Microsoft? That just feels wrong.
- romanovcode 13y ago
  It's 2014, times have changed.
  - teddyh 13y ago
    I don’t think Microsoft has changed enough to earn my trust or respect. I doubt they could.
- namenotrequired 13y ago
  Personally, I will try gradys suggestion because I need to use several google products for work; I know Microsoft is just as bad as Google but at least it's not Google, so they'll both only have more partial data.

chestnut-tree 13y ago

My biggest concern with Google is the phenomenal amount of data they track and record about users. They have no self-restraint when it comes to tracking the online behaviour of users. Now they can track across devices (phone, desktop, tablet) giving them unprecedented knowledge of your online activity. Companies are collecting more data about online behaviour than ever before. And no-one has the same online reach as Google. From analytics, to apps, to fonts to jquery - there's barely a site that doesn't link in one form or another back to Google in some way. Google's digital fingerprints reach into every corner of the web. I've said this before, but I was hoping that 2014 would be the year we become more privacy-conscious, but I don't actually think that will be the case. Google get an incredibly easy ride on the subject of privacy and online tracking from the tech community. They're probably salivating at the prospect of capturing even more precise user behaviour through an OS (Chrome) that potentially captures everything you do online. Google aren't capturing this data anonymously either. The tech community's response to this seems largely to be - so what? For anyone who cares about privacy, that's pretty depressing.

salient 13y ago
If I were Larry Page, I would put homomorphic encryption research at a much higher priority than quantum computers, or even researching robots (unless they intend to make most of their money from robots within a few short years, and not rely as much on search money, but that seems pretty unlikely in the short term). If they plan on continuing to make money from tracking users, then they'd better figure out a way to do it very securely and without invading user privacy, otherwise they're going to feel an increasingly bigger pain in terms of public perception of Google over this, which could lead to them losing money in the long term, too. I would also forget about tracking "everything possible" until then, and encrypt end-to-end stuff like chatting and video-calls (maybe they can do this one less costly through P2P or a hybrid system). I very much doubt they see a ton of money as a return from tracking and data mining people's chats online. And the downside is quite huge, because those online chats can be abused by the governments. So why not secure them properly? A little cost to them, huge privacy benefit to their users. The goal should be to only track public, and not private information (at least in the short term, and then they should use homomorphic encryption for public information, too, as that will become increasingly more revealing, too, in the future). I would also pay a lot of attention to what the Dark Mail Alliance is doing with e-mail encryption, and I would at the very least implement their protocol as an option for people who want to talk securely with others, from inside Gmail. They don't necessarily have to make all of Gmail encrypted by default, although that would obviously be very nice, but probably not very practical until they figure out homomorphic encryption. There are also other things they could do to make e-mails a little more secure against abusive governments. In the US, ECPA allows the government to take the e-mail after 180 days without a warrant. So how about you ask people to give a password to G-mail, that's locally stored, and can automatically encrypt emails older than 179 days. If you need to access your own 6+ months old e-mails, then you're just asked to insert your password to access them. I don't see this as a huge issue for convenience, since the vast majority of 6+ months old e-mail is never accessed again by most people anyway. The memories of a privacy-focused Google are certainly getting blurry, but I seem to remember there was a time when Google really cared about user privacy, and didn't have the same mentality as NSA for "collecting it all", storing it forever, and using it forever with data-mining. Google has a lot of very smart people working for them. I'm sure they can come up with many more and much better solutions than even I proposed here. The problem is they have to want it. If it doesn't come as an objective from the top, then it's not going to happen. From Wired, last year: > Lloyd made his pitch, proposing a quantum version of Google’s search engine whereby users could make queries and receive results without Google knowing which questions were asked. The men were intrigued. But after conferring with their business manager the next day, Brin and Page informed Lloyd that his scheme went against their business plan. “They want to know everything about everybody who uses their products and services,” he joked. http://www.wired.com/wiredscience/2013/10/computers-big-data/all/ http://www.wired.com/wiredscience/2013/10/computers-big-data... Google needs to lose that attitude. Adapt or die, Google. And by adapt, I mean having their business incentives once again aligned with those of their users, or it's not going to end well for them.

mark_l_watson 13y ago

I agree with the article. I wrote last week (http://markwatson.com/blog/2013-12/practical-internet-security.html http://markwatson.com/blog/2013-12/practical-internet-securi...) about using two browsers. Chrome with default security settings: used only for gmail, google search, facebook, and twitter; Firefox with ghostery: all normal web browsing, search using duckduckgo and bing. My setup encourages me to at least use duckduckgo and/or bing when I am using firefox.

andrenotgiant 13y ago
FYI: I would keep tabs on Ghostery, there's a for-profit company behind it that works with Ad Networks - They don't do anything bad now but experience shows that a massive user-base that you make zero revenue off can be a tempting fruit when profits start falling. http://en.wikipedia.org/wiki/Ghostery#Criticism http://en.wikipedia.org/wiki/Ghostery#Criticism
- mark_l_watson 13y ago
  Thanks Andre, I will watch out for that.

ceph_ 13y ago

>1. Google is making arbitrary rules on how sites should behave, because they have a monopoly. How are rules against paid linking scams or procedure generated content farms considered arbitrary? It's clearly trying to game the system, and the rules are explicitly laid out to tell you NOT to do it. >2. Google needs these rules, because Google’s rankings are apparently trivial to game. If this were true, you can make millions executing your plan for any number of websites. It's not.

alextingle 13y ago
> paid linking scams One man's advertising is another man's paid linking scam. Do you know where to draw the line? Google doesn't - or at least they can't create an algorithm that knows the difference, even after significant investment in the problem.
bushido 13y ago
> How are rules against paid linking scams or procedure generated content farms considered arbitrary? It's clearly trying to game the system, and the rules are explicitly laid out to tell you NOT to do it. The whole case with the delisting/penalization and subsequent(and extremely speedy) re-listing of RapGenius is a great example of Google's current arbitrary practices. I covered this is some details on another thread[0] in a couple of replies[1][2][3][4]. [0] https://news.ycombinator.com/item?id=7010997 https://news.ycombinator.com/item?id=7010997 [1] https://news.ycombinator.com/item?id=7011063 https://news.ycombinator.com/item?id=7011063 [2] https://news.ycombinator.com/item?id=7011111 https://news.ycombinator.com/item?id=7011111 [3] https://news.ycombinator.com/item?id=7011356 https://news.ycombinator.com/item?id=7011356 [4] https://news.ycombinator.com/item?id=7011597 https://news.ycombinator.com/item?id=7011597
- ceph_ 13y ago
  The rule they broke is clearly laid out [1]. The practice of manual penalties is as well [2]. They outline a course of action on how to remove the offending links [3]. How is this arbitrary? [1]https://support.google.com/webmasters/answer/66356 https://support.google.com/webmasters/answer/66356 [2]https://support.google.com/webmasters/answer/2604824 https://support.google.com/webmasters/answer/2604824 [3]https://support.google.com/webmasters/answer/2700611 https://support.google.com/webmasters/answer/2700611

r0h1n 13y ago

Perfectly valid expectation with near impossible odds of coming true. Google has gotten so rich, entrenched and popular that IMHO no competitor can dislodge it. I say this as a thoroughly-disappointed user who's tried nearly all the alternatives to Google in the various segments that it operates in. I've managed to stop using nearly all Google services except Android (running CM) and Search. As others have pointed out in the thread, DDG is nowhere near as good, especially if you're not in the US (I'm in India). After having forced myself to use DDG for a month, I've now resigned myself to Google searches with a couple of extra steps: - all searches performed while logged out of all Google services - browser plugins to rewrite all Google tracking URLs from search results So sure, we need viable search engine competition. But don't wait up for it either.

tsurantino 13y ago
"Google has gotten so rich, entrenched and popular that IMHO no competitor can dislodge it." Isn't this the definition of a monopoly? And if so, isn't that reason enough to consider search as a public good or a publicly regulated means of accessing information?
- aggronn 13y ago
  Sure, but what are you saying the implications of this be? Is handicapping the best options for search really whats best for consumers?
- r0h1n 13y ago
  The key word there is "IMHO" Judging from the various anti-trust cases that have been brought on against Google around the world it's clear that proving that is nearly impossible too. More so because Google operates in a sector (Internet software) that is theoretically open to infinite competition and zero switching costs.
  - gress 13y ago
    That theory - infinite competition and zero switching costs - seems to be completely ungrounded in any kind of reality.

CombAce 13y ago

or: We need an inclusionist wikipedia - where every possible search term starts with an empty page/any possible search term would have an entry, even if it's just for redirecting a spelling mistake. Eventually every search term would grow its own little community of experts. Unfortunately, as pg said, "Deletionists rule Wikipedia."

examsheart 13y ago
Appreciate the lateral thinking. Several issues related to evil SEO-like behavior would need to be addressed as well: * Blow away my competitor's links on the topic I'm related to * Always put my links at the top on my topic page * Sneak my link into other pages not directly related to my topic (viagra spam on comp.lang.*)
mrfusion 13y ago
That's a really smart idea. I wonder how you'd get a list of search queries to launch it. Maybe you could require a tiny amount of bitcoin to edit a page to prevent spammers?
pcloadletter 13y ago
The growth would be exponential. Every day Google sees 15% of queries that have never been seen before. Every single day.
weinzierl 13y ago
That's what Wikia[1] initially tried to do. [1] http://en.wikipedia.org/wiki/Wikia http://en.wikipedia.org/wiki/Wikia

AnthonyMouse 13y ago

The complaint seems to boil down to how Google deals with spammers. The question is, how is competition going to make it better? If we instead had five search engines that each had 20% market share, would people stop trying to game them? Would false positives in efforts to thwart spammers be eliminated? How? The problem isn't even one that responds to market forces -- the "victims" (sites that should rank highly in organic results) are no one's customers. If you want to flip the script, they're the product. They have no leverage, and having more search engines leaves them with still no leverage. Only the users of the search engine have leverage because they can switch to another search engine -- but they can do that today. The problem is the alternatives are no better. What we need is a cost-effective accurate way to identify spammers and exclude them from high rankings in search results. Someone who could do that more effectively could challenge Google, because there is a market for spam-free search results. But if that was an easy problem to solve then why hasn't Google solved it? They have the right incentives. It's just not easy, because spammers adapt. Whenever a search engine does something to thwart spammers, the spammers do something different. Having more search engines doesn't make it easier, it makes it harder because each search engine has less resources to dedicate to it and they have to duplicate each other's work, and the cost to sites of legitimate optimization for a larger number of search engines increases which creates an even larger advantage for major institutions over small timers who can't afford the higher cost. It's silly to say "why doesn't Google do X" and list some abstract thing you think would solve it which they've already considered and declined to do. There is probably a reason. Maybe manually curating every website is too expensive. Maybe arbitration proceedings would be overrun with spammers trying to challenge legitimate removals of their spam. And if they're wrong, don't speculate about it on a blog, prove it by building a better search engine. So far no one has been able to do it.

marcosdumay 13y ago
Most of his complaints are reasonable (some aren't), what's happening is that Google is not evil enough for them not turning into actual damage. But if you distance yourself from the current context, you'll be able to see how the means Google use to deal with spammers are dangerous, and we are betting the Internet on Google not being evil (or somebody replacing them fast after they do become). Things would be better if we had 5 search engines with 20% market share. But then, I completely agree that just complaining about the issue is useless. Anybody that (thinks that he) can build a competitor for Google will try it or not based on his odds of success, tolerance to risk, etc, not because somebody is complaining.

avalaunch 13y ago

The internet is now littered with badly written content linking to more badly written content, all in an attempt to increase inbound links and ultimately rank higher in Google. And since Google can't tell the difference between badly written and well written content companies continue to pump this crap out. A lot of problems might be solved with some healthy competition but I don't think this particular problem would be. It would require a new way of ranking that severely devalued inbound links and that seems like a really, really hard nut to crack.

jpdelatorre 13y ago

Here's an idea... how about instead of using just one search engine website, browsers will have a functionality that allows you to search for something and have it query to 3 different search engine at the same time? Users can configure which 3 search engine he/she wants to use. That way, website owners won't be so affected when one search engine screwed him up.

GoodIntentions 13y ago
https://en.wikipedia.org/wiki/Dogpile https://en.wikipedia.org/wiki/Dogpile I haven't used dogpile in forever - actually thought they were defunct, but your comment brought it to mind. Seems like what you're after?

jorgecastillo 13y ago

I don't think anything can compete with Google when it comes to search in English. I almost feel like Google can read my mind. With the adequate query you can find anything you've ever read on the internet if it's still online. Yesterday night I wanted to re-read a blog post about a guy that made a few bucks with QNX, but I didn't remember much I tried a few queries and I got the expected blog post with this: https://www.google.com/search?q=netherlands+blog+qnx+a+few+bucks https://www.google.com/search?q=netherlands+blog+qnx+a+few+b... http://jacquesmattheij.com/how-to-make-a-quick-buck http://jacquesmattheij.com/how-to-make-a-quick-buck I just tried the same query with DDG and Bing and they don't even come close to giving me the link I want.

throwawaykf03 13y ago
Just tried it on Bing: http://www.bing.com/search?q=Netherlands+qnx+a+quick+buck http://www.bing.com/search?q=Netherlands+qnx+a+quick+buck Worked for me. Edit: Different query from yours, my bad. I mixed both your links up.
srathi 13y ago
I feel the same. I wanted to find a video about a zombie prank in a New York cafe and did a Google search for it. The first result was the actual video that I needed.[1] Needless to say, Bing fails here.[2] [1] https://www.google.com/search?q=telekinesis+prank https://www.google.com/search?q=telekinesis+prank [2] http://www.bing.com/search?q=telekinesis+prank http://www.bing.com/search?q=telekinesis+prank
- throwawaykf03 13y ago
  The 2nd and 3rd results on Bing the same as the 1st one on Google.

sigil 13y ago

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, and PageRank is only a tiny part of that. Try building a search engine sometime. To hackers inspired by this article: here be dragons. DuckDuckGo took the only sane approach and aggregated results from existing search engine apis, then gradually mixed in some secret sauce. Even then, is DDG viable competition for Google? Will it ever be? All the same, I join you in wishing for some Innovators Dilemma-style disruption here. A "toy" service comes along one day that's a substitute for Google search, but only within a tiny niche. Google doesn't take it seriously enough until it's too late...and we have a real ballgame on our hands again. Thought experiment: what could that disruptive niche be?

Dewie 13y ago
> Thought experiment: what could that disruptive niche be? Internal search engines could get a lot better. I don't know how people who make websites make them, what tech they use etc., if it's mostly incompetence that does this, but the fact that I'd rather use plain Google than an internal search most of the time tells me that Google is just too good.
ForHackernews 13y ago
> Even then, is DDG viable competition for Google? I'd say "yes". I've switched to DDG as my primary search engine, and I'd say they return a good result about 80-90% of the time. They're definitely not as good as Google, but you can always add "g!" to redirect to Google results. I'd say their best selling point is their focus on privacy, but oddly they don't seem to be touting it very loudly or trying to make hay from the recent NSA revelations.
rogerbinns 13y ago
> Even then, is DDG viable competition ... An irritation for me with DDG is them formatting result URLs incorrectly, and then ignoring any feedback about it. Two examples are them adding spurious www. prefixes to Google code results, and leaving out slashes when creating Apple developer URLs. DDG could distinguish themselves by playing bazaar to Google's cathedral, but they don't appear to. A search engine that uses crowd sourcing and feedback would be disruptive IMO. The examples. https://duckduckgo.com/?q=apsw https://duckduckgo.com/?q=apsw - note the second link is on google hosting and shows code.google.com/p/apsw/ but clicking on it gives page not found because it goes to WWW.code.google.com/p/apsw/. https://duckduckgo.com/?q=NSString https://duckduckgo.com/?q=NSString and note infobox at top which goes to Apple developer but clicking gives an error. The link should have a slash between Reference and NSString at the end.

drewvolpe 13y ago

This is one of PG's frighteningly ambitious problems and I believe it's an important (and really hard) problem. I think the answer is to focus on an important niche and grow from there. DDG has done a good job by focusing on privacy; they're currently at nearly 4m queries per day. I'm starting by building the search engine I want to use while writing code, which does parallel searches of different parts of the web dynamically based on the query. I have an initial (ugly and very limited) prototype here: http://gigglebang.com/ http://gigglebang.com/ I'm looking for co-founders. If this is an interesting problem to you, email me: gb@dewdrops.net

greenyoda 13y ago
Thanks, your prototype looks interesting; I'll try it out. The one-click redirection to DuckDuckGo, Python, etc. is very convenient. It would be nice if you could add a privacy policy to your "about" page that disclosed what kind of data you collect about your users' queries.

nilved 13y ago

DuckDuckGo provides better results than Google. We have viable alternatives.

BTurkE 13y ago
You know their web results are from Bing, right?
- nilved 13y ago
  Bing is one of many places that they get results from, yes.
alan_cx 13y ago
I am sorry to say this, but I have been trying to move to DDG for a while now. Its currently my default search where ever I can make it a default search, and frankly, for me, it's pretty much useless. Most results point to US sites, and in most of my search cases, thats no good. Unless my search is incredibly simple, I end up back on google.co.uk. OK, ethics wise, I don't like google one little bit, but in the end, google gives me useful results. DDG rarely does. I simply do not know how people, and there is a lot of them, come to the conclusion that DDG results are in any way better than google, apart from the fact that DDG is not google. It would not totally surprise me if people were more making do with DDG because its not google to make a point, and as a result being over generous about its utility. Maybe for Americans, DDG does give better results, I cant say, but I'd like to know on what criteria that is based. But for me, a non American, like I say, DDG is regrettably mostly useless.

marc0 13y ago

http://www.yandex.com http://www.yandex.com is a real competitor

gregholmberg 13y ago
On a whim, I asked yandex.com: 'k-d tree'. Nine results were about the space-partitioning data structure. Also, there is a tree-trimming service on Long Island called K&D Tree Masters. Surprisingly, pleasantly -- all ten items are useful to me.

gesman 13y ago

One of the viable venues to compete with Google is not by developing "new, better search engine", but rather "better vertical platform". What's the best search engine where buyers go? Not google.com and never really been google, it's amazon.com and ebay.com. What's the best search engine to find answers on technical questions? Stackoverflow.com is approaching google very fast. So the answer would be for sharp minds to develop sharper and better vertical portals/platforms and pick google apart piece by piece.

weinzierl 13y ago
Not to forget: What's the best search engine to find people? facebook.com
lubujackson 13y ago
That's exactly what my startup is trying to address - how do you help users navigate 100, 1000 or even a million search engines? Ultimately, you need a way to browse search engines by topic and bundle them into custom lists for your specific needs. It's early going, but check Nuggety out if you're interested: http://nuggety.com/ http://nuggety.com/
- gesman 13y ago
  There is no need to navigate million of search engines. There aren't too many to speak of. Aggregating services are successful only where there are millions of equal service providers (like hotels or restaurants) and the one needs a one-stop place (aggregator) to search. When Google (and Bing and others are far secondary) aggregator model couldn't really do much.
- namenotrequired 13y ago
  I'm interested, but I've checked the front page and am still confused what / who it's for, could you clarify?
  - lubujackson 13y ago
    Sure. I know the site needs some more clarity and that's what I'm working on now. The basic idea is a Pinterest-type community where you can build a list of sites on any topic and share lists with others. But each list lets you search each website directly from the same search box (results come up in a new tab, like Kayak). So what is it good for? Any subject where you want depth or breadth over surface results. So collectors and researchers can build a portal for themselves. And there are some interesting possibilities for specific needs, like a search list for many places to find a cached URL: http://nuggety.com/u/nuggety/cached-webpages http://nuggety.com/u/nuggety/cached-webpages or a search list to search any of Google's 190 international websites: http://nuggety.com/u/nuggety/international-google http://nuggety.com/u/nuggety/international-google

antics 13y ago

The comments here about building search engines that viably compete with Google in terms of quality are hilariously wrong. Hilariously wrong. The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem. Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might be able to make up a small margin, but our search relevance staff -- good though they are -- cannot compete with a brilliant workforce that is 5 times larger than them. If we are going to actually compete with Google, the problem is not hiring smart people, it's hiring 5 times the current number of smart people employed for this task at MS. Never mind buying such a team. MS can afford that. How do you even find that many people in 2-3 years? When you consider the rest of the data, the case for building something like Google really begins to look grim. For example, how much does it really cost to build a search engine? We've poured at least tens of millions of dollars into just search relevance (I'm not even counting infrastructure). That's good mileage, considering that this way is littered with the corpses of companies like Cuil, who made investments in this area and failed miserably. Still, while this has been a great deal for us, we're still not there yet, and it's not clear we will be in the very near future. And so it's worth wondering: if MS can't buy something like that, realistically, who can? DDG? lol no. In the end, this is the true dominating cost of building a search engine: people capital. Other bottlenecks, like engineering debt, politics, etc. pale against the sheer, awe-inspiring investment of Google in people capital.

ChuckMcM 13y ago
This. When you do build a competitive search engine and you see how much money Microsoft has put behind theirs to essentially no avail (how many times have angry shareholders tried to get them to kill it off?) you start to realize a key fact. If you want search engine competition you have to take Google's Ad business away. Its weird I know but Google's search ads pay $80 - $100 RPMs and other guys ads pay $30 RPMs. If Microsoft could use Google's ad network they would be a solid contributor to the bottom line of Microsoft (which is why they can't of course). If you could magically peel that Ad network/agency into its own entity and require it to give non-discriminatory terms to everyone, I believe we would have a pretty vibrant search space. My reasoning there is that the money associated with search advertising would fall into buckets that were much more closely aligned with market share, as opposed to today where someone like Microsoft can have a large share of the search 'eyeballs' but only a fraction of the revenue because their Ads don't have the RPM numbers.

digisth 13y ago

Is there any new information regarding the future plans for Bing? This article from a couple of months ago had it looking ominous: http://www.bloomberg.com/news/2013-11-08/microsoft-ceo-candidate-elop-said-to-mull-windows-shift.html http://www.bloomberg.com/news/2013-11-08/microsoft-ceo-candi... "Besides emphasizing Office, Elop would be prepared to sell or shut down major businesses to sharpen the company’s focus, the people said. He would consider ending Microsoft’s costly effort to take on Google with its Bing search engine, and would also consider selling healthy businesses such as the Xbox game console if he determined they weren’t critical to the company’s strategy, the people said."

thedawn 13y ago

You are so right. I agree with every word. Competition is very important and a real helper to the us the users.

yodha77 13y ago

as lot of people have said.. u cannot beat google at search. but, there is a way.. push information that users are looking for before they even think about it (similar to Google NOW). search is still a sophisticated operation for regular users. by showing the right information at the right time, users don't even have to search.

arikrak 13y ago

Yes, search is hard. Look how most websites that build a search for their own site still do a worse job than Google's site search. And its not like people haven't tried to take on Google. Google always faces the threat of companies succeeding in smaller areas of search, but so far it's dealt with those pretty well too. The other threat it faces is a (big) company developing an alternative approach, e.g IBM Watson.

namenotrequired 13y ago

Alternatives: https://www.blippex.org/ https://www.blippex.org/ http://www.yandex.com/ http://www.yandex.com/ https://duckduckgo.com/ https://duckduckgo.com/ And of course: http://www.bing.com/ http://www.bing.com/ http://www.yahoo.com/ http://www.yahoo.com/

namenotrequired 13y ago
Oh and also: https://epicsearch.in/ https://epicsearch.in/

adamseabrook 13y ago

We (meanpath) released[1] some crawl data which may be of interest to those of you hacking away on a search engine project. [1] http://blog.meanpath.com/meanpath-january-2014-index/ http://blog.meanpath.com/meanpath-january-2014-index/