15 ms·

What "viable search engine competition" really looks like

120 points by antics 13y ago

notastartup 13y ago
can you elaborate on search relevance? is it mostly natural language processing? matching keyword queries to document corpus
- antics 13y ago
  Search relevance loosely refers to whether a query returns relevant results. There are different ways of measuring this, none of which is really bulletproof.
  - notastartup 13y ago
    say one types 'sex' into a search engine from that query how does one know user's true intent? Is he looking for a wikipedia article? is he looking to hook up? is he looking for porn? is he referring to the gender? is he just bored and trolling? since google's reach is part of our lives, wouldn't they have a very good idea based on your previous history, what you clicked on, what search results you clicked on, what you typed in google chrome. couldn't microsoft have pulled off the same thing?
    - antics 13y ago
      You don't know for sure what a user means by a query. You can make informed guesses. Typical tricks include looking at what users in the same area have searched for, looking at previous queries (turns out a lot of searches are repeated), looking for "replaceable results" (results that appear on multiple pages), and so on. And no MS could not have pulled off the same thing. The data MS has about you is not even close to the data GOogle has on you.
      - lazyjones 13y ago
        > The data MS has about you is not even close to the data GOogle has on you. Also, users have adapted and are trained to work with Google nowdays, they memorize how to find particular results on Google (even though Google puts a lot of effort into breaking features like "link:...").
    - hahainternet 13y ago
      Interestingly enough my topic result (search history turned off, but signed in to a Google account) was www.funnyordie.com/topic/sex‎ I've never cared about funnyordie, didn't even realise it still existed. Wonder how they picked that, or if other users results differ.
      - notastartup 13y ago
        that is my top result as well, and I haven't been on funnyordie since it stopped being funny around 2008, possibly earlier.
  - thrownaway2424 13y ago
    At Google there is an internal metric that always shows Google search quality leading Bing's. That's fine, but I always wondered if Bing has the same internal search quality metrics and they are always ahead?
- 300bps 13y ago
  Tonight I typed Eagles into Google and it showed me all the information I needed about the game that starts at 8:10 pm. That is search relevance. I've noticed Bing doing a lot of this type of thing for the past year or two but Google is definitely the master of it.
  - hahainternet 13y ago
    Last night I asked Google how much a driving theory lesson costs. It spoke the answer back to me that it had extracted using NLP from the official website. That is the power of their infrastructure.
  - bigbang 13y ago
    Since Google knows a lot about me (for eg) by parsing my emails (I use gmail) and knows about the sites I visit too (Chrome and thru Adsense on 3rd websites), Google is in a much better position to provide me with more relevant results(and ads) than Bing, since I don't use much Microsoft's online services.
  - tluyben2 13y ago
    Google has a lot of ways of knowing what I want to find, which definitely has disadvantages; I get presented with the same ads & results even if I try to find other stuff. I get presented with the same article or ad in different searches numerous times to the point I go on to anonymous browsing / ddg. The good thing; when just searched for something about UIView and later type something "populating a table", it'll give me SO articles (and such) about UITableView which is what I meant but forgot to type add ios/or uitable. The bad thing; when i'm doing research, it'll give me the same stuff over and over again and I have to page quite far and add / remove query info / play with search tools to get to different things which might not be the most popular or might not be what Google thinks I want. The awful thing; the ads... I use ad blockers, but when I get ads (gmail / some sites), they are always the same. For Facebook I just think their algorithm is broken or maybe they have none (I only get ads about dating, cheating, stocks all of which I have no interest in nor ever shown), but for Google they show things I am interested in but I already know all of them, have been there, knew most of them before those ads and they cannot possibly sell me anything. If I wanted something from them I would go there without those ads, so why are they targeting me like that? Show me ads I don't know and if I can use it, I would click. I'm technical, but I hear the same annoyance from others; a friend of mine rents out houses; the search results when he wants to research the competition are usually almost only his own sites (presumably because he visits them a lot?) and ads for his own properties. He is very non technical, so explaining this all to him is not really sinking in.
thoughtloom 13y ago
I find this title to be misleading. It implies the post will demonstrate what viable search engine competition would look like, but just explains the problems that Bing is having with Google. Are we to assume that Bing v. Google is viable search engine competition? While Microsoft's problems are real, this offers little in the way of solutions. That said; interesting read.
- antics 13y ago
  Hmm, I didn't really say that I had the solution. If I did I'd start my own company. :) I said that I would outline the state of making search engines competitive. Here I basically focus the post around areas that are pain points for competition, which I think is pretty relevant to the writeup I set out to make. Do you feel something in particular sucks about the article? I'm happy to change it! That said, I use Bing as a comparison point because it's really the only other point that's comparable in terms of, like, infrastructure and stuff. It's contextualizing more than anything. I might have removed all the references to "Bing", but I don't think it would add clarity.
  - thoughtloom 13y ago
    The article is good as is, but I'd have titled it ~'What the barriers to "viable search engine competition" really look like.'
deleted 13y ago
[deleted]
- antics 13y ago
  Which I say in the first sentence of paragraph 3.
  - deleted 13y ago
    [deleted]
    - antics 13y ago
      Hahaha, well, my bias is certainly not minimal! I work at Microsoft! But I see your point. In fairness I've tried to be honest about our weaknesses and not shy away from the things that are bad. I think I've been reasonably successful, but of course I'd love to hear from you if you think I'm giving MS too much credit.
    - mst 13y ago
      Given the article is explicitly about things he's learned about the process of competing with Google through working for Microsoft, the fact that he works for Microsoft is precisely why he's credible. What would you prefer? Somebody with no relevant experience talking about the subject? I'm sorry, I really don't understand how actually knowing what he's talking about 'marginalises his credibility'.
lazyjones 13y ago
Short and very informative, though I wondered why patents are no concern for a competitor of Google (they have various broad and trivial patents on using historical data, user feedback, moderation for better search results). As the author wrote, aiming at feature parity with Google is probably a bad idea. DDG seems to do this right: they have some better features and one distinct advantage over Google with their stance on privacy. Perhaps Google will reach a certain creepiness factor with all the collected data they're factoring into search results (e.g. I have this suspicion that content from gmail is used too) and at some point it might work against them even for the average user (or am I just biased?). One "feature" that Google has failed to do properly is custom "site search", i.e. providing a search interface for arbitrary websites. Perhaps this is another opportunity for a new competitor or even Bing (though their "site search" was shut down in 2011 apparently), there is plenty of demand (and lots of crappy custom solutions).
- antics 13y ago
  Just a small note about sitesearch, Google are definitely not the market leader there. They don't seem to care much about this feature.
  - wslh 13y ago
    I wrote an opinionated blog post exactly about that saying[1]: "When each site understands its own data better than Google, its internal search results will surpass Google’s. Google will no doubt continue to provide better global results, but the two-tiered search would decentralize efforts to improve algorithms." I see it as one of the Achilles' heel of Google. Even if what they are doing seem impossible to copy. - [1] Letters from the Future: Challenging Google’s Search Engine: http://blog.databigbang.com/letters-from-the-future-challenging-googles-search-engine/ http://blog.databigbang.com/letters-from-the-future-challeng...
    - lazyjones 13y ago
      Currently, it is very easy for websites to provide better search results (but still not good) compared to Google and other search engines like DDG due to the limited crawling, except perhaps for the top few 1000 websites with small enough number of pages/URLs so that at least Google can keep them fully indexed. This is also where dedicated "site search" providers can beat these search engines easily: they can offer a full crawl and depending on the size of the resulting index, they can bill the website owner (e.g. by storage used in GB or number of URLs).
mmmmax 13y ago
Can someone who has the page loaded add a mirror or link to a cached version?
- antics 13y ago
  Is it down for you? It's up for me. It's static HTML hosted on GitHub Pages, so it should remain unbroken...
  - tdumitrescu 13y ago
    down for me too, can't connect
    - antics 13y ago
      Huh, weird. It would be here, if Google had indexed it yet: http://webcache.googleusercontent.com/search?q=cache:http://blog.nullspace.io/building-search-engines.html/ http://webcache.googleusercontent.com/search?q=cache:http://...
jordan_litko 13y ago
If Bing were to overtake Google and become the preferred search engine, would anything really be much different? I'm no search engine expert but I'm pretty comfortable assuming that Bing would have(and does have)had the same issues with rap genius as Google did. The only real difference is that as the market leader Google had to publicly react to rap genius 'gaming' of their system. I think what this shows is that yes, this is precisely a product problem because if Bing were at the top we would be having this exact same discussion only with reversed roles. Correct me if I'm wrong but Google vs Bing is not a lot different than Coke vs Pepsi. Until we have someone come along with a search engine which is actually a better product, market share will only really be a function of effective marketing. That said, I think your article does a good job outlining some of the barriers that any potential competitor in this arena should be aware of.
- antics 13y ago
  The common "problem" that Bing and Google could both potentially be upset with RapGenius is the there is no accountability to the site owners. The search engine's job is to serve results that people want to see. In general, you can absolutely count on them to make decisions that they think will benefit their total relevance in aggregate, even if it hurts a particular subset of queries. But that's not really so much of a product problem as a problem with the problem itself. In other words, it's not really clear that a better product will fix that, because the incentives will probably always point to doing this rather than not.
- azakai 13y ago
  > If Bing were to overtake Google and become the preferred search engine, would anything really be much different? No one wants that (except Microsoft). What we want is for Bing or something else to bring more balance to the search market. Today Google is so dominant that people SEO specifically for it, and Google decides what is and isn't ok to do, and can then decimate site's traffic based on that (as we saw with Rockstar in a recent example). But if there were several successful search engines with no single dominant winner, things would be better. SEO would matter less because you would need to optimize for multiple targets, so you would get away with less tricks (or you need to work much harder to trick everyone, again with the result of less trickery). And no single company could decide the fate of every other company that needs to get traffic through web searches, which is what we have now.
asmman1 13y ago
Imagine if Microsoft begins saving users' information using its operating system. As result they got more information on you even than Google or anyone else.
nostromo 13y ago
I enjoy hating on Microsoft as much as the next guy, but as far as I'm concerned Bing is basically as good as Google. The problem for Microsoft is the same problem that faced Apple. Being "as good as" isn't enough to get people to switch. You have to be 10 times better. Back when Google launched, it was 10x better than Yahoo. Bing seems to have quickly gotten to 1x, but hasn't moved much further.
- username223 13y ago
  "The problem for Microsoft is the same problem that faced Apple." Pretty much: Apple was faced with a competitor who owned a vast majority of the PC market. They never took over the majority, but they managed to earn a profit from a minority of people who were willing to pay a premium for computers that worked better than the crapware offered by most Windows OEMs. Windows was cheap, and just good enough to write TPS reports. Google now offers similar crapware: heavily-gamed and -"personalized" search results slathered with ads and just good enough to be better than most alternatives. There's plenty of room for a company to offer something better: paid subscription search, less-complete search with fewer ads and more accuracy, or something else (I'm not a "10x founder," so I can't think of these things.).
- naner 13y ago
  Bing and DDG are good for stuff that is easy to find. If you rely on a search engine a lot they fall short about 20% of the time.
- FreeFull 13y ago
  I still regularly make searches in duckduckgo (which gets results from Bing) and then find that I have to switch Google to actually find what I'm looking for. The worst thing about this is that Google's results don't seem to be as good as they used to be, and there isn't really any other search engine that matches it yet that I've seen.
  - s3r3nity 13y ago
    Hmmm but DDG will never be quite as good as Bing or Google simply because DDG doesn't keep any histories or track any other activity. Really good search engines get quality and relevance from all those things that DDG purports NOT to track. It's partly a machine learning problem that I just am very very skeptical DDG can get around.
Houshalter 13y ago
What if you made a search engine that only works on specific sites? For the most part users don't want to search the entire web for every small blog in existence, they want to go to a relatively small number of specific sites. For example, reddit has a terrible search engine that users often complain about and comments aren't indexed at all. But searching with google often gives subpar results and they don't take into account things like upvotes at all. (I also think sites like reddit are punished by Google's algorithm pretty harshly.) This might make the problem smaller and more tractable and a place where you could get an advantage over Google.
- PaulHoule 13y ago
  Yep, an effective general purpose search engine is really a collection of effective vertical (subject-intelligent) and horizontal (task-specialized) search engines. The truth behind IBM Watson is that it's based on a methodology that makes it possible to put huge teams working on different parts of the problem without tripping on each other's feet.
- nostrademons 13y ago
  Google already has a product for that that's fairly widely used: https://www.google.com/cse/ https://www.google.com/cse/
- dreamfactory2 13y ago
  How about distributing the indexing to browser plugins? Apart from delegating the work and displacing crawlers, it would also help with relevance, currency, and avoiding link farms.
bra-ket 13y ago
points 1, 4, 6 from the article could have been written in 1998 as well. how can 2 guys in a garage wipe out AltaVista, Lykos, and whoever else was doing search back then (with lots of smart people, data, resources, state-of-the-art software and user adoption to compete against)? What is missing is a serious discussion on search quality (which made Google Google). If Larry and Sergey started today, would they be able to compete against Google by providing better search results, or has search quality reached a "global maximum" of sorts? Does search still suck?
- BryantD 13y ago
  AV made a substantial unforced error at exactly the wrong time which resulted in a huge drop in search quality. It wasn't as much a matter of better algorithms as one might think. There's also a pretty big difference in scale. The AV team working on search in 2000 wasn't all that big.
Kerrick 13y ago
> IE can track, and has tracked users behavior even when they’re not on Bing. (We got in trouble for this once, hehe.) Am I the only one concerned with both the fact that IE tracks its users, and the attitude of this Microsoft employee about it?
- jamesaguilar 13y ago
  Personally, I'm amazed that they let him blog stuff in this detail about MS's search business practices. Not speaking for the org or about the official stance, but I would not consider doing so as a member of SI at Google. And I always had the intuition that MS was even more restrictive about such things. Maybe not?
  - antics 13y ago
    I didn't say anything that isn't completely public knowledge. And let's not forget that MS once employed Robert Scoble. So we can't be all bad. EDIT: and let's be realistic. The papers on search that come out of MSR expose way more about our infrastructure than the paltry sum presented in this post.
    - jamesaguilar 13y ago
      Insiders can often link bits of public knowledge in a way that effectively discloses confidential information. Every company has a different threshold for where this is, so maybe you're on the safe side of that for your org. BTW, I'm not trying to accuse you of malfeasance, I'm just saying I wouldn't do what you're doing.
      - antics 13y ago
        Hmmmmm. Well that is definitely worrying...
      - antics 13y ago
        Fun update, a Google executive just emailed me to say how much he liked this post. So there's that.
adam419 13y ago
Bings' lack of market share compared to Google is not because of any of the reasons listed. It has to do with the perception people have, and the fact that they more regularly associate searching the web with Google and "googling it" vs. Bing and "binging it". At least this is the case for average, non-hacker news people. Why does Google have this perception? I don't know, but I will guess it likely has to do with it being the one search engine that really caught on and was able to sustain its momentum. Competing against google would not be done by any of these incremental/less important technical metrics but more having to do with changing the notion of searching the web, and how, in a likely very niched way at first.
- eruditely 13y ago
  I am not sure if this is true, but your framing of the problem space is good. There should be much more public discussion(maybe there is) about how to achieve this.
- thrownaway2424 13y ago
  Public perception is also the reason that Microsoft can't hire machine learning and search quality engineers, because the question "Do you want to work at Microsoft on search quality?" starts with "Do you want to work at Microsoft?" and for many people the answer to that question is "no" regardless of the remainder of the sentence.
patio11 13y ago
To say nothing of the engineering challenges with crawling the entire Internet. Your first cut will likely refresh every several months or so. Remember when Google did that, back in like 2005? It was very effing hard then. Google now crawls pages newly linked from authority sites essentially instantaneously.
amaks 13y ago
"Bing exists than it is that it is equal to or better than Google in every way" How is Bing better than Google (Search) in every way? With worse relevance and smaller index? Although, IIRC, Bing used to train their neural net based on Google search results to match the Google search quality (remember the 'torsoraphy' incident)?
- robrenaud 13y ago
  You misread that sentence. > it is more important that Bing exists than OTHER CONDITION Where OTHER CONDITION is what you've quoted. He isn't calming that OTHER CONDITION is true, merely that it isn't the most important thing.
username223 13y ago
FTA: "social may pose an existential threat to Google’s style of search" No. Sometimes I want general information: "what is the airspeed of a laden swallow?" Sometimes I want information dependent upon the interests of the hundred or so people I know: "what do you all do in Toledo?" Sometimes I want information dependent upon my interests: "if I enjoyed 'Requiem for a Dream', what other movies should I watch?" These things conflict: I could not possibly care less what my friends think is the airspeed of a laden swallow. I use Google for #1, Facebook for #2, and several other sites for #3. Google has (or had) a perfectly viable business providing the best answers to general questions, and it's a shame that they're screwing it up.
- antics 13y ago
  Hi, I'm the author. Sorry, but you're wrong. Understanding a good amount of detail about your users, and in particular, about their social context, is a massive win for search relevance. Google invested quite a bit in acquiring Aardvark, for example. So, even if you don't ask your friends about something, the information is still really important.
  - s3r3nity 13y ago
    This. I think people get caught up too much in the small fraction of queries that are facts of the universe ("What's the universal gravitational constant to 3 decimal places?") versus the much much larger fraction that depends upon much larger contexts ("Movies playing in theaters nearby," or "Rock music," etc.)
- s3r3nity 13y ago
  "I use Google for #1, Facebook for #2, and several other sites for #3." I think this is the crux of the issue, not that Google thinks social is the end-all answer to everything -- similarly, something like Wolfram Alpha isn't trying to solve the "which restaurant nearby should my friends and I hit tonight?" type of questions. Rather, it's more that Google wants you to be able to approach its products with ANY question or ANY search problem, and it can answer it. Now the problem is whether or not they can pull that off, or you really need independent separate tools/services for each little question to answer.
taxonomyman 13y ago
Web search with up to the top million most popular web sites removed: https://millionshort.com/ https://millionshort.com/
apunic 13y ago
I just checked out Bing with some well known search queries and the results are very close to Google's. What makes me slightly uncomfortable are - the name and the branding. It sounds irrational but the name annoys me and I just do not want to type it. Bing doesn't sound sophisticated, smart or anything and the big picture on the landing page is unwanted and makes the product feel unfocussed. Maybe the reason is just that Bing never did something way better than Google and thus, there were no opportunities to load this name with a good reputation with some achievements. - the speed: Google is (or feels) still way faster here and there (when entering something into the search it instantly switch to the SERP view)
- jkrems 13y ago
  This! The branding of Bing is terrible. When I see the big picture I always think: this has nothing to do with what I'm looking for - and "what I'm looking for" should be the single most important thing for a search engine.
apunic 13y ago
I don't get the point of this post and I think that almost all the lessons the OP wrote down are wrong. Is this post an excuse why MS is performing that bad with Bing? Why is there no one from MS stepping forward and shouting, yes Bing is still behind but we will fight tooth and nail to crush Google -- no, instead some MS employee is writing a non-approved post where he gives 6 lessons why MS sucks and why life is so hard, wtf? Maybe this is the reason why MS falls behind year by year because people behind MS are too laid back and lack fighting spirit. Nobody knows what MS is today, especially nowadays because MS doesn't have any CEO. We know that MS was the company which controlled the personal computer experience in large parts and we know that this experience was ruled by the web and by search for one decade and there is no excuse that MS totally forgot that search must be a integral part.

120 points by antics 13y ago

can you elaborate on search relevance? is it mostly natural language processing? matching keyword queries to document corpus

antics 13y ago
Search relevance loosely refers to whether a query returns relevant results. There are different ways of measuring this, none of which is really bulletproof.
- notastartup 13y ago
  say one types 'sex' into a search engine from that query how does one know user's true intent? Is he looking for a wikipedia article? is he looking to hook up? is he looking for porn? is he referring to the gender? is he just bored and trolling? since google's reach is part of our lives, wouldn't they have a very good idea based on your previous history, what you clicked on, what search results you clicked on, what you typed in google chrome. couldn't microsoft have pulled off the same thing?
  - antics 13y ago
    You don't know for sure what a user means by a query. You can make informed guesses. Typical tricks include looking at what users in the same area have searched for, looking at previous queries (turns out a lot of searches are repeated), looking for "replaceable results" (results that appear on multiple pages), and so on. And no MS could not have pulled off the same thing. The data MS has about you is not even close to the data GOogle has on you.
    - lazyjones 13y ago
      > The data MS has about you is not even close to the data GOogle has on you. Also, users have adapted and are trained to work with Google nowdays, they memorize how to find particular results on Google (even though Google puts a lot of effort into breaking features like "link:...").
  - hahainternet 13y ago
    Interestingly enough my topic result (search history turned off, but signed in to a Google account) was www.funnyordie.com/topic/sex‎ I've never cared about funnyordie, didn't even realise it still existed. Wonder how they picked that, or if other users results differ.
    - notastartup 13y ago
      that is my top result as well, and I haven't been on funnyordie since it stopped being funny around 2008, possibly earlier.
- thrownaway2424 13y ago
  At Google there is an internal metric that always shows Google search quality leading Bing's. That's fine, but I always wondered if Bing has the same internal search quality metrics and they are always ahead?
300bps 13y ago
Tonight I typed Eagles into Google and it showed me all the information I needed about the game that starts at 8:10 pm. That is search relevance. I've noticed Bing doing a lot of this type of thing for the past year or two but Google is definitely the master of it.
- hahainternet 13y ago
  Last night I asked Google how much a driving theory lesson costs. It spoke the answer back to me that it had extracted using NLP from the official website. That is the power of their infrastructure.
- bigbang 13y ago
  Since Google knows a lot about me (for eg) by parsing my emails (I use gmail) and knows about the sites I visit too (Chrome and thru Adsense on 3rd websites), Google is in a much better position to provide me with more relevant results(and ads) than Bing, since I don't use much Microsoft's online services.
- tluyben2 13y ago
  Google has a lot of ways of knowing what I want to find, which definitely has disadvantages; I get presented with the same ads & results even if I try to find other stuff. I get presented with the same article or ad in different searches numerous times to the point I go on to anonymous browsing / ddg. The good thing; when just searched for something about UIView and later type something "populating a table", it'll give me SO articles (and such) about UITableView which is what I meant but forgot to type add ios/or uitable. The bad thing; when i'm doing research, it'll give me the same stuff over and over again and I have to page quite far and add / remove query info / play with search tools to get to different things which might not be the most popular or might not be what Google thinks I want. The awful thing; the ads... I use ad blockers, but when I get ads (gmail / some sites), they are always the same. For Facebook I just think their algorithm is broken or maybe they have none (I only get ads about dating, cheating, stocks all of which I have no interest in nor ever shown), but for Google they show things I am interested in but I already know all of them, have been there, knew most of them before those ads and they cannot possibly sell me anything. If I wanted something from them I would go there without those ads, so why are they targeting me like that? Show me ads I don't know and if I can use it, I would click. I'm technical, but I hear the same annoyance from others; a friend of mine rents out houses; the search results when he wants to research the competition are usually almost only his own sites (presumably because he visits them a lot?) and ads for his own properties. He is very non technical, so explaining this all to him is not really sinking in.

thoughtloom 13y ago

I find this title to be misleading. It implies the post will demonstrate what viable search engine competition would look like, but just explains the problems that Bing is having with Google. Are we to assume that Bing v. Google is viable search engine competition? While Microsoft's problems are real, this offers little in the way of solutions. That said; interesting read.

antics 13y ago
Hmm, I didn't really say that I had the solution. If I did I'd start my own company. :) I said that I would outline the state of making search engines competitive. Here I basically focus the post around areas that are pain points for competition, which I think is pretty relevant to the writeup I set out to make. Do you feel something in particular sucks about the article? I'm happy to change it! That said, I use Bing as a comparison point because it's really the only other point that's comparable in terms of, like, infrastructure and stuff. It's contextualizing more than anything. I might have removed all the references to "Bing", but I don't think it would add clarity.
- thoughtloom 13y ago
  The article is good as is, but I'd have titled it ~'What the barriers to "viable search engine competition" really look like.'

deleted 13y ago

[deleted]

antics 13y ago
Which I say in the first sentence of paragraph 3.
- deleted 13y ago
  [deleted]
  - antics 13y ago
    Hahaha, well, my bias is certainly not minimal! I work at Microsoft! But I see your point. In fairness I've tried to be honest about our weaknesses and not shy away from the things that are bad. I think I've been reasonably successful, but of course I'd love to hear from you if you think I'm giving MS too much credit.
  - mst 13y ago
    Given the article is explicitly about things he's learned about the process of competing with Google through working for Microsoft, the fact that he works for Microsoft is precisely why he's credible. What would you prefer? Somebody with no relevant experience talking about the subject? I'm sorry, I really don't understand how actually knowing what he's talking about 'marginalises his credibility'.

lazyjones 13y ago

Short and very informative, though I wondered why patents are no concern for a competitor of Google (they have various broad and trivial patents on using historical data, user feedback, moderation for better search results). As the author wrote, aiming at feature parity with Google is probably a bad idea. DDG seems to do this right: they have some better features and one distinct advantage over Google with their stance on privacy. Perhaps Google will reach a certain creepiness factor with all the collected data they're factoring into search results (e.g. I have this suspicion that content from gmail is used too) and at some point it might work against them even for the average user (or am I just biased?). One "feature" that Google has failed to do properly is custom "site search", i.e. providing a search interface for arbitrary websites. Perhaps this is another opportunity for a new competitor or even Bing (though their "site search" was shut down in 2011 apparently), there is plenty of demand (and lots of crappy custom solutions).

antics 13y ago
Just a small note about sitesearch, Google are definitely not the market leader there. They don't seem to care much about this feature.
- wslh 13y ago
  I wrote an opinionated blog post exactly about that saying[1]: "When each site understands its own data better than Google, its internal search results will surpass Google’s. Google will no doubt continue to provide better global results, but the two-tiered search would decentralize efforts to improve algorithms." I see it as one of the Achilles' heel of Google. Even if what they are doing seem impossible to copy. - [1] Letters from the Future: Challenging Google’s Search Engine: http://blog.databigbang.com/letters-from-the-future-challenging-googles-search-engine/ http://blog.databigbang.com/letters-from-the-future-challeng...
  - lazyjones 13y ago
    Currently, it is very easy for websites to provide better search results (but still not good) compared to Google and other search engines like DDG due to the limited crawling, except perhaps for the top few 1000 websites with small enough number of pages/URLs so that at least Google can keep them fully indexed. This is also where dedicated "site search" providers can beat these search engines easily: they can offer a full crawl and depending on the size of the resulting index, they can bill the website owner (e.g. by storage used in GB or number of URLs).

mmmmax 13y ago

Can someone who has the page loaded add a mirror or link to a cached version?

antics 13y ago
Is it down for you? It's up for me. It's static HTML hosted on GitHub Pages, so it should remain unbroken...
- tdumitrescu 13y ago
  down for me too, can't connect
  - antics 13y ago
    Huh, weird. It would be here, if Google had indexed it yet: http://webcache.googleusercontent.com/search?q=cache:http://blog.nullspace.io/building-search-engines.html/ http://webcache.googleusercontent.com/search?q=cache:http://...

jordan_litko 13y ago

If Bing were to overtake Google and become the preferred search engine, would anything really be much different? I'm no search engine expert but I'm pretty comfortable assuming that Bing would have(and does have)had the same issues with rap genius as Google did. The only real difference is that as the market leader Google had to publicly react to rap genius 'gaming' of their system. I think what this shows is that yes, this is precisely a product problem because if Bing were at the top we would be having this exact same discussion only with reversed roles. Correct me if I'm wrong but Google vs Bing is not a lot different than Coke vs Pepsi. Until we have someone come along with a search engine which is actually a better product, market share will only really be a function of effective marketing. That said, I think your article does a good job outlining some of the barriers that any potential competitor in this arena should be aware of.

antics 13y ago
The common "problem" that Bing and Google could both potentially be upset with RapGenius is the there is no accountability to the site owners. The search engine's job is to serve results that people want to see. In general, you can absolutely count on them to make decisions that they think will benefit their total relevance in aggregate, even if it hurts a particular subset of queries. But that's not really so much of a product problem as a problem with the problem itself. In other words, it's not really clear that a better product will fix that, because the incentives will probably always point to doing this rather than not.
azakai 13y ago
> If Bing were to overtake Google and become the preferred search engine, would anything really be much different? No one wants that (except Microsoft). What we want is for Bing or something else to bring more balance to the search market. Today Google is so dominant that people SEO specifically for it, and Google decides what is and isn't ok to do, and can then decimate site's traffic based on that (as we saw with Rockstar in a recent example). But if there were several successful search engines with no single dominant winner, things would be better. SEO would matter less because you would need to optimize for multiple targets, so you would get away with less tricks (or you need to work much harder to trick everyone, again with the result of less trickery). And no single company could decide the fate of every other company that needs to get traffic through web searches, which is what we have now.

asmman1 13y ago

Imagine if Microsoft begins saving users' information using its operating system. As result they got more information on you even than Google or anyone else.

nostromo 13y ago

I enjoy hating on Microsoft as much as the next guy, but as far as I'm concerned Bing is basically as good as Google. The problem for Microsoft is the same problem that faced Apple. Being "as good as" isn't enough to get people to switch. You have to be 10 times better. Back when Google launched, it was 10x better than Yahoo. Bing seems to have quickly gotten to 1x, but hasn't moved much further.

username223 13y ago
"The problem for Microsoft is the same problem that faced Apple." Pretty much: Apple was faced with a competitor who owned a vast majority of the PC market. They never took over the majority, but they managed to earn a profit from a minority of people who were willing to pay a premium for computers that worked better than the crapware offered by most Windows OEMs. Windows was cheap, and just good enough to write TPS reports. Google now offers similar crapware: heavily-gamed and -"personalized" search results slathered with ads and just good enough to be better than most alternatives. There's plenty of room for a company to offer something better: paid subscription search, less-complete search with fewer ads and more accuracy, or something else (I'm not a "10x founder," so I can't think of these things.).
naner 13y ago
Bing and DDG are good for stuff that is easy to find. If you rely on a search engine a lot they fall short about 20% of the time.
FreeFull 13y ago
I still regularly make searches in duckduckgo (which gets results from Bing) and then find that I have to switch Google to actually find what I'm looking for. The worst thing about this is that Google's results don't seem to be as good as they used to be, and there isn't really any other search engine that matches it yet that I've seen.
- s3r3nity 13y ago
  Hmmm but DDG will never be quite as good as Bing or Google simply because DDG doesn't keep any histories or track any other activity. Really good search engines get quality and relevance from all those things that DDG purports NOT to track. It's partly a machine learning problem that I just am very very skeptical DDG can get around.

Houshalter 13y ago

What if you made a search engine that only works on specific sites? For the most part users don't want to search the entire web for every small blog in existence, they want to go to a relatively small number of specific sites. For example, reddit has a terrible search engine that users often complain about and comments aren't indexed at all. But searching with google often gives subpar results and they don't take into account things like upvotes at all. (I also think sites like reddit are punished by Google's algorithm pretty harshly.) This might make the problem smaller and more tractable and a place where you could get an advantage over Google.

PaulHoule 13y ago
Yep, an effective general purpose search engine is really a collection of effective vertical (subject-intelligent) and horizontal (task-specialized) search engines. The truth behind IBM Watson is that it's based on a methodology that makes it possible to put huge teams working on different parts of the problem without tripping on each other's feet.
nostrademons 13y ago
Google already has a product for that that's fairly widely used: https://www.google.com/cse/ https://www.google.com/cse/
dreamfactory2 13y ago
How about distributing the indexing to browser plugins? Apart from delegating the work and displacing crawlers, it would also help with relevance, currency, and avoiding link farms.

bra-ket 13y ago

points 1, 4, 6 from the article could have been written in 1998 as well. how can 2 guys in a garage wipe out AltaVista, Lykos, and whoever else was doing search back then (with lots of smart people, data, resources, state-of-the-art software and user adoption to compete against)? What is missing is a serious discussion on search quality (which made Google Google). If Larry and Sergey started today, would they be able to compete against Google by providing better search results, or has search quality reached a "global maximum" of sorts? Does search still suck?

BryantD 13y ago
AV made a substantial unforced error at exactly the wrong time which resulted in a huge drop in search quality. It wasn't as much a matter of better algorithms as one might think. There's also a pretty big difference in scale. The AV team working on search in 2000 wasn't all that big.

Kerrick 13y ago

> IE can track, and has tracked users behavior even when they’re not on Bing. (We got in trouble for this once, hehe.) Am I the only one concerned with both the fact that IE tracks its users, and the attitude of this Microsoft employee about it?

jamesaguilar 13y ago
Personally, I'm amazed that they let him blog stuff in this detail about MS's search business practices. Not speaking for the org or about the official stance, but I would not consider doing so as a member of SI at Google. And I always had the intuition that MS was even more restrictive about such things. Maybe not?
- antics 13y ago
  I didn't say anything that isn't completely public knowledge. And let's not forget that MS once employed Robert Scoble. So we can't be all bad. EDIT: and let's be realistic. The papers on search that come out of MSR expose way more about our infrastructure than the paltry sum presented in this post.
  - jamesaguilar 13y ago
    Insiders can often link bits of public knowledge in a way that effectively discloses confidential information. Every company has a different threshold for where this is, so maybe you're on the safe side of that for your org. BTW, I'm not trying to accuse you of malfeasance, I'm just saying I wouldn't do what you're doing.
    - antics 13y ago
      Hmmmmm. Well that is definitely worrying...
    - antics 13y ago
      Fun update, a Google executive just emailed me to say how much he liked this post. So there's that.

adam419 13y ago

Bings' lack of market share compared to Google is not because of any of the reasons listed. It has to do with the perception people have, and the fact that they more regularly associate searching the web with Google and "googling it" vs. Bing and "binging it". At least this is the case for average, non-hacker news people. Why does Google have this perception? I don't know, but I will guess it likely has to do with it being the one search engine that really caught on and was able to sustain its momentum. Competing against google would not be done by any of these incremental/less important technical metrics but more having to do with changing the notion of searching the web, and how, in a likely very niched way at first.

eruditely 13y ago
I am not sure if this is true, but your framing of the problem space is good. There should be much more public discussion(maybe there is) about how to achieve this.
thrownaway2424 13y ago
Public perception is also the reason that Microsoft can't hire machine learning and search quality engineers, because the question "Do you want to work at Microsoft on search quality?" starts with "Do you want to work at Microsoft?" and for many people the answer to that question is "no" regardless of the remainder of the sentence.

patio11 13y ago

To say nothing of the engineering challenges with crawling the entire Internet. Your first cut will likely refresh every several months or so. Remember when Google did that, back in like 2005? It was very effing hard then. Google now crawls pages newly linked from authority sites essentially instantaneously.

amaks 13y ago

"Bing exists than it is that it is equal to or better than Google in every way" How is Bing better than Google (Search) in every way? With worse relevance and smaller index? Although, IIRC, Bing used to train their neural net based on Google search results to match the Google search quality (remember the 'torsoraphy' incident)?

robrenaud 13y ago
You misread that sentence. > it is more important that Bing exists than OTHER CONDITION Where OTHER CONDITION is what you've quoted. He isn't calming that OTHER CONDITION is true, merely that it isn't the most important thing.

username223 13y ago

FTA: "social may pose an existential threat to Google’s style of search" No. Sometimes I want general information: "what is the airspeed of a laden swallow?" Sometimes I want information dependent upon the interests of the hundred or so people I know: "what do you all do in Toledo?" Sometimes I want information dependent upon my interests: "if I enjoyed 'Requiem for a Dream', what other movies should I watch?" These things conflict: I could not possibly care less what my friends think is the airspeed of a laden swallow. I use Google for #1, Facebook for #2, and several other sites for #3. Google has (or had) a perfectly viable business providing the best answers to general questions, and it's a shame that they're screwing it up.

antics 13y ago
Hi, I'm the author. Sorry, but you're wrong. Understanding a good amount of detail about your users, and in particular, about their social context, is a massive win for search relevance. Google invested quite a bit in acquiring Aardvark, for example. So, even if you don't ask your friends about something, the information is still really important.
- s3r3nity 13y ago
  This. I think people get caught up too much in the small fraction of queries that are facts of the universe ("What's the universal gravitational constant to 3 decimal places?") versus the much much larger fraction that depends upon much larger contexts ("Movies playing in theaters nearby," or "Rock music," etc.)
s3r3nity 13y ago
"I use Google for #1, Facebook for #2, and several other sites for #3." I think this is the crux of the issue, not that Google thinks social is the end-all answer to everything -- similarly, something like Wolfram Alpha isn't trying to solve the "which restaurant nearby should my friends and I hit tonight?" type of questions. Rather, it's more that Google wants you to be able to approach its products with ANY question or ANY search problem, and it can answer it. Now the problem is whether or not they can pull that off, or you really need independent separate tools/services for each little question to answer.

taxonomyman 13y ago

Web search with up to the top million most popular web sites removed: https://millionshort.com/ https://millionshort.com/

apunic 13y ago

I just checked out Bing with some well known search queries and the results are very close to Google's. What makes me slightly uncomfortable are - the name and the branding. It sounds irrational but the name annoys me and I just do not want to type it. Bing doesn't sound sophisticated, smart or anything and the big picture on the landing page is unwanted and makes the product feel unfocussed. Maybe the reason is just that Bing never did something way better than Google and thus, there were no opportunities to load this name with a good reputation with some achievements. - the speed: Google is (or feels) still way faster here and there (when entering something into the search it instantly switch to the SERP view)

jkrems 13y ago
This! The branding of Bing is terrible. When I see the big picture I always think: this has nothing to do with what I'm looking for - and "what I'm looking for" should be the single most important thing for a search engine.

apunic 13y ago

I don't get the point of this post and I think that almost all the lessons the OP wrote down are wrong. Is this post an excuse why MS is performing that bad with Bing? Why is there no one from MS stepping forward and shouting, yes Bing is still behind but we will fight tooth and nail to crush Google -- no, instead some MS employee is writing a non-approved post where he gives 6 lessons why MS sucks and why life is so hard, wtf? Maybe this is the reason why MS falls behind year by year because people behind MS are too laid back and lack fighting spirit. Nobody knows what MS is today, especially nowadays because MS doesn't have any CEO. We know that MS was the company which controlled the personal computer experience in large parts and we know that this experience was ruled by the web and by search for one decade and there is no excuse that MS totally forgot that search must be a integral part.