5 ms·

Ultimately I think it’s because the internet is just too big! Google were only able to do it because they had the right algorithm early in the age of the intern

Ultimately I think it’s because the internet is just too big! Google were only able to do it because they had the right algorithm early in the age of the internet. They were then able to grow with it to achieve the scale required. Starting from scratch now on a general internet search engine would be close to impossible without 10s, if not 100s, billions of investment. And you would need that to build the index before even beginning to be competitive. No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation. I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible. You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine. Edit: Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search. We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.

hn_throwaway_99 5y ago
I think your post is key. I would be interested in anyone posting what they think is an economically viable path to challenging Google. Pretty much the one company that has done it at scale (Microsoft) isn't exactly eating Google's lunch (though I think there are a few areas that Bing is better). It would just take at astronomical amount of capital to challenge Google at generic Internet search, and the few organizations with that capital are pretty clear they don't think it's a good bet. For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area. Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.
- ryandrake 5y ago
  > Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later. It's telling, that a Google search scoped to a particular web site (xyz site:website.com) almost always produces better results than that web site's actual built-in search engine. If a web site that knows in detail its own content, its own users, and its own domain, still can't make their built-in search better, how does anyone else have a chance?
  - cupofpython 5y ago
    my take is that the big bottleneck in search is not the index or the search but in understanding what it is the user is searching for in the first place. that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)
  - mbrameld 5y ago
    I don't think the fact that the web site owner hasn't implemented better search than Google for their site is evidence that they can't do it, simply that they haven't.
    - tetris11 5y ago
      This entirely. Big companies grow complacent. Which to me, in time, equates to Google's timely downfall.
      - mbg721 5y ago
        No empire lasts forever; when was the last time you went to Sears?
belter 5y ago
Could not agree more. Not only that, would be willing to pay, for a domain specific, high quality, ad free search engine. In the same range of prices of a Netflix or Amazon Prime subscription.
temp8964 5y ago
You do have a point. However, the internet today is very different from its early days. Its content is much more centralized. In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN. So it seems to me, to build a usable search engine, you only need to index most popular websites.
kevin_thibedeau 5y ago
There is an opportunity for niche applications. I would like a search engine that only indexes discussion forums. These are hard to surface on the SEO driven major search providers.
- notreallyserio 5y ago
  Kagi is working on this with its "lens" feature. I think it works pretty dang well. Most results I get when using the discussions lens come from Reddit, but you can block or reduce the ranking of sites, making this less of an issue, if it is even an issue for you. I suggest trying it out and joining their discord to share feedback (they're responsive).
DesiLurker 5y ago
So basically google is the MLS listings of the web now. Sounds like a job for regulators to step in for both the problems. Agree we need a coop of search engines, but I'll add that probably an org like mozilla is best suited for it, that way it can stay non-profit and recycle any profits back into R&D. personally I believe most of these solid business plan companies can be replaced by a simpler open-source version as IMO a lot of their energy is spend in securing revenue lines & profits.
- eternityforest 5y ago
  I suspect Mozilla would penalize sites that track you, which is most sites, which would cause people to not use it and go back to Google when it wasn't showing them what they expected. Mozilla is probably in the best position to do it, I'm just not sure they actually would do it in a way that appealed to consumers.
nobodyandproud 5y ago
Another way is to add search for walled gardens: The most common by far are corporate documents and datastores. It’s also a harder problem to solve than Google’s algorithm. After two decades Microsoft seems to have brought in support, but they left the hard part to others.
- stingraycharles 5y ago
  Isn’t Algolia pretty much eating this market? They seem to be very dominant on the problem of “here’s my dataset, please provide me with an api / interface to search it”
  - csa 5y ago
    Maybe? When searching for something on HN, I get much better results using Google and site: rather than using the embedded algolia search. I would have to think that this would be true of a corporate data set as well. Their api might be what sets them apart, though. I’m not sure… not in that space.
epicureanideal 5y ago
Maybe a search engine for niches, where the mid-sized contributors to those niches may even pay something to support the search engine? And new people contributing sites to those niches could pay some small fee to be listed (maybe under certain conditions or after 1 year free) and would be "elected" into the search results by a random selection of others in the niche? Where the interests of the group are aligned because they all want more people to use the search engine to find them, so they are incentivized to grow the amount of quality content available.
brentadamson 5y ago
`We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.` Can you elaborate?
- samwillis 5y ago
  Something like a cooperative where companies become the franchisee owner for a specific search vertical. They could get access to technology, software and knowledge, but the costs of infrastructure would be the responsibility of the member. As well as having their own “front page”, website and branding there would then be a “meta” search engine that intelligently searches the franchisees search engines based on the query. If someone searches for something on one engine and it doesn’t have the results the “meta” search engine could be checked and direct the user to another engine that would better fit their query.
  - brentadamson 5y ago
    To clarify, if I search for "Bob Marley" what would happen?
asperous 5y ago
I think are right but not because of the size of the internet. Although the web has grown, the cost of computing and bandwidth has decreased and there’s a lot of open tools, technologies, and open datasets like YaCy and Common Crawl. The data you would want to index is a tiny fraction of the total internet. The challenge comes from the massive investments in turning that data into a good search engine and cutting through spam. That is the investment I think Google has scaled over time. Secondly, mind share; the moat Google has is immense, although I think DuckDuckGo and Bing has shown given enough time it is possible to get mindshare slowly; though if anything this demonstrates the challenge.
tmsh 5y ago
> Ultimately I think it’s because the internet is just too big! I would disagree actually. I think the majority of searches people do and the solutions they would be satisfied with require only indexing major sites. Create a better UX attached to PageRank of the top 10-100k+ Alexa ranked sites and you'd have a competitor that could focus on other improvements. At the bottom you could search Google or DDG if you want more niche results. I.e., optimizing UX of the p90 searches would be enough for people to move and the p90 of searches are happy with a pretty narrow range of sites. (You could insert value prop of: no ads, no tracking, and crowd-sourced refinement -- i.e., realtime upvote/downvoting -- of the top results, previews, caching, more knowledge graph stuff on the right side, deeper metadata processing of top sites (i.e., avoiding the need to 'site:*'), or some other revolutionary UX like a one sentence hypothesis solution of exactly what the person is asking for again upvoted/downvoted for refinement -- Google does this but there is a lot of room for improvement). It's fun to think about. There is an order of magnitude in both precision and accuracy that can be improved for huge swaths of common questions people ask on Google.
Spooky23 5y ago
The internet is big, but information has shrunk. Content has been slurped into walled gardens.
MR4D 5y ago
> Ultimately I think it’s because the internet is just too big! This doesn't address why MSFT can't compete with Google. If you solve that problem, then you really get to the heart of the solution.
freediver 5y ago
> No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation. Isn't this exactly the reason why one should do it, as Google is the essence of the online advertising industry and this very regulation could threaten its existence? >I think there is massive opportunity for domain specific search engines though Challenge with niche search engine is discoverability. Imagine you had 10 of the greatest vertical search engines available today - how do you remember which one to use and how? The beauty of a general web search is that it is an easy to remember entry point to all of what is out there. > You then potentially move towards a building “meta” search engines Ever wondered why is this called meta search engine, and an email software that connects to multiple email servers is called email client and not "meta email". "Search engine client" sounds like a more appropriate name.

hn_throwaway_99 5y ago

I think your post is key. I would be interested in anyone posting what they think is an economically viable path to challenging Google. Pretty much the one company that has done it at scale (Microsoft) isn't exactly eating Google's lunch (though I think there are a few areas that Bing is better). It would just take at astronomical amount of capital to challenge Google at generic Internet search, and the few organizations with that capital are pretty clear they don't think it's a good bet. For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area. Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.

ryandrake 5y ago
> Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later. It's telling, that a Google search scoped to a particular web site (xyz site:website.com) almost always produces better results than that web site's actual built-in search engine. If a web site that knows in detail its own content, its own users, and its own domain, still can't make their built-in search better, how does anyone else have a chance?
- cupofpython 5y ago
  my take is that the big bottleneck in search is not the index or the search but in understanding what it is the user is searching for in the first place. that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)
- mbrameld 5y ago
  I don't think the fact that the web site owner hasn't implemented better search than Google for their site is evidence that they can't do it, simply that they haven't.
  - tetris11 5y ago
    This entirely. Big companies grow complacent. Which to me, in time, equates to Google's timely downfall.
    - mbg721 5y ago
      No empire lasts forever; when was the last time you went to Sears?

belter 5y ago

Could not agree more. Not only that, would be willing to pay, for a domain specific, high quality, ad free search engine. In the same range of prices of a Netflix or Amazon Prime subscription.

temp8964 5y ago

You do have a point. However, the internet today is very different from its early days. Its content is much more centralized. In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN. So it seems to me, to build a usable search engine, you only need to index most popular websites.

kevin_thibedeau 5y ago

There is an opportunity for niche applications. I would like a search engine that only indexes discussion forums. These are hard to surface on the SEO driven major search providers.

notreallyserio 5y ago
Kagi is working on this with its "lens" feature. I think it works pretty dang well. Most results I get when using the discussions lens come from Reddit, but you can block or reduce the ranking of sites, making this less of an issue, if it is even an issue for you. I suggest trying it out and joining their discord to share feedback (they're responsive).

DesiLurker 5y ago

So basically google is the MLS listings of the web now. Sounds like a job for regulators to step in for both the problems. Agree we need a coop of search engines, but I'll add that probably an org like mozilla is best suited for it, that way it can stay non-profit and recycle any profits back into R&D. personally I believe most of these solid business plan companies can be replaced by a simpler open-source version as IMO a lot of their energy is spend in securing revenue lines & profits.

eternityforest 5y ago
I suspect Mozilla would penalize sites that track you, which is most sites, which would cause people to not use it and go back to Google when it wasn't showing them what they expected. Mozilla is probably in the best position to do it, I'm just not sure they actually would do it in a way that appealed to consumers.

nobodyandproud 5y ago

Another way is to add search for walled gardens: The most common by far are corporate documents and datastores. It’s also a harder problem to solve than Google’s algorithm. After two decades Microsoft seems to have brought in support, but they left the hard part to others.

stingraycharles 5y ago
Isn’t Algolia pretty much eating this market? They seem to be very dominant on the problem of “here’s my dataset, please provide me with an api / interface to search it”
- csa 5y ago
  Maybe? When searching for something on HN, I get much better results using Google and site: rather than using the embedded algolia search. I would have to think that this would be true of a corporate data set as well. Their api might be what sets them apart, though. I’m not sure… not in that space.

epicureanideal 5y ago

Maybe a search engine for niches, where the mid-sized contributors to those niches may even pay something to support the search engine? And new people contributing sites to those niches could pay some small fee to be listed (maybe under certain conditions or after 1 year free) and would be "elected" into the search results by a random selection of others in the niche? Where the interests of the group are aligned because they all want more people to use the search engine to find them, so they are incentivized to grow the amount of quality content available.

brentadamson 5y ago

`We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.` Can you elaborate?

samwillis 5y ago
Something like a cooperative where companies become the franchisee owner for a specific search vertical. They could get access to technology, software and knowledge, but the costs of infrastructure would be the responsibility of the member. As well as having their own “front page”, website and branding there would then be a “meta” search engine that intelligently searches the franchisees search engines based on the query. If someone searches for something on one engine and it doesn’t have the results the “meta” search engine could be checked and direct the user to another engine that would better fit their query.
- brentadamson 5y ago
  To clarify, if I search for "Bob Marley" what would happen?

asperous 5y ago

I think are right but not because of the size of the internet. Although the web has grown, the cost of computing and bandwidth has decreased and there’s a lot of open tools, technologies, and open datasets like YaCy and Common Crawl. The data you would want to index is a tiny fraction of the total internet. The challenge comes from the massive investments in turning that data into a good search engine and cutting through spam. That is the investment I think Google has scaled over time. Secondly, mind share; the moat Google has is immense, although I think DuckDuckGo and Bing has shown given enough time it is possible to get mindshare slowly; though if anything this demonstrates the challenge.

tmsh 5y ago

> Ultimately I think it’s because the internet is just too big! I would disagree actually. I think the majority of searches people do and the solutions they would be satisfied with require only indexing major sites. Create a better UX attached to PageRank of the top 10-100k+ Alexa ranked sites and you'd have a competitor that could focus on other improvements. At the bottom you could search Google or DDG if you want more niche results. I.e., optimizing UX of the p90 searches would be enough for people to move and the p90 of searches are happy with a pretty narrow range of sites. (You could insert value prop of: no ads, no tracking, and crowd-sourced refinement -- i.e., realtime upvote/downvoting -- of the top results, previews, caching, more knowledge graph stuff on the right side, deeper metadata processing of top sites (i.e., avoiding the need to 'site:*'), or some other revolutionary UX like a one sentence hypothesis solution of exactly what the person is asking for again upvoted/downvoted for refinement -- Google does this but there is a lot of room for improvement). It's fun to think about. There is an order of magnitude in both precision and accuracy that can be improved for huge swaths of common questions people ask on Google.

Spooky23 5y ago

The internet is big, but information has shrunk. Content has been slurped into walled gardens.

MR4D 5y ago

> Ultimately I think it’s because the internet is just too big! This doesn't address why MSFT can't compete with Google. If you solve that problem, then you really get to the heart of the solution.

freediver 5y ago

> No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation. Isn't this exactly the reason why one should do it, as Google is the essence of the online advertising industry and this very regulation could threaten its existence? >I think there is massive opportunity for domain specific search engines though Challenge with niche search engine is discoverability. Imagine you had 10 of the greatest vertical search engines available today - how do you remember which one to use and how? The beauty of a general web search is that it is an easy to remember entry point to all of what is out there. > You then potentially move towards a building “meta” search engines Ever wondered why is this called meta search engine, and an email software that connects to multiple email servers is called email client and not "meta email". "Search engine client" sounds like a more appropriate name.