5 ms·

It is an interesting exercise to look at the barriers to entry for search engines. Clearly there are technology challenges (internet scale web crawler, indexer

by mgreg 6y ago

It is an interesting exercise to look at the barriers to entry for search engines. Clearly there are technology challenges (internet scale web crawler, indexer, search interface, ad technology) but these are not novel challenges anymore. Funding is an issue as operating a general search index is likely very expensive with high upfront investment. Marginal costs are nearly zero as the major expense is likely fixed (crawling and indexing the internet) while variable costs are a small percentage (serving search results for each request). Google, like Nike, Gucci, and Procter & Gamble, has spent billions to build their brand perception and cement the habit of using their products despite there often being minimal actual differences between competitive products. But the critical one really does seem to be attracting new users. The history of DDG, ClickZ, Bing all seem to validate this. Just getting a user to _try_ a new search engine is very difficult despite it being "one click away." Certainly a deal with a browser maker / mobile manufacturer would aid this but it's priced amazingly out of the reach of competitors denying perhaps the most obvious way to get trials and traffic. Google clearly knows this. How can it be true that switching is so easy ("just a click away") as to suggest that being the default is meaningless, and yet pay >$8 billion a year to be that default.

post_below 6y ago
You're missing the most novel technology: ranking. Without that everything else is pretty much useless (which is why DDG primarily republishes Bing results). You're right that drawing users is a big challenge, but privacy gives you a way in there (again DDG). It just isn't enough if you can't provide better results. Google works hard to maintain their dominance on a lot of fronts, some questionable and in need of regulation, but the biggest reason for their position is, as it always was, ranking algos. Which is extremely expensive in engineering time because ranking isn't static but rather a constant war with gaming.
- mgreg 6y ago
  Thanks for pointing out ranking search results as a technology investment required; I tend to think of it as part of indexing but a good call out. I purposely didn't want to go down the "quality" rat hole because that tends to devolve into subjective opinions (see DDG threads throughout HN). My personal, subjective opinion is that blind tests would likely reveal very little differences and thus justify their major investment in brand advertising made by Google. (not blind but compare https://duckduckgo.com/?q=best+wallpaper https://duckduckgo.com/?q=best+wallpaper with https://www.google.com/search?q=best+wallpaper; https://www.google.com/search?q=best+wallpaper; do you really want 3 pinterest results to this query; this is the burden of being the dominant search engine - you're the one people will game) --edited for typos
  - jeffbee 6y ago
    That seems like a search result not worth comparing. It is, as they say, a stupid question with only stupid answers. Let's compare the last search I actually did on Google: https://duckduckgo.com/?q=how+to+partition+a+plane+into+k-many+equal+weighted+areas&ia=web https://duckduckgo.com/?q=how+to+partition+a+plane+into+k-ma... https://www.google.com/search?q=how+to+partition+a+plane+into+k-many+equal+weighted+areas https://www.google.com/search?q=how+to+partition+a+plane+int... DDG results aren't "bad" as such but they're not as good as the Google results. The three videos that DDG gives me at the top of the SERP are totally irrelevant and the Wikipedia panel DDG offers is while not completely unrelated still doesn't address my query. The google results contain no irrelevant videos, no knowledge panel, just ten research papers. Much better results.
  - dalbasal 6y ago
    I'm not sure either matter, ultimately. It's the industrial structure, rather than the demands of the job (provide a good search engine) that matter here. In order to be a sustainable google competitor, you need to be a competing search business, not just a competing search engine. The business model is such that market share affects unit value. A search business with 1% of google's traffic (or 1% of fb's social traffic) does not make 1% os google's revenue. More scale, more unit value. Meanwhile, the way most/many small search engines monetize is via adwords. Adwords has already been found guilty (EU court) of abusing this nasty grip on their (minimal) competition's revenue avenue.
  - ma2rten 6y ago
    I actually don't see 3 pinterest results, maybe there is some personalization at play. However you can't compare results based on just one search query. Not to mention that your query is probably quite common. It's easy for a search engine to return good results for common queries. The difficulty of search is in the long-tail. Bing invested billions of dollar into search. They had traditional advertising including celebrity endorsements. They were unable to compete with Google on quality. Google has large team of engineers working on search quality and they evaluate each change using human raters.
- feanaro 6y ago
  > Google works hard to maintain their dominance on a lot of fronts, some questionable and in need of regulation, but the biggest reason for their position is, as it always was, ranking algos. I've seen this repeatedly brought into question in recent years and I've also observed it myself. Google's search quality is suffering horribly and it seems at least partly related to them moving away from the PageRank-like algorithms that made it famous (or, more reasonably, including many new additional components to it which are unrelated to PageRank and which are drowning out the signal).
  - post_below 6y ago
    PageRank stopped working when black hat SEO figured out how to game it. They had to evolve PageRank (into things like hubs and authorities, etc..) to maintain quality. If you think their results are bad, try a rudimentary search query on the raw data from one of the open web index projects. Google sucks in a lot of ways but they're still really really good at search.
    - higerordermap 6y ago
      Yeah, I too often think that Google search quality is decreasing. But I am not sure anyone can do better given current tech, because search engine metrics are almost always gamed. There are some limited solutions. If they allow to filter out pages full of tracking and ads, it will filter out lot of shit content, but that probably harms their business model. They might have allowed to exclude some sites from search, too..
pythux 6y ago
This. Since you mention Cliqz, this article might be relevant: https://0x65.dev/blog/2019-12-22/google-competition-is-just-one-click-and-27-billion-us-dollars-away.html https://0x65.dev/blog/2019-12-22/google-competition-is-just-... Google spends billions to secure its position, which makes it virtually impossible for new companies to enter the market.
wyldfire 6y ago
I think if DDG is state of the art design it will probably still suffer the lack of context leg up that google gets from my logged-in searches. That said I often use incognito mode for Google searches and it's still quite good.
fastball 6y ago
Yet Google is still my favorite search engine, and I've seriously tried them all.
- lanstin 6y ago
  Yeah. I have made DDG my default but for specific queries (e.g. not “IMDb Manchurian candidate” but “terraform aws_subnet data”) I end up going back to google like half the time.
CurtHagenlocher 6y ago
I imagine user clicks are an important signal which helps a search engine refine its results. If no one ever clicks on a particular link for they were shown when searching for "foo", then there's something about that link that makes it a less attractive result.
martinald 6y ago
I could be wrong but I always thought the fact Google has such high penetration of Google Analytics makes their ranking a lot easier. They can see how much time users spend on site and other behaviour. This must make it a lot easier to know which results are "good".
- eli 6y ago
  Pretty sure the privacy policy for Analytics prevents them from doing this.
  - martinald 6y ago
    Yeah wasn't sure if that's the case or not. Why else would the offer analytics for free though? It's been (relatively) recently they've allowed people to import GA sessions for retargetting. Always thought it was for being able to improve rankings.
    - eli 6y ago
      They charge for it above a certain volume and for various enterprise features. It’s always integrated well with Google Ads. I imagine it also boxes out other companies who might want that data.
  - mgreg 6y ago
    Perhaps worth a re-read. "Google uses the information shared by sites and apps to deliver our services, maintain and improve them, develop new services, measure the effectiveness of advertising, protect against fraud and abuse, and personalize content and ads you see on Google and on our partners’ sites and apps. See our Privacy Policy to learn more about how we process data for each of these purposes and our Advertising page for more about Google ads, how your information is used in the context of advertising, and how long Google stores this information." https://policies.google.com/technologies/partner-sites https://policies.google.com/technologies/partner-sites
    - eli 6y ago
betaby 6y ago
What happened to to the ClickZ's source code after the project shutdown?
dalbasal 6y ago
>> How can it be true that switching is so easy ("just a click away") as to suggest that being the default is meaningless, and yet pay >$8 billion a year to be that default. When stuff gets paradoxicalish, it can be because the paradigm/model is wrong. In this case, the model is the economic-legal model formulating monopoly theories and antitrust legislation. They were thinking mostly in terms of factories. Thinking of "barriers to entry" in terms of fixed/marginal costs makes a different kind of sense for factories. You can't pay an $nbn "fixed cost" price to "build a fb." That's like the old outsourcing joke/prank: Job: Build me an app that makes $1m. Will pay $10k. It also isn't factory unit economics that determines success for a Google, FB or many other modern mega-businesses. Owning 1% of fb's social media traffic does not yield 1% of fb or adwords' revenue. It's more of a "rewards to scale" than an "economies of scale" thing. Revenue goes up. Unit cost may go down, but that's besides the point for fb or adwords ad revenues. The scale itself matters, not the savings it generates. TLDR: trying to examine google's anatomy using a model based on Bell or Ford circa 1929 will result in confusing questions. Meanwhile, monopolies have a logic of their own. They're valuable, and therefore worth paying to have. That's how deals like this should be viewed, imo. It's strong circumstantial evidence of monopoly. The deal, in itself, shouldn't be the object of objection.
polote 6y ago
> but these are not novel challenges anymore. You clearly underestimate the complexity of turning 4 keywords into 3 links that answer these keywords. This is so hard that even Google is not very good at it. But Google is way better than the competition. Only the crawler infrastructure you need to populate your index requires several thousands of dollars of server per month, that's a lot for a company without founding. Then you need to remove spam, you need to understands each document that you crawl, you need to rank them, ... A simple exercise that you can do is to just do a search for your personal knowledge base, you will see that it is much harder that you thought

by mgreg 6y ago

post_below 6y ago

You're missing the most novel technology: ranking. Without that everything else is pretty much useless (which is why DDG primarily republishes Bing results). You're right that drawing users is a big challenge, but privacy gives you a way in there (again DDG). It just isn't enough if you can't provide better results. Google works hard to maintain their dominance on a lot of fronts, some questionable and in need of regulation, but the biggest reason for their position is, as it always was, ranking algos. Which is extremely expensive in engineering time because ranking isn't static but rather a constant war with gaming.

mgreg 6y ago
Thanks for pointing out ranking search results as a technology investment required; I tend to think of it as part of indexing but a good call out. I purposely didn't want to go down the "quality" rat hole because that tends to devolve into subjective opinions (see DDG threads throughout HN). My personal, subjective opinion is that blind tests would likely reveal very little differences and thus justify their major investment in brand advertising made by Google. (not blind but compare https://duckduckgo.com/?q=best+wallpaper https://duckduckgo.com/?q=best+wallpaper with https://www.google.com/search?q=best+wallpaper; https://www.google.com/search?q=best+wallpaper; do you really want 3 pinterest results to this query; this is the burden of being the dominant search engine - you're the one people will game) --edited for typos
- jeffbee 6y ago
  That seems like a search result not worth comparing. It is, as they say, a stupid question with only stupid answers. Let's compare the last search I actually did on Google: https://duckduckgo.com/?q=how+to+partition+a+plane+into+k-many+equal+weighted+areas&ia=web https://duckduckgo.com/?q=how+to+partition+a+plane+into+k-ma... https://www.google.com/search?q=how+to+partition+a+plane+into+k-many+equal+weighted+areas https://www.google.com/search?q=how+to+partition+a+plane+int... DDG results aren't "bad" as such but they're not as good as the Google results. The three videos that DDG gives me at the top of the SERP are totally irrelevant and the Wikipedia panel DDG offers is while not completely unrelated still doesn't address my query. The google results contain no irrelevant videos, no knowledge panel, just ten research papers. Much better results.
- dalbasal 6y ago
  I'm not sure either matter, ultimately. It's the industrial structure, rather than the demands of the job (provide a good search engine) that matter here. In order to be a sustainable google competitor, you need to be a competing search business, not just a competing search engine. The business model is such that market share affects unit value. A search business with 1% of google's traffic (or 1% of fb's social traffic) does not make 1% os google's revenue. More scale, more unit value. Meanwhile, the way most/many small search engines monetize is via adwords. Adwords has already been found guilty (EU court) of abusing this nasty grip on their (minimal) competition's revenue avenue.
- ma2rten 6y ago
  I actually don't see 3 pinterest results, maybe there is some personalization at play. However you can't compare results based on just one search query. Not to mention that your query is probably quite common. It's easy for a search engine to return good results for common queries. The difficulty of search is in the long-tail. Bing invested billions of dollar into search. They had traditional advertising including celebrity endorsements. They were unable to compete with Google on quality. Google has large team of engineers working on search quality and they evaluate each change using human raters.
feanaro 6y ago
> Google works hard to maintain their dominance on a lot of fronts, some questionable and in need of regulation, but the biggest reason for their position is, as it always was, ranking algos. I've seen this repeatedly brought into question in recent years and I've also observed it myself. Google's search quality is suffering horribly and it seems at least partly related to them moving away from the PageRank-like algorithms that made it famous (or, more reasonably, including many new additional components to it which are unrelated to PageRank and which are drowning out the signal).
- post_below 6y ago
  PageRank stopped working when black hat SEO figured out how to game it. They had to evolve PageRank (into things like hubs and authorities, etc..) to maintain quality. If you think their results are bad, try a rudimentary search query on the raw data from one of the open web index projects. Google sucks in a lot of ways but they're still really really good at search.
  - higerordermap 6y ago
    Yeah, I too often think that Google search quality is decreasing. But I am not sure anyone can do better given current tech, because search engine metrics are almost always gamed. There are some limited solutions. If they allow to filter out pages full of tracking and ads, it will filter out lot of shit content, but that probably harms their business model. They might have allowed to exclude some sites from search, too..

pythux 6y ago

This. Since you mention Cliqz, this article might be relevant: https://0x65.dev/blog/2019-12-22/google-competition-is-just-one-click-and-27-billion-us-dollars-away.html https://0x65.dev/blog/2019-12-22/google-competition-is-just-... Google spends billions to secure its position, which makes it virtually impossible for new companies to enter the market.

wyldfire 6y ago

I think if DDG is state of the art design it will probably still suffer the lack of context leg up that google gets from my logged-in searches. That said I often use incognito mode for Google searches and it's still quite good.

fastball 6y ago

Yet Google is still my favorite search engine, and I've seriously tried them all.

lanstin 6y ago
Yeah. I have made DDG my default but for specific queries (e.g. not “IMDb Manchurian candidate” but “terraform aws_subnet data”) I end up going back to google like half the time.

CurtHagenlocher 6y ago

I imagine user clicks are an important signal which helps a search engine refine its results. If no one ever clicks on a particular link for they were shown when searching for "foo", then there's something about that link that makes it a less attractive result.

martinald 6y ago

I could be wrong but I always thought the fact Google has such high penetration of Google Analytics makes their ranking a lot easier. They can see how much time users spend on site and other behaviour. This must make it a lot easier to know which results are "good".

eli 6y ago
Pretty sure the privacy policy for Analytics prevents them from doing this.
- martinald 6y ago
  Yeah wasn't sure if that's the case or not. Why else would the offer analytics for free though? It's been (relatively) recently they've allowed people to import GA sessions for retargetting. Always thought it was for being able to improve rankings.
  - eli 6y ago
    They charge for it above a certain volume and for various enterprise features. It’s always integrated well with Google Ads. I imagine it also boxes out other companies who might want that data.
- mgreg 6y ago
  Perhaps worth a re-read. "Google uses the information shared by sites and apps to deliver our services, maintain and improve them, develop new services, measure the effectiveness of advertising, protect against fraud and abuse, and personalize content and ads you see on Google and on our partners’ sites and apps. See our Privacy Policy to learn more about how we process data for each of these purposes and our Advertising page for more about Google ads, how your information is used in the context of advertising, and how long Google stores this information." https://policies.google.com/technologies/partner-sites https://policies.google.com/technologies/partner-sites
  - eli 6y ago

betaby 6y ago

What happened to to the ClickZ's source code after the project shutdown?

dalbasal 6y ago

>> How can it be true that switching is so easy ("just a click away") as to suggest that being the default is meaningless, and yet pay >$8 billion a year to be that default. When stuff gets paradoxicalish, it can be because the paradigm/model is wrong. In this case, the model is the economic-legal model formulating monopoly theories and antitrust legislation. They were thinking mostly in terms of factories. Thinking of "barriers to entry" in terms of fixed/marginal costs makes a different kind of sense for factories. You can't pay an $nbn "fixed cost" price to "build a fb." That's like the old outsourcing joke/prank: Job: Build me an app that makes $1m. Will pay $10k. It also isn't factory unit economics that determines success for a Google, FB or many other modern mega-businesses. Owning 1% of fb's social media traffic does not yield 1% of fb or adwords' revenue. It's more of a "rewards to scale" than an "economies of scale" thing. Revenue goes up. Unit cost may go down, but that's besides the point for fb or adwords ad revenues. The scale itself matters, not the savings it generates. TLDR: trying to examine google's anatomy using a model based on Bell or Ford circa 1929 will result in confusing questions. Meanwhile, monopolies have a logic of their own. They're valuable, and therefore worth paying to have. That's how deals like this should be viewed, imo. It's strong circumstantial evidence of monopoly. The deal, in itself, shouldn't be the object of objection.

polote 6y ago

> but these are not novel challenges anymore. You clearly underestimate the complexity of turning 4 keywords into 3 links that answer these keywords. This is so hard that even Google is not very good at it. But Google is way better than the competition. Only the crawler infrastructure you need to populate your index requires several thousands of dollars of server per month, that's a lot for a company without founding. Then you need to remove spam, you need to understands each document that you crawl, you need to rank them, ... A simple exercise that you can do is to just do a search for your personal knowledge base, you will see that it is much harder that you thought