22 ms·

The solution proposed by Kagi—separate the search index from the rest of Google—seems to make the most sense. Kagi explains it more here: https://blog.kagi.com/

The solution proposed by Kagi—separate the search index from the rest of Google—seems to make the most sense. Kagi explains it more here: https://blog.kagi.com/dawn-new-era-search https://blog.kagi.com/dawn-new-era-search

luckydata 1y ago
It's such a ridiculous proposal that would completely destroy Google's business. If that's the goal fine, but let's not pretend that any of those remedies are anything beyond a death sentence.
- Disposal8433 1y ago
  [flagged]
  - minwcnt5 1y ago
    Sorry, but corporations are not people despite what some people will tell you. They would definitely NOT survive in any recognizable form with "only a few billion dollars", because the stock price is a function of profits. Take away most of the profits, and most of the company's value gets wiped out, most of the employees would leave or get laid off, and anything of value that remains would quickly become worthless. Users would all move to the government-sanctioned replacement monopoly, likely X. To say nothing about the thousands of ordinary people who have large Alphabet holdings in their retirement portfolios and would be wiped out. Google is practically the definition of a "too big to fail" company. They need to be reigned in to allow more competition, but straight up destroying the company would be a move so colossally stupid I could just see the Trump regime doing it.
- ketzo 1y ago
  Really? Google would still have an astonishingly large lead in the ad markets.
  - riku_iki 1y ago
    Not sure how they could hold lead in case they lose search traffic.
- alabastervlog 1y ago
  If they're dominating or one of only two or three important options in multiple other areas and the index is the only reason... I mean, that's a strong argument both that they're monopolists and that they're terrible at allocating the enormous amount of capital they have. That's really the only thing keeping them around? All their other lines of business collectively aren't enough to keep them alive? Yikes, scathing indictment.
- riku_iki 1y ago
  > It's such a ridiculous proposal that would completely destroy Google's business. it won't. My bet is that bing and some other indexes are 95% Ok for average Joe. But relevance ranking is much tougher problem, and "google.com" is household brand with many other functions(maps, news, stocks, weather, knowledge graph, shopping, videos), and that's what is foundation of google monopoly. I think this shared index thing will actually kill competition even more, since every players will use only index owned by google now.
  - AtlasBarfed 1y ago
    At this point, why are you so concerned about Google's business? This was 10 years ago. I could argue a moral Superior that Google possessed over Microsoft and Facebook, but man those days are looooooong gone.
- const_cast 1y ago
  I don't know, I don't think it will. I mean, they're still going to be the number 1 name in adtech and analytics. And they're still gonna have pretty decent personalized ads because of analytics. Plus, that just one part of their business. There's also Android, which is a money printing machine with the Google Store (although that's under attack too).
mullingitover 1y ago
Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, so googlebot wins because they’re the dominant search engine. It makes sense to break that out so everyone has access to the same dataset at FRAND pricing. My heart just wants Google to burn to the ground, but my brain says this is the more reasonable approach.
- hkpack 1y ago
  Most of the tech is set for being a monopoly due to the negligible variable cost associated with serving a customer. Thus being even slightly in front of others is reinforced and the gap only widens.
- oceanplexian 1y ago
  > Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient hardware. Actually there's never been a better time in the history of the Internet to have competing search engines since there's never been so much abundance of performance, bandwidth, and software available at historic low prices or for free.
  - deleted 1y ago
    [deleted]
  - stackskipton 1y ago
    Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.
  - Onavo 1y ago
    In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.
  - immibis 1y ago
rubitxxx2 1y ago
Assuming the simplified diagram of Google’s architecture, sure, it looks like you’re just splitting off a well-isolated part, but it would be a significant hardship to do it in reality. Why not also require Apple to split off only the phone and messaging part of its iPhone, Meta to split off only the user feed data, and for the U.S. federal government to run only out of Washington D.C.? This isn’t the breakup of AT&T in the early 1980s where you could say all the equipment and wiring just now belongs to separate entities. (It wasn’t that simple, but it wasn’t like trying to extract an organ.) I think people have to understand that and know that what they’re doing is killing Google, and it was already on its way into mind-numbed enterprise territory.
- 486sx33 1y ago
  Google killed Google. They should not have decided to become evil. Search can easily be removed, G Suite should be separate too.
  - freedomben 1y ago
    > Search can easily be removed This strikes me like "two easy steps to draw an owl. First draw the head, then draw the body". I generally support some sort of breakup, but hand waving the complexities away is not going to do anybody any good
- lolinder 1y ago
  > Apple to split off only the phone and messaging part of its iPhone Ooh, can we? My wife is super jealous of my ability to install custom apps for phone calls and messaging on Android, it'd be great if Apple would open theirs up to competition. Competition in the SMS app space would also likely help break up the usage of iMessage as a tool to pressure people into getting an iPhone so they get the blue bubble.
  - rubitxxx2 1y ago
    > Ooh, can we? If the dream of a Star Trek future reputation-based government run by AI which secretly manipulates the vote comes true, yes we can! Either that or we could organize competitors to lobby the US or EU for more lawsuits in exchange for billions in kickbacks! (Not implying anything by this.)
ChuckMcM 1y ago
At Blekko we advocated for this as well. Google has two interlocked monopolies, one is the search index and the other is their advertising service. We often joked that if Google reasonable and non-discriminatory priced access to their index, both to themselves and to others, AND they allowed someone to put what ever ads they wanted on those results. That change the landscape dramatically. Google would carve out their crawler/indexer/ranker business and sell access to themselves and others which would allow that business an income that did NOT go back to the parent company (had to be disbursed inside as capex or opex for the business). Then front ends would have a good shot, DDG for example could front the index with the value proposition of privacy. Someone else could front the index with a value proposition of no-ads ever. A third party might front that index attuned to specific use cases like literature search. It would be a very different world.
- indolering 1y ago
  Then why do we see all of these alt search engines and SEO services building out independent indexes? Why don't the competitors cooperate in this fashion already?
  - mondrian 1y ago
    Because everyone worships Thiel's "competition is for losers" and dreams of being a monopoly. Monopolies being the logical outcome of a deregulated environment, for which these companies lobby.
    - dantheman 1y ago
      Throughout history there are very few monopolies and they don't normally last that long; that is unless they get are granted special privileges by the government.
      - mondrian 1y ago
        Concentration is the default in an unregulated environment. Sure pure monopolies with 100% market control are rare but concentration is rampant. A handful of companies dominating tech, airlines, banks, media.
dang 1y ago
Discussed at the time, in case anyone is curious: Dawn of a new era in Search: Balancing innovation, competition, and public good - https://news.ycombinator.com/item?id=41393475 https://news.ycombinator.com/item?id=41393475 - Aug 2024 (79 comments)
CobrastanJorji 1y ago
This just in: small search engine company thinks it's a great idea for small search engine companies to have the same search index as Google. Also, I love this bit: "[Google's] search results are of the best quality among its advertising-driven peers." I can just feel the breath of the guy who jumped in to say "wait, you can't just admit that Google's results are better than Kagi's! You need to add some sorta qualifier there that doesn't apply to us."
- sfpotter 1y ago
  Have you used Kagi or Google recently? Kagi works way better.
  - sdwr 1y ago
    Then why do they want Google's search index?
    - sfpotter 1y ago
      So they can work even better...?
      - deleted 1y ago
        [deleted]
    - hmottestad 1y ago
      Crawling the web is costly. I assume it's cheaper to use the results from someone else's crawling. I don't know what Kagi is using to argue that they should have access to Google's indexes, but I'd guess it's some form of anti trust.
      - guhcampos 1y ago
        Let me add more: crawling the web is costly for EVERYONE. The more crawlers out there, the more useless traffic is being served by every single website on the internet. In an ideal world, there would be a single authoritative index, just as we have with web domains, and all players would cooperate into building, maintaining and improving it, so websites would not need to be constantly hammered by thousands of crawlers everyday.
jmyeet 1y ago
That's like asking the foxes how the farmer should manage his chickens. Kagi is a (wannabe) competitor. Likewise, YC's interest here is in making money by having viable startups and having them acquired. I also don't think crawling the Web is the hard part. It's extraordinarily easy to do it badly [1] but what's the solution here? To have a bunch of wannabe search engines crawl Google's index instead? I've thought about this and I wonder if trying to replicate a general purpose search engine here is the right approach or not. Might it not be easier to target a particular vertical, at least to start with? I refuse to believe Google cannot be bested in every single vertical or that the scale of the job can't be segmented to some degree. [1]: https://stackoverflow.blog/2009/06/16/the-perfect-web-spider-storm/ https://stackoverflow.blog/2009/06/16/the-perfect-web-spider...
giancarlostoro 1y ago
This solution would also yield search engines that will actually be useful and powerful like old Google search was. They have crippled it drastically over the years. Used to be I could find exact quotes of forum posts from memory verbatim. I can't do that on Google or YouTube anymore. It's really dumbed down and watered down.
9cb14c1ec0 1y ago
Googles c suite is clearly not thinking ahead here. They could have helped to slow down the anti-trust lawsuits by opening up their search index to whichever AI company wants to pay for it. Web crawling is expensive, and lots of companies are spending wild amounts of money on it. There is a very clear market arbitrage opportunity between the cost of crawling the web and Google's cost of serving up their existing data.
ankit219 1y ago
Woudl the search index contain only raw data about the websites? Or would some sort of ranking be there? If it's teh latter, its a neat way to ask a company to sell their users data to a third party because any kind of ranking comes via aggregation of users' actions. Without involving any user consent at all.
ethan_smith 1y ago
Separating the index creates a commodity data layer that preserves Google's crawling investment while enabling innovation at the ranking/interface layer, similar to how telecom unbundling worked for ISPs.
Eridrus 1y ago
I feel like there's some conceptual drift going on in Kagi's blog post wrt their proposed remedy. They argue that the search index is an essential facility, and per their link "The essential facilities doctrine attacks a form of exclusionary conduct by which an undertaking controls the conditions of access to an asset forming a ‘bottleneck’ for rivals to compete". But unlike physical locations where bridges/ports can be built, the ability to crawl the internet is not excludable by Google. They do argue that the web is not friendly to new crawlers, but what Kagi wants is not just the raw index itself, but also all the serving/ranking built on top of it so that they do not have to re-engineer it themselves. It's also worth noting that Bing exists, and presumably has it's own index of the web and no evidence has been presented that the raw index content itself is the reason that Bing is not competitive.
onlyrealcuzzo 1y ago
Then you'd just end up with all the ads being scams, and people not wanting to search on Google, because all the top results are scams instead of things they might actually be interested in that are not scams.

luckydata 1y ago

It's such a ridiculous proposal that would completely destroy Google's business. If that's the goal fine, but let's not pretend that any of those remedies are anything beyond a death sentence.

Disposal8433 1y ago
[flagged]
- minwcnt5 1y ago
  Sorry, but corporations are not people despite what some people will tell you. They would definitely NOT survive in any recognizable form with "only a few billion dollars", because the stock price is a function of profits. Take away most of the profits, and most of the company's value gets wiped out, most of the employees would leave or get laid off, and anything of value that remains would quickly become worthless. Users would all move to the government-sanctioned replacement monopoly, likely X. To say nothing about the thousands of ordinary people who have large Alphabet holdings in their retirement portfolios and would be wiped out. Google is practically the definition of a "too big to fail" company. They need to be reigned in to allow more competition, but straight up destroying the company would be a move so colossally stupid I could just see the Trump regime doing it.
ketzo 1y ago
Really? Google would still have an astonishingly large lead in the ad markets.
- riku_iki 1y ago
  Not sure how they could hold lead in case they lose search traffic.
alabastervlog 1y ago
If they're dominating or one of only two or three important options in multiple other areas and the index is the only reason... I mean, that's a strong argument both that they're monopolists and that they're terrible at allocating the enormous amount of capital they have. That's really the only thing keeping them around? All their other lines of business collectively aren't enough to keep them alive? Yikes, scathing indictment.
riku_iki 1y ago
> It's such a ridiculous proposal that would completely destroy Google's business. it won't. My bet is that bing and some other indexes are 95% Ok for average Joe. But relevance ranking is much tougher problem, and "google.com" is household brand with many other functions(maps, news, stocks, weather, knowledge graph, shopping, videos), and that's what is foundation of google monopoly. I think this shared index thing will actually kill competition even more, since every players will use only index owned by google now.
- AtlasBarfed 1y ago
  At this point, why are you so concerned about Google's business? This was 10 years ago. I could argue a moral Superior that Google possessed over Microsoft and Facebook, but man those days are looooooong gone.
const_cast 1y ago
I don't know, I don't think it will. I mean, they're still going to be the number 1 name in adtech and analytics. And they're still gonna have pretty decent personalized ads because of analytics. Plus, that just one part of their business. There's also Android, which is a money printing machine with the Google Store (although that's under attack too).

mullingitover 1y ago

Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, so googlebot wins because they’re the dominant search engine. It makes sense to break that out so everyone has access to the same dataset at FRAND pricing. My heart just wants Google to burn to the ground, but my brain says this is the more reasonable approach.

hkpack 1y ago
Most of the tech is set for being a monopoly due to the negligible variable cost associated with serving a customer. Thus being even slightly in front of others is reinforced and the gap only widens.
oceanplexian 1y ago
> Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient hardware. Actually there's never been a better time in the history of the Internet to have competing search engines since there's never been so much abundance of performance, bandwidth, and software available at historic low prices or for free.
- deleted 1y ago
  [deleted]
- stackskipton 1y ago
  Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.
- Onavo 1y ago
  In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.
- immibis 1y ago

rubitxxx2 1y ago

Assuming the simplified diagram of Google’s architecture, sure, it looks like you’re just splitting off a well-isolated part, but it would be a significant hardship to do it in reality. Why not also require Apple to split off only the phone and messaging part of its iPhone, Meta to split off only the user feed data, and for the U.S. federal government to run only out of Washington D.C.? This isn’t the breakup of AT&T in the early 1980s where you could say all the equipment and wiring just now belongs to separate entities. (It wasn’t that simple, but it wasn’t like trying to extract an organ.) I think people have to understand that and know that what they’re doing is killing Google, and it was already on its way into mind-numbed enterprise territory.

486sx33 1y ago
Google killed Google. They should not have decided to become evil. Search can easily be removed, G Suite should be separate too.
- freedomben 1y ago
  > Search can easily be removed This strikes me like "two easy steps to draw an owl. First draw the head, then draw the body". I generally support some sort of breakup, but hand waving the complexities away is not going to do anybody any good
lolinder 1y ago
> Apple to split off only the phone and messaging part of its iPhone Ooh, can we? My wife is super jealous of my ability to install custom apps for phone calls and messaging on Android, it'd be great if Apple would open theirs up to competition. Competition in the SMS app space would also likely help break up the usage of iMessage as a tool to pressure people into getting an iPhone so they get the blue bubble.
- rubitxxx2 1y ago
  > Ooh, can we? If the dream of a Star Trek future reputation-based government run by AI which secretly manipulates the vote comes true, yes we can! Either that or we could organize competitors to lobby the US or EU for more lawsuits in exchange for billions in kickbacks! (Not implying anything by this.)

ChuckMcM 1y ago

At Blekko we advocated for this as well. Google has two interlocked monopolies, one is the search index and the other is their advertising service. We often joked that if Google reasonable and non-discriminatory priced access to their index, both to themselves and to others, AND they allowed someone to put what ever ads they wanted on those results. That change the landscape dramatically. Google would carve out their crawler/indexer/ranker business and sell access to themselves and others which would allow that business an income that did NOT go back to the parent company (had to be disbursed inside as capex or opex for the business). Then front ends would have a good shot, DDG for example could front the index with the value proposition of privacy. Someone else could front the index with a value proposition of no-ads ever. A third party might front that index attuned to specific use cases like literature search. It would be a very different world.

indolering 1y ago
Then why do we see all of these alt search engines and SEO services building out independent indexes? Why don't the competitors cooperate in this fashion already?
- mondrian 1y ago
  Because everyone worships Thiel's "competition is for losers" and dreams of being a monopoly. Monopolies being the logical outcome of a deregulated environment, for which these companies lobby.
  - dantheman 1y ago
    Throughout history there are very few monopolies and they don't normally last that long; that is unless they get are granted special privileges by the government.
    - mondrian 1y ago
      Concentration is the default in an unregulated environment. Sure pure monopolies with 100% market control are rare but concentration is rampant. A handful of companies dominating tech, airlines, banks, media.

dang 1y ago

Discussed at the time, in case anyone is curious: Dawn of a new era in Search: Balancing innovation, competition, and public good - https://news.ycombinator.com/item?id=41393475 https://news.ycombinator.com/item?id=41393475 - Aug 2024 (79 comments)

CobrastanJorji 1y ago

This just in: small search engine company thinks it's a great idea for small search engine companies to have the same search index as Google. Also, I love this bit: "[Google's] search results are of the best quality among its advertising-driven peers." I can just feel the breath of the guy who jumped in to say "wait, you can't just admit that Google's results are better than Kagi's! You need to add some sorta qualifier there that doesn't apply to us."

sfpotter 1y ago
Have you used Kagi or Google recently? Kagi works way better.
- sdwr 1y ago
  Then why do they want Google's search index?
  - sfpotter 1y ago
    So they can work even better...?
    - deleted 1y ago
      [deleted]
  - hmottestad 1y ago
    Crawling the web is costly. I assume it's cheaper to use the results from someone else's crawling. I don't know what Kagi is using to argue that they should have access to Google's indexes, but I'd guess it's some form of anti trust.
    - guhcampos 1y ago
      Let me add more: crawling the web is costly for EVERYONE. The more crawlers out there, the more useless traffic is being served by every single website on the internet. In an ideal world, there would be a single authoritative index, just as we have with web domains, and all players would cooperate into building, maintaining and improving it, so websites would not need to be constantly hammered by thousands of crawlers everyday.

jmyeet 1y ago

That's like asking the foxes how the farmer should manage his chickens. Kagi is a (wannabe) competitor. Likewise, YC's interest here is in making money by having viable startups and having them acquired. I also don't think crawling the Web is the hard part. It's extraordinarily easy to do it badly [1] but what's the solution here? To have a bunch of wannabe search engines crawl Google's index instead? I've thought about this and I wonder if trying to replicate a general purpose search engine here is the right approach or not. Might it not be easier to target a particular vertical, at least to start with? I refuse to believe Google cannot be bested in every single vertical or that the scale of the job can't be segmented to some degree. [1]: https://stackoverflow.blog/2009/06/16/the-perfect-web-spider-storm/ https://stackoverflow.blog/2009/06/16/the-perfect-web-spider...

giancarlostoro 1y ago

This solution would also yield search engines that will actually be useful and powerful like old Google search was. They have crippled it drastically over the years. Used to be I could find exact quotes of forum posts from memory verbatim. I can't do that on Google or YouTube anymore. It's really dumbed down and watered down.

9cb14c1ec0 1y ago

Googles c suite is clearly not thinking ahead here. They could have helped to slow down the anti-trust lawsuits by opening up their search index to whichever AI company wants to pay for it. Web crawling is expensive, and lots of companies are spending wild amounts of money on it. There is a very clear market arbitrage opportunity between the cost of crawling the web and Google's cost of serving up their existing data.

ankit219 1y ago

Woudl the search index contain only raw data about the websites? Or would some sort of ranking be there? If it's teh latter, its a neat way to ask a company to sell their users data to a third party because any kind of ranking comes via aggregation of users' actions. Without involving any user consent at all.

ethan_smith 1y ago

Separating the index creates a commodity data layer that preserves Google's crawling investment while enabling innovation at the ranking/interface layer, similar to how telecom unbundling worked for ISPs.

Eridrus 1y ago

I feel like there's some conceptual drift going on in Kagi's blog post wrt their proposed remedy. They argue that the search index is an essential facility, and per their link "The essential facilities doctrine attacks a form of exclusionary conduct by which an undertaking controls the conditions of access to an asset forming a ‘bottleneck’ for rivals to compete". But unlike physical locations where bridges/ports can be built, the ability to crawl the internet is not excludable by Google. They do argue that the web is not friendly to new crawlers, but what Kagi wants is not just the raw index itself, but also all the serving/ranking built on top of it so that they do not have to re-engineer it themselves. It's also worth noting that Bing exists, and presumably has it's own index of the web and no evidence has been presented that the raw index content itself is the reason that Bing is not competitive.

onlyrealcuzzo 1y ago

Then you'd just end up with all the ads being scams, and people not wanting to search on Google, because all the top results are scams instead of things they might actually be interested in that are not scams.