4 ms·

Maybe because search is an extremely hard problem and requires hundreds of millions (if not billions) of investment to remain competitive?

eviks 2y ago
What's incredibly hard allowing client side filtering like kagi is doing? How many millions do you need to spend to decrease your own spending on AI for every query?
Night_Thastus 2y ago
Also: There is a MASSIVE incentive for foul play from everyone being searched. Every single business wants their results at the top. Everyone is grasping for every single advertiser's cents. The monetary encouragement to game the system is so incredibly strong. Google has to fight that uphill battle 24/7. I have a lot of issues with Google as a company, but I do not envy their position when it comes to the web search. It is a cursed problem no matter how you look at it.
- mc32 2y ago
  If they only divided it into information search ("library" knowledge) and business search (Yelp/YP) where you can tweak your preferences. When you search for "vacuum cleaner", you either want online stores or you may want a local store. In either case it's a business query rather than looking for reviews or specs. What's really polluting search, including Google, is the on-demand content generation based on your query. It's a sea of flotsam.
  - ryandrake 2y ago
    Unfortunately, if you did that, the businesses would still do their best to muscle in on the "information search" results. Marketers gotta Market, and placement on a SERP is a zero sum game. If one competitor managed to get themselves highly ranked in the "information search" then there would be incentive for all competitors to try.
    - mc32 2y ago
      I guess the idea is you disallow them from optimizing at all in the info search so much that you derank them on the biz for some not inconsequential amount of time if they misbehave.
- ta_1138 2y ago
  No doubt, but there's a whole lot of problems shaped like that in tech today. Look at any two sided market: Amazon/Etsy's third party sellers also want to be at the top, and there are bad actors trying to scam the intermediary and the customer. The same low friction onboarding that allows the company to succeed as an intermediary is also facilitating the fraud. Merchants can be fraudsters, advertisers can be fraudsters, users can be fraudsters producing fake clicks. There's economic incentives everywhere. See the relatively recent news of spotify getting generated songs, which are then played by fake listeners. The companies are entering this situations with their eyes wide open. It's a necessary problem when you want to be an intermediary at a large enough scale. And if you don't have the scale, the per-interaction costs are really high, and your company gets beaten by someone with an easier onboarding funnel.
- jldugger 2y ago
  >It is a cursed problem no matter how you look at it. But the funding model is also extremely blessed: you have a stream of people telling you every page load what they're interested in at that exact moment, and it's not hard to match them up with advertisers for those keywords. And arguably the SEO parasitism problem is worsened by their monopoly -- to the extent that optimizing a site for one engine deoptimizes it on on others, the stronger the lead engine #1 has the more incentive there is to game it.
duskwuff 2y ago
It's only getting harder and harder for newcomers to compete. Especially with the rise of abusive AI scrapers, webmasters are increasingly hostile to new web crawlers. Google and Bing get a pass because they're well-known, but any new service is fighting an uphill battle; a lot of sites will block them right off the bat. Nor does it help that some large sites (like Reddit) have inside deals with Google to feed them updates directly.

eviks 2y ago

What's incredibly hard allowing client side filtering like kagi is doing? How many millions do you need to spend to decrease your own spending on AI for every query?

Night_Thastus 2y ago

Also: There is a MASSIVE incentive for foul play from everyone being searched. Every single business wants their results at the top. Everyone is grasping for every single advertiser's cents. The monetary encouragement to game the system is so incredibly strong. Google has to fight that uphill battle 24/7. I have a lot of issues with Google as a company, but I do not envy their position when it comes to the web search. It is a cursed problem no matter how you look at it.

mc32 2y ago
If they only divided it into information search ("library" knowledge) and business search (Yelp/YP) where you can tweak your preferences. When you search for "vacuum cleaner", you either want online stores or you may want a local store. In either case it's a business query rather than looking for reviews or specs. What's really polluting search, including Google, is the on-demand content generation based on your query. It's a sea of flotsam.
- ryandrake 2y ago
  Unfortunately, if you did that, the businesses would still do their best to muscle in on the "information search" results. Marketers gotta Market, and placement on a SERP is a zero sum game. If one competitor managed to get themselves highly ranked in the "information search" then there would be incentive for all competitors to try.
  - mc32 2y ago
    I guess the idea is you disallow them from optimizing at all in the info search so much that you derank them on the biz for some not inconsequential amount of time if they misbehave.
ta_1138 2y ago
No doubt, but there's a whole lot of problems shaped like that in tech today. Look at any two sided market: Amazon/Etsy's third party sellers also want to be at the top, and there are bad actors trying to scam the intermediary and the customer. The same low friction onboarding that allows the company to succeed as an intermediary is also facilitating the fraud. Merchants can be fraudsters, advertisers can be fraudsters, users can be fraudsters producing fake clicks. There's economic incentives everywhere. See the relatively recent news of spotify getting generated songs, which are then played by fake listeners. The companies are entering this situations with their eyes wide open. It's a necessary problem when you want to be an intermediary at a large enough scale. And if you don't have the scale, the per-interaction costs are really high, and your company gets beaten by someone with an easier onboarding funnel.
jldugger 2y ago
>It is a cursed problem no matter how you look at it. But the funding model is also extremely blessed: you have a stream of people telling you every page load what they're interested in at that exact moment, and it's not hard to match them up with advertisers for those keywords. And arguably the SEO parasitism problem is worsened by their monopoly -- to the extent that optimizing a site for one engine deoptimizes it on on others, the stronger the lead engine #1 has the more incentive there is to game it.

duskwuff 2y ago

It's only getting harder and harder for newcomers to compete. Especially with the rise of abusive AI scrapers, webmasters are increasingly hostile to new web crawlers. Google and Bing get a pass because they're well-known, but any new service is fighting an uphill battle; a lot of sites will block them right off the bat. Nor does it help that some large sites (like Reddit) have inside deals with Google to feed them updates directly.