5 ms·

we've decided we can't be bothered with traditional search Traditional search (at least on the web) is dying. The entire edifice is drowning under a rapidly ri

we've decided we can't be bothered with traditional search Traditional search (at least on the web) is dying. The entire edifice is drowning under a rapidly rising tide of spam and scam sites. No one, including Google, knows what to do about it so we're punting on the whole project and hoping AI will swoop in like deus ex machina and save the day.

romwell 2y ago
Narrator: it did not, in fact, save the day.
petre 2y ago
AI will generate even more spam and scam sites more trivially.
- ses1984 2y ago
  What do you mean “will”, we are a few years past that point.
AnimalMuppet 2y ago
But it can't save the day. The problem with Google search is that it indexes all the web, and there's (as you say) a rising tide of scam and spam sites. The problem with AI is that it scoops up all the web as training data, and there's a rising tide of scam and spam sites.
lokar 2y ago
It took the scam/spam sites a few years to catch up to Google search. Just wait a bit, equilibrium will return.
akoboldfrying 2y ago
>The entire edifice is drowning under a rapidly rising tide of spam and scam sites. You make this claim with such confidence, but what is it based on? There have always been hordes of spam and scam websites. Can you point to anything that actually indicates that the ratio is now getting worse?
- chongli 2y ago
  There have always been hordes of spam and scam websites. Can you point to anything that actually indicates that the ratio is now getting worse? No, there haven't always been hordes of spam and scam websites. I remember the web of the 90s. When Google first arrived on the scene every site on the results page was a real site, not a spam/scam site.
  - ShroudedNight 2y ago
    That was PageRank flexing its capability. There were lots of sites with reams of honeypot text that caught the other search engines.
masfuerte 2y ago
Google results are not polluted with spam because Google doesn't know how to deal with it. Google results are polluted with spam because it is more profitable for Google. This is a conscious decision they made five years ago.
- chongli 2y ago
  because it is more profitable for Google Then why are DuckDuckGo results also (arguably even more so) polluted with spam/scam sites? I doubt DDG is making any profit from those sites since Google essentially owns the display ad business.
  - JohnDone 2y ago
    Ddg is actually Bing. Search as a service.
    - djvuvtgcuehb 2y ago
      And Bing is google.
- redwall_hp 2y ago
  If you own the largest ad network that spam sites use and own the traffic firehose, pointing the hose at the spam sites and ensuring people spend more time clicking multiple results that point to ad-filled sites will make you more money. Google not only has multiple monopolies, but a cut and dry perverse incentive to produce lower quality results to make the whole session longer instead of short and effective.
photonthug 2y ago
Maybe it is naive but I think search would probably work again if they could roll back code to 10 or 15 years ago and just make search engines look for text in webpages. Google wasn’t crushed by spam, they decided to stop doing text search and build search bubbles that are user specific, location-specific, decided to surface pages that mention search terms in metadata instead of in text users might read, etc. Oh yeah, and about a decade before LLMs were actually usable, they started to sabotage simple substring searches and kind of force this more conversational interface. That’s when simple search terms stopped working very well, and you had to instead ask yourself “hmm how would a very old person or a small child phrase this question for a magic oracle” This is how we get stuff like: Did you mean “when did Shakespeare die near my location”? If anyone at google cared more about quality than printing money, that thirsty gambit would at least be at the bottom of the page instead of the top.
- layer8 2y ago
  > just make search engines look for text in webpages. Google’s verbatim search option roughly does that for me (plus an ad blocker that removes ads from the results page). I have it activated by default as a search shortcut. (To activate it, one can add “tbs=li:1” as a query parameter to the Google search URL.)
  - alex1138 2y ago
    To me the stupidest thing was the removal of things like + and -. You can say it's because of Google+ but annoyingly duckduckgo also doesn't seem to honor it. Kagi seems to and I hope they don't follow the others down the road of stupid
  - tru3_power 2y ago
    Wow what? Thanks!
  - jcgrillo 2y ago
    > ?tbs=li:1 Thank you, this is almost life-alteringly good to know.
skissane 2y ago
I personally think a big problem with search is major search engines try to be all things to all people and hence suffer as a result. For example: a beginner developer is possibly better served by some SEO-heavy tutorial blog post; an experienced developer would prefer results weighted towards the official docs, the project’s bug tracker and mailing list, etc. But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later. One positive about AI: if an AI is doing the search, it likely wants the more advanced material not the more beginner-focused one. It can take more advanced material and simplify it for the benefit of less experienced users. It is (I suspect) less likely to make mistakes if you ask it to simplify the more advanced material than if you just gave it more beginner-oriented material instead. So if AI starts to replace humans as the main clients of search, that may reverse some of the pressure to “dumb it down”.
- photonthug 2y ago
  > But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later. I mostly agree with your interesting comment, and I think your analysis basically jives with my sibling comment. But one thing I take issue with is the idea that this type of thing is a good faith effort, because it’s more like a convenient excuse. Explaining substring search or even include/exclude ops to children and grandparents is actually easy. Setting preferences for tutorials vs API docs would also be easy. But companies don’t really want user-directed behavior as much as they want to herd users to preferred content with algorithms, then convince the user it was their idea or at least the result of relatively static ranking processes. The push towards more fuzzy semantic search and “related content” everywhere is not to cater to novice users but to blur the line between paid advertisement and organic user-directed discovery. No need to give megacorp the benefit of the doubt on stuff like this, or make the underlying problems seem harder than they are. All platforms land in this place by convergent evolution wherein the driving forces are money and influence, not insurmountable technical difficulties or good intentions for usability.
- consp 2y ago
cyanydeez 2y ago
If only google was trying to solve search rather than shareholdet values.
skydhash 2y ago
> Traditional search (at least on the web) is dying. That's not my experience at all. While there are scammy sites, using the search engines as an index instead of an oracle still yields useful results. It only requires to learn the keywords which you can do by reading the relevant materials .
- chongli 2y ago
  How do you read the relevant materials if you haven’t found them yet? It’s a chicken and egg problem. If your goal is to become an expert in a subject but you’re currently a novice, search can’t help you if it’s only giving you terrible results until you “crack the code.”
ponector 2y ago
>> No one, including Google, knows what to do about it I'm sure they can. But they have no incentive. Try to Google an item, and it will show you a perfect match of sponsored ads and some other not-so-relevant non-sponsored results
AtlasBarfed 2y ago
There's no way the search AI will beat out the spamgen AI. Tailoring/retraining the main search AI will be so much more expensive that retraining the spam special purpose AIs.
quickthrowman 2y ago
Google could fix the problem if they wanted to, but it’s not in their interests to fix it since the spam sites generally buy ads from Google and/or display Google ads on their spam websites. Google wants to maximize their income, so..
layer8 2y ago
Without a usable web search index, AI will be in trouble eventually as well. There is no substitute for it.
rubyfan 2y ago
AI will make the problem of low quality, fake, fraudulent and arbitrage content way worse. I highly doubt it will improve searching for quality content at all.
deleted 2y ago
[deleted]
fmos 2y ago
Kagi has fixed traditional search for me.

romwell 2y ago

Narrator: it did not, in fact, save the day.

petre 2y ago

AI will generate even more spam and scam sites more trivially.

ses1984 2y ago
What do you mean “will”, we are a few years past that point.

AnimalMuppet 2y ago

But it can't save the day. The problem with Google search is that it indexes all the web, and there's (as you say) a rising tide of scam and spam sites. The problem with AI is that it scoops up all the web as training data, and there's a rising tide of scam and spam sites.

lokar 2y ago

It took the scam/spam sites a few years to catch up to Google search. Just wait a bit, equilibrium will return.

akoboldfrying 2y ago

>The entire edifice is drowning under a rapidly rising tide of spam and scam sites. You make this claim with such confidence, but what is it based on? There have always been hordes of spam and scam websites. Can you point to anything that actually indicates that the ratio is now getting worse?

chongli 2y ago
There have always been hordes of spam and scam websites. Can you point to anything that actually indicates that the ratio is now getting worse? No, there haven't always been hordes of spam and scam websites. I remember the web of the 90s. When Google first arrived on the scene every site on the results page was a real site, not a spam/scam site.
- ShroudedNight 2y ago
  That was PageRank flexing its capability. There were lots of sites with reams of honeypot text that caught the other search engines.

masfuerte 2y ago

Google results are not polluted with spam because Google doesn't know how to deal with it. Google results are polluted with spam because it is more profitable for Google. This is a conscious decision they made five years ago.

chongli 2y ago
because it is more profitable for Google Then why are DuckDuckGo results also (arguably even more so) polluted with spam/scam sites? I doubt DDG is making any profit from those sites since Google essentially owns the display ad business.
- JohnDone 2y ago
  Ddg is actually Bing. Search as a service.
  - djvuvtgcuehb 2y ago
    And Bing is google.
redwall_hp 2y ago
If you own the largest ad network that spam sites use and own the traffic firehose, pointing the hose at the spam sites and ensuring people spend more time clicking multiple results that point to ad-filled sites will make you more money. Google not only has multiple monopolies, but a cut and dry perverse incentive to produce lower quality results to make the whole session longer instead of short and effective.

photonthug 2y ago

Maybe it is naive but I think search would probably work again if they could roll back code to 10 or 15 years ago and just make search engines look for text in webpages. Google wasn’t crushed by spam, they decided to stop doing text search and build search bubbles that are user specific, location-specific, decided to surface pages that mention search terms in metadata instead of in text users might read, etc. Oh yeah, and about a decade before LLMs were actually usable, they started to sabotage simple substring searches and kind of force this more conversational interface. That’s when simple search terms stopped working very well, and you had to instead ask yourself “hmm how would a very old person or a small child phrase this question for a magic oracle” This is how we get stuff like: Did you mean “when did Shakespeare die near my location”? If anyone at google cared more about quality than printing money, that thirsty gambit would at least be at the bottom of the page instead of the top.

layer8 2y ago
> just make search engines look for text in webpages. Google’s verbatim search option roughly does that for me (plus an ad blocker that removes ads from the results page). I have it activated by default as a search shortcut. (To activate it, one can add “tbs=li:1” as a query parameter to the Google search URL.)
- alex1138 2y ago
  To me the stupidest thing was the removal of things like + and -. You can say it's because of Google+ but annoyingly duckduckgo also doesn't seem to honor it. Kagi seems to and I hope they don't follow the others down the road of stupid
- tru3_power 2y ago
  Wow what? Thanks!
- jcgrillo 2y ago
  > ?tbs=li:1 Thank you, this is almost life-alteringly good to know.

skissane 2y ago

I personally think a big problem with search is major search engines try to be all things to all people and hence suffer as a result. For example: a beginner developer is possibly better served by some SEO-heavy tutorial blog post; an experienced developer would prefer results weighted towards the official docs, the project’s bug tracker and mailing list, etc. But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later. One positive about AI: if an AI is doing the search, it likely wants the more advanced material not the more beginner-focused one. It can take more advanced material and simplify it for the benefit of less experienced users. It is (I suspect) less likely to make mistakes if you ask it to simplify the more advanced material than if you just gave it more beginner-oriented material instead. So if AI starts to replace humans as the main clients of search, that may reverse some of the pressure to “dumb it down”.

photonthug 2y ago
> But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later. I mostly agree with your interesting comment, and I think your analysis basically jives with my sibling comment. But one thing I take issue with is the idea that this type of thing is a good faith effort, because it’s more like a convenient excuse. Explaining substring search or even include/exclude ops to children and grandparents is actually easy. Setting preferences for tutorials vs API docs would also be easy. But companies don’t really want user-directed behavior as much as they want to herd users to preferred content with algorithms, then convince the user it was their idea or at least the result of relatively static ranking processes. The push towards more fuzzy semantic search and “related content” everywhere is not to cater to novice users but to blur the line between paid advertisement and organic user-directed discovery. No need to give megacorp the benefit of the doubt on stuff like this, or make the underlying problems seem harder than they are. All platforms land in this place by convergent evolution wherein the driving forces are money and influence, not insurmountable technical difficulties or good intentions for usability.
consp 2y ago

cyanydeez 2y ago

If only google was trying to solve search rather than shareholdet values.

skydhash 2y ago

> Traditional search (at least on the web) is dying. That's not my experience at all. While there are scammy sites, using the search engines as an index instead of an oracle still yields useful results. It only requires to learn the keywords which you can do by reading the relevant materials .

chongli 2y ago
How do you read the relevant materials if you haven’t found them yet? It’s a chicken and egg problem. If your goal is to become an expert in a subject but you’re currently a novice, search can’t help you if it’s only giving you terrible results until you “crack the code.”

ponector 2y ago

>> No one, including Google, knows what to do about it I'm sure they can. But they have no incentive. Try to Google an item, and it will show you a perfect match of sponsored ads and some other not-so-relevant non-sponsored results

AtlasBarfed 2y ago

There's no way the search AI will beat out the spamgen AI. Tailoring/retraining the main search AI will be so much more expensive that retraining the spam special purpose AIs.

quickthrowman 2y ago

Google could fix the problem if they wanted to, but it’s not in their interests to fix it since the spam sites generally buy ads from Google and/or display Google ads on their spam websites. Google wants to maximize their income, so..

layer8 2y ago

Without a usable web search index, AI will be in trouble eventually as well. There is no substitute for it.

rubyfan 2y ago

AI will make the problem of low quality, fake, fraudulent and arbitrage content way worse. I highly doubt it will improve searching for quality content at all.

deleted 2y ago

[deleted]

fmos 2y ago

Kagi has fixed traditional search for me.