11 ms·

My theory is that Google lost the spam war. All the leetcode in the world won't teach someone to build an ML model that can tell whether a website is spam. So G

My theory is that Google lost the spam war. All the leetcode in the world won't teach someone to build an ML model that can tell whether a website is spam. So Google outsourced the spam detection problem -- they heavily bias their results towards only the most popular sites, who either have human moderators or paid contributors, and those sites do the spam management that Google's automated approach is incapable of.

xnx 3y ago
Expecting Google's code to flawlessly understand the truth and trustworthiness of all possible sources for every conceivable question is a fair/unbiased way for all topics past present and future is a very high bar. This is a really hard problem, and unlike many other hard problems (rocket science, neurosurgery, particle physics, etc.), Google has millions of financially motivated adversaries actively trying to trick them.
diegocg 3y ago
It is more than the spam war...a lot of content is just no longer produced and exposed to the open web (think about how much content goes into tiktok, discord, etc and you will never get that into your search results). Google has less useful content to index, algorithms can't fix that. There is more spam only because that's the only open content that gets added massively. The winners of this battle will be the places where content is generated (or curated) - and reddit is perhaps the most important content hub generator (it's not just an aggregator anymore - comments about some news can be often more interesting than the news itself). Indexers (and language models) are useless without content to scrap.
- eugenekolo 3y ago
  I can't believe that. There's still so many personal blogs by real people. They just don't show up in the first 3 pages unless you query for them very specifically. Seems most queries result in something like 30% SEO spam sites, 30% quora, 30% reddit, 10% other. Edit: I don't disagree that Discord/Youtube/Other closed gardens have taken open searchable data away, but it's not like there's now no authentic searchable data at all. Perhaps Google also needs to learn to search those closed gardens better.
  - bombcar 3y ago
    Google flourished because it could find forums (and blogs) and mine those, but much of that content has disappeared into Facebook and Discord (and YouTube - we must not discount how many things that would have been easily parseable blogs are now buried in livestreams and videos).
    - drowsspa 3y ago
      Discord is probably the worst of all. I'm not a gamer and I hate it so much that a lot of tech content is now locked behind private Discord channels. Even Facebook is more discoverable than that
      - thatguy0900 3y ago
        Even when you are already on discord, search and trying to read old conversations is awful on discord, because that's not at all what discord was made to do.
    - Tostino 3y ago
      So I've been working on a side project to make a Youtube channel I watch have its content be more discoverable through text. I've had great results by scraping the Youtube transcription, and running that for a few passes through GPT 3.5 with some prompts to essentially act as an editor. The original transcription was often terrible in some spots. Just whole phrases or multiple words mistranscribed throughout. For almost all of them, GPT 3.5 was able to clean them up and restore the original meaning through understanding the context of the monolog and fixing obviously incorrect words or phrases. I've watched through a sample of about 20 of the 3,000 videos I'm working through, and the corrected transcription really did an amazing job at restoring the original meaning from the spoken words that was hard to understand from the original machine transcription.
      - pneumonic 3y ago
        That is exactly where LLMs are useful. (People thinking of them as "AI", meaning AGI is just so wrong. Writing legal briefs??) Using them to ex post facto adjust transcripts in order to make them available and searchable is great.
- reaperducer 3y ago
  a lot of content is just no longer produced and exposed to the open web (think about how much content goes into tiktok, discord, etc and you will never get that into your search results) I see this all the time when trying to find information about old computers. So many of the good vintage computing resources are locked in social services or mailing lists that the information never shows up in search engines. It feels a lot like the days when information was balkanized between AOL, GEnie, CompuServe, American PeopleLink, Delphi, etc. Search engines were supposed to fix that and make all the world's information discoverable. They didn't.
  - noknownsender 3y ago
    I've experienced this a lot myself. It's weird. I used to be considered a Digital Native. Now I feel like an outsider on the internet.
- never_inline 3y ago
  There certainly is content. Often the ones I could find two years ago but now cannot. That's because the web is full of juvenile sub-normie content such as geeksforgeeks (if you consider programming topics for example). It shadows the very specific queries with highly SEO'd Juvenile stuff.
falcolas 3y ago
Google's recency bias in search results hurts them too. There are many older resources which are still valid out there, but you won't find them (easily) via Google Search. Instead, you get the SEO spam which doesn't match the search as well, but is newer.
- bombcar 3y ago
  This is a huge, huge, huge problem and Google I feel could work to solve this. NEWS (heh it contains new) is important to be up-to-date and recent. But so many things are olds and you should give more weight to older pagerank-verified pages.
  - waboremo 3y ago
    All of the Google sections seem to be terrible for their use case. Images are lacking filters and are filled with Pinterest results which send people into a cycle when trying to get to the actual image to save. News will sometimes show old news when it's obvious there should be new information about whatever you're searching, leading you to have to specifically indicate sort by date. Videos will prioritize certain domains even if the video itself is irrelevant to your search (ie searching up an actress, it'll show an IMDB video first, even if the video is very old and is just a generic trailer for some movie they were in ages ago). If you search something relatively generic, the new search bubbles will now hide the other categories (ie searching adele will hide categories like shopping or books). The finance option literally just redirects you to Google Finance now, doesn't even retain your search. They've really let the core search experience be deteriorated so extensively, that we can't blame all of it on SEO.
    - infoseek12 3y ago
      I haven’t given it much thought until just now but it really is surprising how terrible Googles image search is. There are so many images on the Internet yet the image search often fails to retrieve good examples even on straightforward searches. Never mind difficult searches, higher resolution images, or if you want to quickly download an image. It was never very good and it’s gotten perceptible worse.
deeviant 3y ago
The real problem isn't that Google can't detect what spam is, it's that this spam drives so much of the traffic, and thus their revenue that they cannot remove it. Google at this point is literally teaming up with spammers. Just go on a android phone and swipe left from homepage for "Google", count how many clicks it takes to get to an 100% AI generated spam page.
- dylan604 3y ago
  for those of us without an android device to try this, care to enlighten us? the way it is described it sounds like a lot of of action required which seems like a good thing, but your point seems like it's trying to make goog's results bad. i'm really not following your point
  - shigawire 3y ago
    This shows a bunch of articles based on your search history - many of which are incredibly poor quality and appear to be generated in some way. I tried to curate mine for a bit and thumbs down useless sites or content I don't care about. I didn't really succeed in making this feed helpful so I mostly ignore it.
    - balder1991 3y ago
      I thought it was only me who thought that. Those articles are so much worse than, say, Google News app. Once in a while there’s something relevant, but it’s always clickbait stuff.
    - TRiG_Ireland 3y ago
      Google appears to think that I have one sole interest, which is The Lord of the Rings. And that I want to read endless poorly researched and sensationally headlined articles on differences between the novel and the films. It never recommends me anything else in that section.
- bobthepanda 3y ago
  We have all those posts kvetching about how small email providers have become collateral damage in the arms race against email spam. I don’t think anyone is truly successful in that space. The issue is that ML is very expensive compared to spam generation. And the moment another search engine becomes popular, then the spam cannons will be used against it instead.
eastof 3y ago
Maybe the ML model is predicting that popular curated sites generally have less spam?
s1artibartfast 3y ago
I wonder if Google wants to prioritize video to try to address this spam problem. Sometimes when I search a topic all I get is video in the natural results, not a video search, and I suspect I'm part of an A versus B test. I personally don't find it to be a compelling solution and click away because I'm not willing to sift through 10 10 minute videos to see if there is relevant content.
s1artibartfast 3y ago
Google won the spam War, they were just fighting for the opposite side. I know it's trite but it bears repeating: Google only cares about user satisfaction if it positively impacts their bottom line. As long as they can monetize remaining users more than what they lose from users abandoning search, the trend will continue
TX81Z 3y ago
[flagged]
bamfly 3y ago
For sure. They threw in the towel around '08 or '09. The prior years of yo-yoing between front page results containing ~60% webspam and 0% webspam settled into a permanent 60%, and they evidently just heavily downranked and/or started ignoring low-traffic sites to keep it from growing to 100%. Their shifting from text-only "ethical" ads to being a more-ordinary web ad service, and putting ads inline with results, roughly corresponds with that, IIRC, which probably isn't a coincidence. Your ad-"results" probably get more clicks if much of the rest of first result page is crap. Many of the webspammers probably funnel money to Google one way or another, now. It screwed up their incentives to keep fighting that battle, I'd guess, which is likely part of why they stopped putting so much effort into it.
Guest9081239812 3y ago
I think this causes further problems as well. These big companies know they can easily be listed at the top of Google, and therefore they pump out low quality articles for every popular key phrase. They can write a "Top 10 X of 2023" list and bring in lots of traffic and referral sales, while a smaller site has no chance of doing the same. Then, the big site takes all that income, and pumps out more low quality articles. It's a rich get richer scenario. If I search, "The top 10 movies on Netflix", I get sent to this page... https://decider.com/list/top-10-movies-on-netflix-most-popular-right-now/ https://decider.com/list/top-10-movies-on-netflix-most-popul... The top 5 movies above are... 1. The Boss Baby (2017) 2. The Angry Birds Movie (2016) 3. We're the Millers (2013) 4. Zookeeper (2011) 5. Mean Girls (2004) Is this really the best the world has to offer for my search query? I could spend a week putting together a far better page for that query, but what's the point? It's not going to rank anywhere notable in the Google results. There's no incentive for me to produce that quality content.
pnathan 3y ago
I do agree that Google lost the spam war. This is particularly notable when looking for recipes.
deleted 3y ago
[deleted]
choppaface 3y ago
Google lost the spam war when Matt Cutts left. Google effectively lost its community outreach, and simultaneously pivoted to aggressive moat protection through AMP, Chrome, Jedi Blue, etc. It’s less that the spammers won and more that Google lost their aim at high-quality content.
durumu 3y ago
I mean I think you’re mostly right but to be fair to Google it’s an incredibly adversarial. Thousands of people have made it their full time job to try to trick Google into thinking their blogspam is good content.

xnx 3y ago

Expecting Google's code to flawlessly understand the truth and trustworthiness of all possible sources for every conceivable question is a fair/unbiased way for all topics past present and future is a very high bar. This is a really hard problem, and unlike many other hard problems (rocket science, neurosurgery, particle physics, etc.), Google has millions of financially motivated adversaries actively trying to trick them.

diegocg 3y ago

It is more than the spam war...a lot of content is just no longer produced and exposed to the open web (think about how much content goes into tiktok, discord, etc and you will never get that into your search results). Google has less useful content to index, algorithms can't fix that. There is more spam only because that's the only open content that gets added massively. The winners of this battle will be the places where content is generated (or curated) - and reddit is perhaps the most important content hub generator (it's not just an aggregator anymore - comments about some news can be often more interesting than the news itself). Indexers (and language models) are useless without content to scrap.

eugenekolo 3y ago
I can't believe that. There's still so many personal blogs by real people. They just don't show up in the first 3 pages unless you query for them very specifically. Seems most queries result in something like 30% SEO spam sites, 30% quora, 30% reddit, 10% other. Edit: I don't disagree that Discord/Youtube/Other closed gardens have taken open searchable data away, but it's not like there's now no authentic searchable data at all. Perhaps Google also needs to learn to search those closed gardens better.
- bombcar 3y ago
  Google flourished because it could find forums (and blogs) and mine those, but much of that content has disappeared into Facebook and Discord (and YouTube - we must not discount how many things that would have been easily parseable blogs are now buried in livestreams and videos).
  - drowsspa 3y ago
    Discord is probably the worst of all. I'm not a gamer and I hate it so much that a lot of tech content is now locked behind private Discord channels. Even Facebook is more discoverable than that
    - thatguy0900 3y ago
      Even when you are already on discord, search and trying to read old conversations is awful on discord, because that's not at all what discord was made to do.
  - Tostino 3y ago
    So I've been working on a side project to make a Youtube channel I watch have its content be more discoverable through text. I've had great results by scraping the Youtube transcription, and running that for a few passes through GPT 3.5 with some prompts to essentially act as an editor. The original transcription was often terrible in some spots. Just whole phrases or multiple words mistranscribed throughout. For almost all of them, GPT 3.5 was able to clean them up and restore the original meaning through understanding the context of the monolog and fixing obviously incorrect words or phrases. I've watched through a sample of about 20 of the 3,000 videos I'm working through, and the corrected transcription really did an amazing job at restoring the original meaning from the spoken words that was hard to understand from the original machine transcription.
    - pneumonic 3y ago
      That is exactly where LLMs are useful. (People thinking of them as "AI", meaning AGI is just so wrong. Writing legal briefs??) Using them to ex post facto adjust transcripts in order to make them available and searchable is great.
reaperducer 3y ago
a lot of content is just no longer produced and exposed to the open web (think about how much content goes into tiktok, discord, etc and you will never get that into your search results) I see this all the time when trying to find information about old computers. So many of the good vintage computing resources are locked in social services or mailing lists that the information never shows up in search engines. It feels a lot like the days when information was balkanized between AOL, GEnie, CompuServe, American PeopleLink, Delphi, etc. Search engines were supposed to fix that and make all the world's information discoverable. They didn't.
- noknownsender 3y ago
  I've experienced this a lot myself. It's weird. I used to be considered a Digital Native. Now I feel like an outsider on the internet.
never_inline 3y ago
There certainly is content. Often the ones I could find two years ago but now cannot. That's because the web is full of juvenile sub-normie content such as geeksforgeeks (if you consider programming topics for example). It shadows the very specific queries with highly SEO'd Juvenile stuff.

falcolas 3y ago

Google's recency bias in search results hurts them too. There are many older resources which are still valid out there, but you won't find them (easily) via Google Search. Instead, you get the SEO spam which doesn't match the search as well, but is newer.

bombcar 3y ago
This is a huge, huge, huge problem and Google I feel could work to solve this. NEWS (heh it contains new) is important to be up-to-date and recent. But so many things are olds and you should give more weight to older pagerank-verified pages.
- waboremo 3y ago
  All of the Google sections seem to be terrible for their use case. Images are lacking filters and are filled with Pinterest results which send people into a cycle when trying to get to the actual image to save. News will sometimes show old news when it's obvious there should be new information about whatever you're searching, leading you to have to specifically indicate sort by date. Videos will prioritize certain domains even if the video itself is irrelevant to your search (ie searching up an actress, it'll show an IMDB video first, even if the video is very old and is just a generic trailer for some movie they were in ages ago). If you search something relatively generic, the new search bubbles will now hide the other categories (ie searching adele will hide categories like shopping or books). The finance option literally just redirects you to Google Finance now, doesn't even retain your search. They've really let the core search experience be deteriorated so extensively, that we can't blame all of it on SEO.
  - infoseek12 3y ago
    I haven’t given it much thought until just now but it really is surprising how terrible Googles image search is. There are so many images on the Internet yet the image search often fails to retrieve good examples even on straightforward searches. Never mind difficult searches, higher resolution images, or if you want to quickly download an image. It was never very good and it’s gotten perceptible worse.

deeviant 3y ago

The real problem isn't that Google can't detect what spam is, it's that this spam drives so much of the traffic, and thus their revenue that they cannot remove it. Google at this point is literally teaming up with spammers. Just go on a android phone and swipe left from homepage for "Google", count how many clicks it takes to get to an 100% AI generated spam page.

dylan604 3y ago
for those of us without an android device to try this, care to enlighten us? the way it is described it sounds like a lot of of action required which seems like a good thing, but your point seems like it's trying to make goog's results bad. i'm really not following your point
- shigawire 3y ago
  This shows a bunch of articles based on your search history - many of which are incredibly poor quality and appear to be generated in some way. I tried to curate mine for a bit and thumbs down useless sites or content I don't care about. I didn't really succeed in making this feed helpful so I mostly ignore it.
  - balder1991 3y ago
    I thought it was only me who thought that. Those articles are so much worse than, say, Google News app. Once in a while there’s something relevant, but it’s always clickbait stuff.
  - TRiG_Ireland 3y ago
    Google appears to think that I have one sole interest, which is The Lord of the Rings. And that I want to read endless poorly researched and sensationally headlined articles on differences between the novel and the films. It never recommends me anything else in that section.
bobthepanda 3y ago
We have all those posts kvetching about how small email providers have become collateral damage in the arms race against email spam. I don’t think anyone is truly successful in that space. The issue is that ML is very expensive compared to spam generation. And the moment another search engine becomes popular, then the spam cannons will be used against it instead.

eastof 3y ago

Maybe the ML model is predicting that popular curated sites generally have less spam?

s1artibartfast 3y ago

I wonder if Google wants to prioritize video to try to address this spam problem. Sometimes when I search a topic all I get is video in the natural results, not a video search, and I suspect I'm part of an A versus B test. I personally don't find it to be a compelling solution and click away because I'm not willing to sift through 10 10 minute videos to see if there is relevant content.

s1artibartfast 3y ago

Google won the spam War, they were just fighting for the opposite side. I know it's trite but it bears repeating: Google only cares about user satisfaction if it positively impacts their bottom line. As long as they can monetize remaining users more than what they lose from users abandoning search, the trend will continue

TX81Z 3y ago

[flagged]

bamfly 3y ago

For sure. They threw in the towel around '08 or '09. The prior years of yo-yoing between front page results containing ~60% webspam and 0% webspam settled into a permanent 60%, and they evidently just heavily downranked and/or started ignoring low-traffic sites to keep it from growing to 100%. Their shifting from text-only "ethical" ads to being a more-ordinary web ad service, and putting ads inline with results, roughly corresponds with that, IIRC, which probably isn't a coincidence. Your ad-"results" probably get more clicks if much of the rest of first result page is crap. Many of the webspammers probably funnel money to Google one way or another, now. It screwed up their incentives to keep fighting that battle, I'd guess, which is likely part of why they stopped putting so much effort into it.

Guest9081239812 3y ago

I think this causes further problems as well. These big companies know they can easily be listed at the top of Google, and therefore they pump out low quality articles for every popular key phrase. They can write a "Top 10 X of 2023" list and bring in lots of traffic and referral sales, while a smaller site has no chance of doing the same. Then, the big site takes all that income, and pumps out more low quality articles. It's a rich get richer scenario. If I search, "The top 10 movies on Netflix", I get sent to this page... https://decider.com/list/top-10-movies-on-netflix-most-popular-right-now/ https://decider.com/list/top-10-movies-on-netflix-most-popul... The top 5 movies above are... 1. The Boss Baby (2017) 2. The Angry Birds Movie (2016) 3. We're the Millers (2013) 4. Zookeeper (2011) 5. Mean Girls (2004) Is this really the best the world has to offer for my search query? I could spend a week putting together a far better page for that query, but what's the point? It's not going to rank anywhere notable in the Google results. There's no incentive for me to produce that quality content.

pnathan 3y ago

I do agree that Google lost the spam war. This is particularly notable when looking for recipes.

deleted 3y ago

[deleted]

choppaface 3y ago

Google lost the spam war when Matt Cutts left. Google effectively lost its community outreach, and simultaneously pivoted to aggressive moat protection through AMP, Chrome, Jedi Blue, etc. It’s less that the spammers won and more that Google lost their aim at high-quality content.

durumu 3y ago

I mean I think you’re mostly right but to be fair to Google it’s an incredibly adversarial. Thousands of people have made it their full time job to try to trick Google into thinking their blogspam is good content.