8 ms·

I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it. SEO spam has been strangling the web for years. Now, genAI SEO

I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it. SEO spam has been strangling the web for years. Now, genAI SEO spam has escalated that onto inhuman levels. To make matters worse, no one wants to post to the open web anymore because their posts are just going to drown in that sea of spam and only genai's data stealing bots will read them. As the amount of spam posted to the web increases, the amount of worthwhile content posted decreases. Eventually, nothing of value will be posted. (like facebook?) You can lay the blame for the web's death squarely at Google's feet for allowing SEO to hijack search in the first place (or maybe the government is to blame for not breaking up Google's ad/search empire fast enough). Either way, the big companies all know the end is here and are gambling on genai to replace search. Already, places of knowledge are closing their borders and charging fees for genai to access. We have entered the internet's dark age. Fun aside: I think it's hilarious and fitting that Google's genai model sucks. And I hope they lose the genai wars (just out of spite, not because I think any other genai is worth a shit).

amatheus 2y ago
I think there may be a way forward for the internet. Why isn't hacker news being strangled by SEO spam for example? I think the way forward is a much smaller internet predicated on tight-knit communities approving everything that's shared. I don't know how this could scale but maybe that's the point, it must stay small to succeed.
knadh 2y ago
> I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it. I concur. Maybe now is the time to seriously think about alternate visions for "search". I have been toying with an idea[1] along those lines myself out of sheer annoyance at the state of WWW and web "search" in general. [1] https://nadh.in/blog/decentralised-open-indexes/ https://nadh.in/blog/decentralised-open-indexes/
n_ary 2y ago
I do not blame Google for web’s current wasteland landscape, I blame ads and all the grifters who poison the landscape with SEO waste to try make some bucks with ads or other similar means.
- tourmalinetaco 2y ago
  Google is an ad company that specifically lets SEO fill up its search results because those websites make them the most money per click. If Google continued to fight SEO abuse like they were literally founded to do then we wouldn’t be in this mess.
- deleted 2y ago
  [deleted]
- ainiriand 2y ago
  It really is Google's fault. Those blogs SEO goals are only targeting top Google positions.
- rkharsan64 2y ago
  There's several examples where Google's SEO has actively made things worse, even before you look at ads. And to be clear, I do agree that advertising has also done massive damage to the web. All the long, rambling stories you see before recipes? That's just there so the recipes can rank higher. I can't remember the source, but I also remember a website adding AI generated slop to cater to Google's wishes, and leaving a page to explain to normal users why they did so. It's also seen when you look at YouTube videos, where now it's basically necessary to have a clickbait thumbnail + title + reminders to like and subscribe.
RavlaAlvar 2y ago
Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening?
- lelanthran 2y ago
  > Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening? I've said this before (on HN) and I'll say it again: a search engine that refuses to index sites containing advertisements absolutely kills off SEO. What's the point of getting to the top of the search results if you are unable to monetise it? The only way forward is to refuse to index sites with advertisements. Google will obviously not do this (and, in fact, to me it looks like they do the reverse - downrank non-monetised content - because it's in their best interest to serve sites with ads).
  - noiwillnot 2y ago
    > What's the point of getting to the top of the search results if you are unable to monetise it? Promoting your product/service, see brand astroturfing in Reddit.
    - lelanthran 2y ago
      > Promoting your product/service, see brand astroturfing in Reddit. But ... that's the point, is it not? Someone searching for "FOO" will find all those sites who optimised for selling FOO. That's an improvement over finding the top 10 results all optimised on FOO but delivering ads to make money, not FOO.
- throwup238 2y ago
  The same way people make adblockers happen. By making block lists of SEO garbage cruated by human beings. Nextdns and the RPi alternative do it. Kagi has all the infrastructure in place to make it happen at the search level, it just requires more manual work right now.
dazc 2y ago
You cite Facebook as an example of no meaningful content yet put all the blame upon Google? Maybe content creators are just incentivised into giving 90 percent of the public what they want.
- ainiriand 2y ago
  The content added to Facebook was not what made the web what it was. The content indexed by Google, in turn, it was. The countless blogs or cooking recipes, fan sites, etc, were overriden with SEO filler slop over the years. OP example of what a content-devoid internet would look like is exactly what Facebook is now. A site that started as the place to connect and share with your friends and family now is just a place that is filled with AI slop and emptiness.
- bryanrasmussen 2y ago
  I figure if 90% of everything is crap, https://medium.com/luminasticity/90-crap-48e4c79419a9 https://medium.com/luminasticity/90-crap-48e4c79419a9 then if the amount of available content has doubled in a decade, that is probably just too much crap to find the good stuff.
openrisk 2y ago
> I think the web is dead The web cannot 'die'. Everything and everyone moves online. Point one: A return to pre-internet days is not likely. Point two: Walled gardens had their peak moment. The dangers of everybody relying on them for our entire online existence are more than apparent to anybody with a firing neuron (and despite appearances, our species is not entirely inane). Ergo. The only path is forward, a more sane, less monopolized web. Build it and they will come.
- axegon_ 2y ago
  The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet. Just look at stackoverflow and github. Ever since chatgpt was released, the contributions skyrocketed and the quality dropped off the face of the earth and into a dark oblivion.
  - openrisk 2y ago
    what do you mean by "the web"? The web is all the people, companies, public sector etc. that are setting up web servers (whether self-hosted, cloud etc. it doesnt matter). To say that the web is dead means that nobody outside massive social media type sites will still run a server (and SO / github are social media type walled gardens at this point, thats why they get enshittified). But this development would be absurd as a whole. Mind you search engines as we have come to know them might be dead, but thats of their own making. People need to find / communicate useful, truthful information and sooner or later they will get it.
  - nextaccountic 2y ago
    We are talking on a website on the web. Not everything needs to go through a search engine.
  - bryanrasmussen 2y ago
    >The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet. this presupposes the argument that for the web to have value then you must have a search engine to allow you to find things of worth on the web. Which argument you have not made but only taken as a given.
tessierashpool9 2y ago
yes, yes, yes and yes!
dingaling 2y ago
> Eventually, nothing of value will be posted. (like facebook?) I'm in a couple of dozen groups on Facebook and they produce interesting and thoughtful content every day. Of course I wish they were on the open web, but don't blame the network.
- levhawk 2y ago
  After your post I tried to find dotnet groups in facebook, but all I found is spam and almost no humans, which again reminded me about dead internet theory. Could you please recommend some groups?
- freetonik 2y ago
  Would you mind sharing links to those groups? (unless they are local to your community/region/etc.)
- MichaelZuo 2y ago
  The open web as a concept doesn’t really make sense anymore, at least for written content. Since by definition it will need to be open to the majority of the world’s population, regardless of writing skill, but the bottom half of writers are already indistinguishable from cutting edge LLM output. And no one can install some sort of worldwide identity verification system, or it wouldn’t be ‘open’ anymore.
  - carlosjobim 2y ago
    Are you talking about the quality of the prose or about the content of writing? An LLM can only act on information that has been put into it by people, while a real person can create new and important information – even a person with low literal or oratory skills.
    - MichaelZuo 2y ago
      Both. How do you differentiate between say 20 real people with low skills and LLM output lightly edited by a high skilled individual?
anal_reactor 2y ago
I think that the core of the issue isn't "big corporations" but rather the fact that the internet is more accessible than ever, which means that the ratio of users who want to create content for fun, to the users who want to just consume content or create it for monetary reasons, is completely different than years ago.
whazor 2y ago
Well, you could consider a 'search engine' to be a searchable curated website directory. You could manually curate this directory and only use scrapers to allow text search on these websites. This will always be a possible option and a way ensure that your search site is good.
sshine 2y ago
> the web is dead [posted 3 hours ago on the web] m-hm.
tomjen3 2y ago
How do you square your idea that genai sucks with the fact that people are paying and using it, and not just for seo spam?
indigo0086 2y ago
Pessimists live in a world built by optimists
Grimblewald 2y ago
Plenty of awesome old guard still around (for example check out this mind blowingly awesome resource on all things water: https://water.lsbu.ac.uk/ https://water.lsbu.ac.uk/) among many other small web folks who are moving back towards making and maintaining a personal website. All that stands between you and the small web you miss is you. Kagi is fantastic though, I started using it recently and it takes me back to mid 2000's google. they even have a section dedicated to small web; https://kagi.com/smallweb https://kagi.com/smallweb

amatheus 2y ago

I think there may be a way forward for the internet. Why isn't hacker news being strangled by SEO spam for example? I think the way forward is a much smaller internet predicated on tight-knit communities approving everything that's shared. I don't know how this could scale but maybe that's the point, it must stay small to succeed.

knadh 2y ago

> I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it. I concur. Maybe now is the time to seriously think about alternate visions for "search". I have been toying with an idea[1] along those lines myself out of sheer annoyance at the state of WWW and web "search" in general. [1] https://nadh.in/blog/decentralised-open-indexes/ https://nadh.in/blog/decentralised-open-indexes/

n_ary 2y ago

I do not blame Google for web’s current wasteland landscape, I blame ads and all the grifters who poison the landscape with SEO waste to try make some bucks with ads or other similar means.

tourmalinetaco 2y ago
Google is an ad company that specifically lets SEO fill up its search results because those websites make them the most money per click. If Google continued to fight SEO abuse like they were literally founded to do then we wouldn’t be in this mess.
deleted 2y ago
[deleted]
ainiriand 2y ago
It really is Google's fault. Those blogs SEO goals are only targeting top Google positions.
rkharsan64 2y ago
There's several examples where Google's SEO has actively made things worse, even before you look at ads. And to be clear, I do agree that advertising has also done massive damage to the web. All the long, rambling stories you see before recipes? That's just there so the recipes can rank higher. I can't remember the source, but I also remember a website adding AI generated slop to cater to Google's wishes, and leaving a page to explain to normal users why they did so. It's also seen when you look at YouTube videos, where now it's basically necessary to have a clickbait thumbnail + title + reminders to like and subscribe.

RavlaAlvar 2y ago

Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening?

lelanthran 2y ago
> Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening? I've said this before (on HN) and I'll say it again: a search engine that refuses to index sites containing advertisements absolutely kills off SEO. What's the point of getting to the top of the search results if you are unable to monetise it? The only way forward is to refuse to index sites with advertisements. Google will obviously not do this (and, in fact, to me it looks like they do the reverse - downrank non-monetised content - because it's in their best interest to serve sites with ads).
- noiwillnot 2y ago
  > What's the point of getting to the top of the search results if you are unable to monetise it? Promoting your product/service, see brand astroturfing in Reddit.
  - lelanthran 2y ago
    > Promoting your product/service, see brand astroturfing in Reddit. But ... that's the point, is it not? Someone searching for "FOO" will find all those sites who optimised for selling FOO. That's an improvement over finding the top 10 results all optimised on FOO but delivering ads to make money, not FOO.
throwup238 2y ago
The same way people make adblockers happen. By making block lists of SEO garbage cruated by human beings. Nextdns and the RPi alternative do it. Kagi has all the infrastructure in place to make it happen at the search level, it just requires more manual work right now.

dazc 2y ago

You cite Facebook as an example of no meaningful content yet put all the blame upon Google? Maybe content creators are just incentivised into giving 90 percent of the public what they want.

ainiriand 2y ago
The content added to Facebook was not what made the web what it was. The content indexed by Google, in turn, it was. The countless blogs or cooking recipes, fan sites, etc, were overriden with SEO filler slop over the years. OP example of what a content-devoid internet would look like is exactly what Facebook is now. A site that started as the place to connect and share with your friends and family now is just a place that is filled with AI slop and emptiness.
bryanrasmussen 2y ago
I figure if 90% of everything is crap, https://medium.com/luminasticity/90-crap-48e4c79419a9 https://medium.com/luminasticity/90-crap-48e4c79419a9 then if the amount of available content has doubled in a decade, that is probably just too much crap to find the good stuff.

openrisk 2y ago

> I think the web is dead The web cannot 'die'. Everything and everyone moves online. Point one: A return to pre-internet days is not likely. Point two: Walled gardens had their peak moment. The dangers of everybody relying on them for our entire online existence are more than apparent to anybody with a firing neuron (and despite appearances, our species is not entirely inane). Ergo. The only path is forward, a more sane, less monopolized web. Build it and they will come.

axegon_ 2y ago
The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet. Just look at stackoverflow and github. Ever since chatgpt was released, the contributions skyrocketed and the quality dropped off the face of the earth and into a dark oblivion.
- openrisk 2y ago
  what do you mean by "the web"? The web is all the people, companies, public sector etc. that are setting up web servers (whether self-hosted, cloud etc. it doesnt matter). To say that the web is dead means that nobody outside massive social media type sites will still run a server (and SO / github are social media type walled gardens at this point, thats why they get enshittified). But this development would be absurd as a whole. Mind you search engines as we have come to know them might be dead, but thats of their own making. People need to find / communicate useful, truthful information and sooner or later they will get it.
- nextaccountic 2y ago
  We are talking on a website on the web. Not everything needs to go through a search engine.
- bryanrasmussen 2y ago
  >The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet. this presupposes the argument that for the web to have value then you must have a search engine to allow you to find things of worth on the web. Which argument you have not made but only taken as a given.

tessierashpool9 2y ago

yes, yes, yes and yes!

dingaling 2y ago

> Eventually, nothing of value will be posted. (like facebook?) I'm in a couple of dozen groups on Facebook and they produce interesting and thoughtful content every day. Of course I wish they were on the open web, but don't blame the network.

levhawk 2y ago
After your post I tried to find dotnet groups in facebook, but all I found is spam and almost no humans, which again reminded me about dead internet theory. Could you please recommend some groups?
freetonik 2y ago
Would you mind sharing links to those groups? (unless they are local to your community/region/etc.)
MichaelZuo 2y ago
The open web as a concept doesn’t really make sense anymore, at least for written content. Since by definition it will need to be open to the majority of the world’s population, regardless of writing skill, but the bottom half of writers are already indistinguishable from cutting edge LLM output. And no one can install some sort of worldwide identity verification system, or it wouldn’t be ‘open’ anymore.
- carlosjobim 2y ago
  Are you talking about the quality of the prose or about the content of writing? An LLM can only act on information that has been put into it by people, while a real person can create new and important information – even a person with low literal or oratory skills.
  - MichaelZuo 2y ago
    Both. How do you differentiate between say 20 real people with low skills and LLM output lightly edited by a high skilled individual?

anal_reactor 2y ago

I think that the core of the issue isn't "big corporations" but rather the fact that the internet is more accessible than ever, which means that the ratio of users who want to create content for fun, to the users who want to just consume content or create it for monetary reasons, is completely different than years ago.

whazor 2y ago

Well, you could consider a 'search engine' to be a searchable curated website directory. You could manually curate this directory and only use scrapers to allow text search on these websites. This will always be a possible option and a way ensure that your search site is good.

sshine 2y ago

> the web is dead [posted 3 hours ago on the web] m-hm.

tomjen3 2y ago

How do you square your idea that genai sucks with the fact that people are paying and using it, and not just for seo spam?

indigo0086 2y ago

Pessimists live in a world built by optimists

Grimblewald 2y ago

Plenty of awesome old guard still around (for example check out this mind blowingly awesome resource on all things water: https://water.lsbu.ac.uk/ https://water.lsbu.ac.uk/) among many other small web folks who are moving back towards making and maintaining a personal website. All that stands between you and the small web you miss is you. Kagi is fantastic though, I started using it recently and it takes me back to mid 2000's google. they even have a section dedicated to small web; https://kagi.com/smallweb https://kagi.com/smallweb