4 ms·

LLMs put the free web at risk

10 points by anfernan 2y ago

My cofounder and I think the rise of AI has put the free web at risk. Despite their claims, ChatGPT and Perplexity don’t provide live or complete information from the internet. Instead, they index only the parts of webpages their algorithms deem important. These limitations have given rise to a new breed of AI browser agents powered by the same foundational LLM tech. These LLMs + agents will likely account for over 90% of web traffic in the near future. Bots scrape content without compensating websites for it. As humans access the web through LLMs and agents more, they directly view websites less. The loss of ad revenue from human visitors threatens many websites’ ability to provide their content publicly for free. We’ve been considering solutions to these problems, and want to know what you think. 1) Are they problems? 2) How big and serious are they? 3) What are possible solutions? We have some ideas…

deleted 2y ago
[deleted]
tpswa 2y ago
Depends on the site! If you're trying to sell products, you don't necessarily care so long as your products are discoverable by ChatGPT/Perplexity/an agent -- visibility/discoverability is definitely a priority for some businesses. If you're selling ads it definitely is a risk, though licensing deals are one solution but that seems to be only available currently for bigger companies.
- anfernan 2y ago
  Definitely! If your revenue comes from selling something to site visitors, you want to be scraped. If you're selling ads, (as far as we know) your options are to either offer an API or make individual licensing agreements. There's no standardized, go-to place to do this.
paulcole 2y ago
> My cofounder and I think the rise of AI has put the free web at risk To paraphrase Upton Sinclair, “When your paycheck depends on you thinking something, it will be very hard to get you to not think it.”
- anfernan 2y ago
  It's a good line and true! We're not working on anything that monetizes this. At least for now...
mattbrewsbytes 2y ago
I think human written, compelling, deep content will survive. Fluff content sites will likely be squashed by search engines favoring brands and authority over linkage stats because there is so much AI generated content that's mediocre, the search engines have to weed thru that to have relevant results. Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searching something like ratio of oil and vinegar for salad dressing. That person likely is making a salad right now and needs to know and does not care about reading through 5 screens of scrolling content, ads, popups, etc. to get a recipe. Now if someone is researching the history of salad dressings and wants to go deep into types of oils, vinegars, mixture ratios, pairing with different lettuce types, fruits, nuts, etc. having content that provides that level of information is helpful, but to a smaller audience. That audience is more likely to click links to products or ads than someone just looking for a ratio right now. The human impression counts for content sites is likely to go down, a lot for fluff content, but those sites are likely losing impressions from people that weren't clicking the ads or buying products anyhow. I'm not in advertising/marketing so I don't know what impacts that has ... maybe for some time period, ad revenues will go down because sites lose the fluff visitors and number of visitors carries weight in advertising processes, right? Then maybe the processes catch up and realize that the good content sites have higher "stickyness" factor so conversion rates and length of users visit are much more important than page impression counts?
- itsankur 2y ago
  Yeah I mean the salad example is great because in that case it feels ridiculous to have to look at ads. Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet? Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another. And all LLM providers will pay their data providers directly by API or something? original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?
  - 2y ago
deleted 2y ago
[deleted]
scarface_74 2y ago
Getting rid of ad supported content and SEO optimizations? Nothing of value will be lost. There is nothing “free” about the tracking and evilness of adtech. I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours. The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.
- anfernan 2y ago
  Who, me? Haha What site are you referring to? Ours doesn't have a cookie banner
  - scarface_74 2y ago
    I had the wrong website. What you are doing is even worse. You are scraping websites for your own use.
- kjok 2y ago
  But you still have to visit websites (or use apps) to do tasks online like booking tickets, getting stuff delivered, or paying bills. They are all tracking us. Do you also plan to pay for AI agents that do these chores on your behalf?
  - throwaway784989 2y ago
    I don't have to use the web or apps for any of that stuff. They're a convenience, but are not actually necessary for any of that.
  - scarface_74 2y ago
    I guess I was a little overwrought. Ads don’t bother me for ideological reasons. They enshittify everything they touch. Of course I have an ad blocker. All of my bills are auto pay.

10 points by anfernan 2y ago

deleted 2y ago

[deleted]

tpswa 2y ago

Depends on the site! If you're trying to sell products, you don't necessarily care so long as your products are discoverable by ChatGPT/Perplexity/an agent -- visibility/discoverability is definitely a priority for some businesses. If you're selling ads it definitely is a risk, though licensing deals are one solution but that seems to be only available currently for bigger companies.

anfernan 2y ago
Definitely! If your revenue comes from selling something to site visitors, you want to be scraped. If you're selling ads, (as far as we know) your options are to either offer an API or make individual licensing agreements. There's no standardized, go-to place to do this.

paulcole 2y ago

> My cofounder and I think the rise of AI has put the free web at risk To paraphrase Upton Sinclair, “When your paycheck depends on you thinking something, it will be very hard to get you to not think it.”

anfernan 2y ago
It's a good line and true! We're not working on anything that monetizes this. At least for now...

mattbrewsbytes 2y ago

I think human written, compelling, deep content will survive. Fluff content sites will likely be squashed by search engines favoring brands and authority over linkage stats because there is so much AI generated content that's mediocre, the search engines have to weed thru that to have relevant results. Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searching something like ratio of oil and vinegar for salad dressing. That person likely is making a salad right now and needs to know and does not care about reading through 5 screens of scrolling content, ads, popups, etc. to get a recipe. Now if someone is researching the history of salad dressings and wants to go deep into types of oils, vinegars, mixture ratios, pairing with different lettuce types, fruits, nuts, etc. having content that provides that level of information is helpful, but to a smaller audience. That audience is more likely to click links to products or ads than someone just looking for a ratio right now. The human impression counts for content sites is likely to go down, a lot for fluff content, but those sites are likely losing impressions from people that weren't clicking the ads or buying products anyhow. I'm not in advertising/marketing so I don't know what impacts that has ... maybe for some time period, ad revenues will go down because sites lose the fluff visitors and number of visitors carries weight in advertising processes, right? Then maybe the processes catch up and realize that the good content sites have higher "stickyness" factor so conversion rates and length of users visit are much more important than page impression counts?

itsankur 2y ago
Yeah I mean the salad example is great because in that case it feels ridiculous to have to look at ads. Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet? Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another. And all LLM providers will pay their data providers directly by API or something? original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?
- 2y ago

deleted 2y ago

[deleted]

scarface_74 2y ago

Getting rid of ad supported content and SEO optimizations? Nothing of value will be lost. There is nothing “free” about the tracking and evilness of adtech. I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours. The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.

anfernan 2y ago
Who, me? Haha What site are you referring to? Ours doesn't have a cookie banner
- scarface_74 2y ago
  I had the wrong website. What you are doing is even worse. You are scraping websites for your own use.
kjok 2y ago
But you still have to visit websites (or use apps) to do tasks online like booking tickets, getting stuff delivered, or paying bills. They are all tracking us. Do you also plan to pay for AI agents that do these chores on your behalf?
- throwaway784989 2y ago
  I don't have to use the web or apps for any of that stuff. They're a convenience, but are not actually necessary for any of that.
- scarface_74 2y ago
  I guess I was a little overwrought. Ads don’t bother me for ideological reasons. They enshittify everything they touch. Of course I have an ad blocker. All of my bills are auto pay.