4 ms·

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is wi

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up because no one can scrape the web in an automated fashion anymore. It'll all burn down.

nicce 2y ago
> The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. AI companies with best anti-captcha mechanics will win and will inject ads to LLM output in more sophisticated way.
breckenedge 2y ago
Google is already scraping your site and presenting answers directly in search results. If I cared about traffic (hence selling ad space), why would I want my site indexed by Google at all anymore? Lots of advertising-supported sites are going to go dark because only bots will visit them.
djha-skin 2y ago
I actually envision Liapunov stability, like wolf and rabbit populations. In this scenario, we're the rabbits. Human content will increase when AI populations decrease, this providing more food for AI, which will then increase. This drowns out human expression, and the humans will grow quieter. This provides less fodder for the AI, and they decrease. This means less noise and the humans grow louder. The cycle repeats and nauseam.
- GolfPopper 2y ago
  Until broken by the Butlerian Jihad, "Though shalt not make a machine in the likeness of the mind of man."
- keyringlight 2y ago
  I've thought along similar lines for art, what ecological niches are there where AI can't participate, are harder to pull training data from or not economical, where humans can flourish.
  - bashfulpup 2y ago
    Anything we humans deem private in nature from other humans.
- Y_Y 2y ago
  See e.g. https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equations https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...
- InfamousRece 2y ago
  If the logistic driving parameter is large enough it can also lead to complete chaos.
chr1 2y ago
AI is good at solving captchas. But even if everyone added a captcha search engines will continue indexing. Because it is easy to add authentication for search engines to escape captcha, Google will just need to publish a public key.
cle 2y ago
IMO this was one of the real motives for Web Environment Integrity. Allow Google to index but nobody else. We're kind of stuck between a rock and a hard place here. Which do you prefer, entrenched incumbents or affordable/open hosting?
- lelandfe 2y ago
  I’m supremely confident that attestation will arrive in one form or another in the near future. Anonymous browsing and potentially-malicious bots look identical. This was sort of OK up until now.
  - cle 2y ago
    Agreed, it seems inevitable. Unfortunately I think it will also result in further centralization & consolidation into a handful of "trusted" megacorps. If you thought browser fingerprinting for ad tracking was creepy, just wait until they're using your actual fingerprint.
    - natebc 2y ago
      does indeed sound like we're headed right back to AOL. At least this time it'll be faster? Certainly won't be as charming.
sgc 2y ago
It will entrench established search engines even more if they have to move to auth-based crawling, so that the only crawlers will be those you invite. Most people will do this for google, bing, and maybe one or two others if there is a simple tool to do so.
sir-alien 2y ago
Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI
- regularfry 2y ago
  How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.
  - what 2y ago
    Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.
    - lacksconfidence 2y ago
      How about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?
    - regularfry 2y ago
      And in that case both systems end up with a situation new entrants can't enter.
    - usefulcat 2y ago
      I don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.
- nonrandomstring 2y ago
  This sort of positive security model with behavioural analysis is the future. We need to get it built-in to Apache,Nginx,Caddy etc. The trick is spotting crawlers from users. It can be done though.
renegat0x0 2y ago
This cannot be further from the truth. Ad business is not going anywhere. It will grow even bigger. OpenAI goes through initial cycle of enshittification. Google is too big right now. Once they establish dominance you will have to see 5 unskippable ads between prompts, even for paid plan. I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side. Example 'search' project: https://rumca-js.github.io/search https://rumca-js.github.io/search
- nicce 2y ago
  The stated problem was about indexing, accessing content and advertising in that context. > I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side. > Example 'search' project: https://rumca-js.github.io/search https://rumca-js.github.io/search That is not really solution. Since typical indexing still works for masses, your approach is currently unique. But in the end, bots will be capable of reading on web page context if human is capable on reading them. And we get back to the original problem where we try to detect bots from humans. It's the only way.
orthecreedence 2y ago
This is fine, as Google's utility as a search engine has turned into a hot pile of garbage, at least for my cases. Where a decade ago I could put in a few keywords and get relevant results, I now have to guide it with several "quoted phrases" and -exclusions to get the result I'm looking for on the second or third result page. It has crumbled under its own weight, and seems to suggest irrelevant trash to me first and foremost because it's the website of some big player or content farm. Either their algorithm is tuned for mass manipulation or they lost the arms race with SEO cretins (or both). Granted, I'm not looking forward to some LLM condensing all the garbage and handing me a Definitive Answer (TM) based on the information it deems relevant for inclusion.
ethan_smith 2y ago
What about the next-gen of AI that would be able to signup autonomously? Even if implemented auth-walls everywhere right now, whats stopping the companies to get some real cheap labor to create accounts on websites and use them to scrape the content? Is it going to become another race like the adblocker -> detect adblocker -> bypass adblocker detector and so on...?

nicce 2y ago

> The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. AI companies with best anti-captcha mechanics will win and will inject ads to LLM output in more sophisticated way.

breckenedge 2y ago

Google is already scraping your site and presenting answers directly in search results. If I cared about traffic (hence selling ad space), why would I want my site indexed by Google at all anymore? Lots of advertising-supported sites are going to go dark because only bots will visit them.

djha-skin 2y ago

I actually envision Liapunov stability, like wolf and rabbit populations. In this scenario, we're the rabbits. Human content will increase when AI populations decrease, this providing more food for AI, which will then increase. This drowns out human expression, and the humans will grow quieter. This provides less fodder for the AI, and they decrease. This means less noise and the humans grow louder. The cycle repeats and nauseam.

GolfPopper 2y ago
Until broken by the Butlerian Jihad, "Though shalt not make a machine in the likeness of the mind of man."
keyringlight 2y ago
I've thought along similar lines for art, what ecological niches are there where AI can't participate, are harder to pull training data from or not economical, where humans can flourish.
- bashfulpup 2y ago
  Anything we humans deem private in nature from other humans.
Y_Y 2y ago
See e.g. https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equations https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...
InfamousRece 2y ago
If the logistic driving parameter is large enough it can also lead to complete chaos.

chr1 2y ago

AI is good at solving captchas. But even if everyone added a captcha search engines will continue indexing. Because it is easy to add authentication for search engines to escape captcha, Google will just need to publish a public key.

cle 2y ago

IMO this was one of the real motives for Web Environment Integrity. Allow Google to index but nobody else. We're kind of stuck between a rock and a hard place here. Which do you prefer, entrenched incumbents or affordable/open hosting?

lelandfe 2y ago
I’m supremely confident that attestation will arrive in one form or another in the near future. Anonymous browsing and potentially-malicious bots look identical. This was sort of OK up until now.
- cle 2y ago
  Agreed, it seems inevitable. Unfortunately I think it will also result in further centralization & consolidation into a handful of "trusted" megacorps. If you thought browser fingerprinting for ad tracking was creepy, just wait until they're using your actual fingerprint.
  - natebc 2y ago
    does indeed sound like we're headed right back to AOL. At least this time it'll be faster? Certainly won't be as charming.

sgc 2y ago

It will entrench established search engines even more if they have to move to auth-based crawling, so that the only crawlers will be those you invite. Most people will do this for google, bing, and maybe one or two others if there is a simple tool to do so.

sir-alien 2y ago

Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI

regularfry 2y ago
How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.
- what 2y ago
  Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.
  - lacksconfidence 2y ago
    How about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?
  - regularfry 2y ago
    And in that case both systems end up with a situation new entrants can't enter.
  - usefulcat 2y ago
    I don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.
nonrandomstring 2y ago
This sort of positive security model with behavioural analysis is the future. We need to get it built-in to Apache,Nginx,Caddy etc. The trick is spotting crawlers from users. It can be done though.

renegat0x0 2y ago

This cannot be further from the truth. Ad business is not going anywhere. It will grow even bigger. OpenAI goes through initial cycle of enshittification. Google is too big right now. Once they establish dominance you will have to see 5 unskippable ads between prompts, even for paid plan. I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side. Example 'search' project: https://rumca-js.github.io/search https://rumca-js.github.io/search

nicce 2y ago
The stated problem was about indexing, accessing content and advertising in that context. > I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side. > Example 'search' project: https://rumca-js.github.io/search https://rumca-js.github.io/search That is not really solution. Since typical indexing still works for masses, your approach is currently unique. But in the end, bots will be capable of reading on web page context if human is capable on reading them. And we get back to the original problem where we try to detect bots from humans. It's the only way.

orthecreedence 2y ago

This is fine, as Google's utility as a search engine has turned into a hot pile of garbage, at least for my cases. Where a decade ago I could put in a few keywords and get relevant results, I now have to guide it with several "quoted phrases" and -exclusions to get the result I'm looking for on the second or third result page. It has crumbled under its own weight, and seems to suggest irrelevant trash to me first and foremost because it's the website of some big player or content farm. Either their algorithm is tuned for mass manipulation or they lost the arms race with SEO cretins (or both). Granted, I'm not looking forward to some LLM condensing all the garbage and handing me a Definitive Answer (TM) based on the information it deems relevant for inclusion.

ethan_smith 2y ago

What about the next-gen of AI that would be able to signup autonomously? Even if implemented auth-walls everywhere right now, whats stopping the companies to get some real cheap labor to create accounts on websites and use them to scrape the content? Is it going to become another race like the adblocker -> detect adblocker -> bypass adblocker detector and so on...?