3 ms·

It would be nice if this could get standardized http headers so bots could still use sites but they effectively pay for use. That seems like the best of all pos

It would be nice if this could get standardized http headers so bots could still use sites but they effectively pay for use. That seems like the best of all possible worlds to me, the whole point of HTML is that robots can read it, otherwise we'd just be emailing eachother pdfs.

ramses0 1y ago
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ...and the requisite checklist: https://trog.qgl.org/20081217/the-why-your-anti-spam-idea-wont-work-checklist/ https://trog.qgl.org/20081217/the-why-your-anti-spam-idea-wo...
- msgodel 1y ago
  Formalizing it doesn't change that it's being used. If it doesn't work it shouldn't be done, if it does it should be formalized.
overfeed 1y ago
> bots could still use sites but they effectively pay for use. That seems like the best of all possible worlds to me This would make the entire internet a maze of AI-slop content primarily made for other bots to consume. Humans may have to resort to emailing handwritten PDFs to avoid the thoroughly enshittified web.
- dandelany 1y ago
  As opposed to what we have now?
  - overfeed 1y ago
    Yes - things could get worse from the status quo. At the moment, ad networks don't pay for bot impressions when detected - so content farms tend to optimize for what passes for humans. All bets are off if human and bot visitors offer the same economic value via miners, or worse if it turns out that bots are more profitable due to human impatience. Imagine an internet optimized for bot visitors, and indifferent to humans. It would be a kind of refined brainrot aimed at a brainless audience.
DrillShopper 1y ago
They should have to set the evil bit
theamk 1y ago
We already have a standardized system - robots.txt - and AI bots are already ignoring it. Why would more standardized headers matter? Bots will ignore them just as they do today, pretend to be regular users, and get content without paying. (A secondary thing is that AI bots have basically zero benefit for most websites, so unless you are some low-cost crappy content farm, it'll be in your interest to raise the prices to the max so the bots are simply locked out. Which will bring us back to point 1, bots ignoring the headers)
- msgodel 1y ago
  Being indexed in search engines has zero benefit? Also robots.txt is a suggestion but hashcash is enforced server side. I agree it's a tragedy people have started to completely ignore it but you can't ignore server side behavior.
  - MyPasswordSucks 1y ago
    How do you propose the server distinguish between a bot and a human visitor?
  - theamk 1y ago
    Being indexed in search engines is very useful, but does not need any filtering - Google and all other major search engines respect robots.txt, use well-known UA and even publish IP address. AI bots are not search engines, and they have no benefits for the website owners. This can be very clearly seen because they ignore robots.txt, pretend to be regular browsers, and use multiple proxies to avoid IP bans.

ramses0 1y ago

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ...and the requisite checklist: https://trog.qgl.org/20081217/the-why-your-anti-spam-idea-wont-work-checklist/ https://trog.qgl.org/20081217/the-why-your-anti-spam-idea-wo...

msgodel 1y ago
Formalizing it doesn't change that it's being used. If it doesn't work it shouldn't be done, if it does it should be formalized.

overfeed 1y ago

> bots could still use sites but they effectively pay for use. That seems like the best of all possible worlds to me This would make the entire internet a maze of AI-slop content primarily made for other bots to consume. Humans may have to resort to emailing handwritten PDFs to avoid the thoroughly enshittified web.

dandelany 1y ago
As opposed to what we have now?
- overfeed 1y ago
  Yes - things could get worse from the status quo. At the moment, ad networks don't pay for bot impressions when detected - so content farms tend to optimize for what passes for humans. All bets are off if human and bot visitors offer the same economic value via miners, or worse if it turns out that bots are more profitable due to human impatience. Imagine an internet optimized for bot visitors, and indifferent to humans. It would be a kind of refined brainrot aimed at a brainless audience.

DrillShopper 1y ago

They should have to set the evil bit

theamk 1y ago

We already have a standardized system - robots.txt - and AI bots are already ignoring it. Why would more standardized headers matter? Bots will ignore them just as they do today, pretend to be regular users, and get content without paying. (A secondary thing is that AI bots have basically zero benefit for most websites, so unless you are some low-cost crappy content farm, it'll be in your interest to raise the prices to the max so the bots are simply locked out. Which will bring us back to point 1, bots ignoring the headers)

msgodel 1y ago
Being indexed in search engines has zero benefit? Also robots.txt is a suggestion but hashcash is enforced server side. I agree it's a tragedy people have started to completely ignore it but you can't ignore server side behavior.
- MyPasswordSucks 1y ago
  How do you propose the server distinguish between a bot and a human visitor?
- theamk 1y ago
  Being indexed in search engines is very useful, but does not need any filtering - Google and all other major search engines respect robots.txt, use well-known UA and even publish IP address. AI bots are not search engines, and they have no benefits for the website owners. This can be very clearly seen because they ignore robots.txt, pretend to be regular browsers, and use multiple proxies to avoid IP bans.