4 ms·

Do any agents respect agents.txt? Is there a way to opt my websites out of ai data collection?

embedding-shape 6mo ago
Add HTTP Basic Auth in front of your website, then share the credentials with people who are allowed to view your website. Make sure you don't hand our credentials to employees of OpenAI, Anthropic, xAI or Microsoft.
- kennywinker 6mo ago
  Ah, yes. A reasonable situation. If you don’t want your data slurped up by these companies, simply don’t publish anything anywhere. Seems fine.
wolttam 6mo ago
Any measure you put in place can/will be ignored by the actors who never planned to respect your wishes in the first place. That's just how the web works, though.
- cortesoft 6mo ago
  This is true for measures that require the actor to respect your wishes, but doesn't apply to measures that force them to.
  - wolttam 6mo ago
    This is just like security; the most secure system is the one that nobody can use. I think the proof-of-work approach that anubis[0] takes is pretty interesting. I love the idea of having to do a small amount of work for the author of the content in order to get access to their content. It would be interesting to a scheme where the proof-of-work that clients do in systems like anubis actually had a way to directly benefit the author. [0]: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
    - rickydroll 6mo ago
      From working with hashcash and anti-spam, using a proof of work puzzle as a rate modulator is far more effective if you extended the application protocol to include that hashcash token. Using a Hashcash token to HTTP would have multiple wins: - Puts the load on the sender, i.e. server hands off the token request and then drops the connection, leaving it free to serve somebody else. - Token size is dynamic. - rate limiting a client - hug of death resistance - token inflation resistance - CURL and Wget or any other programmatic HTTP request can be made to work with the addition of a Hashcash token. At a crayon sketch level, the protocol works something like this: HTTP request WITH header stamp: available. Server returns a 3xx <stamp size required> Next request Stamp: <stamp matching required stamp size> header HTTP request then completes. Put this protocol into Firefox/Chromium, and the usefulness of this protocol would increase as people upgrade their browsers. If we do not get a `Stamp-Available` header, then the system will revert to the Anubis mechanism. One advantage of the Anubis interrupting is that can be used to advertise the presence and value of upgrading to an Anubis++ browser. I.e. You don't get the interrupting web page telling you all about Anubis and Anubis++, and you join the resistance pushing back against AI scrapers.
- kennywinker 6mo ago
  I don’t think just how the web works is good enough for this situation. This is a new situation that’s never existed before. I have no way to prevent things I create being ingested into an llm and used to make money for another company? With search engines there was quid pro quo. I get traffic they get traffic. This is entirely one-sided and there’s no intellectual property protections or sigils i can use to opt out? Messed up
ghostlyy 6mo ago
partial answer: the major labs (Anthropic, OpenAI) do respect robots.txt for their named crawlers, so blocking ClaudeBot/GPTBot in robots.txt works for those specific bots. What you can't easily opt out of is the indirect ingestion via Common Crawl, scraped datasets, and unnamed crawlers. agents.txt doesn't change that picture. The Allow-Training vs Allow-RAG split in the default is the useful part of the file. They're different operations with different costs to the site owner. Training is a one-time bulk ingest. RAG is a runtime fetch per query. A site owner might reasonably allow one and not the other.
- ninjin 6mo ago
  I can report that Facebook does not respect robots.txt. Heck, I even mailed domain@fb.com with the specific IP ranges and log samples three times over a month and they of did not even respond. Keeps on wasting my CPU cycles to this day by crawling massive development forks (I hope they choke on the data...): $ (cat /var/www/logs/access.log; zcat /var/www/logs/access.log*.gz) | grep 2a03:2880: | wc -l 626396 About three hits per second for months now.
  - drcongo 6mo ago
    I block their entire ASN when they do that.
  - dylan604 6mo ago
    Can you serve them a specific file that would make it expensive on their end?
    - ninjin 6mo ago
      If I had the time and energy, I would make some sort of simple code language model and generate infinite junk and feed that to them in the hope that it ruins their future training runs. But, I lack the former and some of the latter. Alternatively, maybe I would actually read one of those "backdoor papers" and try to inject something like that.
sschueller 6mo ago
Well Claude still thinks it shouldn't read AGENTS.md [1] so they probably also don't care about agents.txt on a web server... [1] https://github.com/anthropics/claude-code/issues/6235 https://github.com/anthropics/claude-code/issues/6235

Add HTTP Basic Auth in front of your website, then share the credentials with people who are allowed to view your website. Make sure you don't hand our credentials to employees of OpenAI, Anthropic, xAI or Microsoft.

kennywinker 6mo ago
Ah, yes. A reasonable situation. If you don’t want your data slurped up by these companies, simply don’t publish anything anywhere. Seems fine.

wolttam 6mo ago

Any measure you put in place can/will be ignored by the actors who never planned to respect your wishes in the first place. That's just how the web works, though.

cortesoft 6mo ago
This is true for measures that require the actor to respect your wishes, but doesn't apply to measures that force them to.
- wolttam 6mo ago
  This is just like security; the most secure system is the one that nobody can use. I think the proof-of-work approach that anubis[0] takes is pretty interesting. I love the idea of having to do a small amount of work for the author of the content in order to get access to their content. It would be interesting to a scheme where the proof-of-work that clients do in systems like anubis actually had a way to directly benefit the author. [0]: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
  - rickydroll 6mo ago
    From working with hashcash and anti-spam, using a proof of work puzzle as a rate modulator is far more effective if you extended the application protocol to include that hashcash token. Using a Hashcash token to HTTP would have multiple wins: - Puts the load on the sender, i.e. server hands off the token request and then drops the connection, leaving it free to serve somebody else. - Token size is dynamic. - rate limiting a client - hug of death resistance - token inflation resistance - CURL and Wget or any other programmatic HTTP request can be made to work with the addition of a Hashcash token. At a crayon sketch level, the protocol works something like this: HTTP request WITH header stamp: available. Server returns a 3xx <stamp size required> Next request Stamp: <stamp matching required stamp size> header HTTP request then completes. Put this protocol into Firefox/Chromium, and the usefulness of this protocol would increase as people upgrade their browsers. If we do not get a `Stamp-Available` header, then the system will revert to the Anubis mechanism. One advantage of the Anubis interrupting is that can be used to advertise the presence and value of upgrading to an Anubis++ browser. I.e. You don't get the interrupting web page telling you all about Anubis and Anubis++, and you join the resistance pushing back against AI scrapers.
kennywinker 6mo ago
I don’t think just how the web works is good enough for this situation. This is a new situation that’s never existed before. I have no way to prevent things I create being ingested into an llm and used to make money for another company? With search engines there was quid pro quo. I get traffic they get traffic. This is entirely one-sided and there’s no intellectual property protections or sigils i can use to opt out? Messed up

ghostlyy 6mo ago

partial answer: the major labs (Anthropic, OpenAI) do respect robots.txt for their named crawlers, so blocking ClaudeBot/GPTBot in robots.txt works for those specific bots. What you can't easily opt out of is the indirect ingestion via Common Crawl, scraped datasets, and unnamed crawlers. agents.txt doesn't change that picture. The Allow-Training vs Allow-RAG split in the default is the useful part of the file. They're different operations with different costs to the site owner. Training is a one-time bulk ingest. RAG is a runtime fetch per query. A site owner might reasonably allow one and not the other.

ninjin 6mo ago
I can report that Facebook does not respect robots.txt. Heck, I even mailed domain@fb.com with the specific IP ranges and log samples three times over a month and they of did not even respond. Keeps on wasting my CPU cycles to this day by crawling massive development forks (I hope they choke on the data...): $ (cat /var/www/logs/access.log; zcat /var/www/logs/access.log*.gz) | grep 2a03:2880: | wc -l 626396 About three hits per second for months now.
- drcongo 6mo ago
  I block their entire ASN when they do that.
- dylan604 6mo ago
  Can you serve them a specific file that would make it expensive on their end?
  - ninjin 6mo ago
    If I had the time and energy, I would make some sort of simple code language model and generate infinite junk and feed that to them in the hope that it ruins their future training runs. But, I lack the former and some of the latter. Alternatively, maybe I would actually read one of those "backdoor papers" and try to inject something like that.

sschueller 6mo ago

Well Claude still thinks it shouldn't read AGENTS.md [1] so they probably also don't care about agents.txt on a web server... [1] https://github.com/anthropics/claude-code/issues/6235 https://github.com/anthropics/claude-code/issues/6235