3 ms·

Actually what is also needed is a notLLMs.txt. robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow w

by nutanc 2y ago

Actually what is also needed is a notLLMs.txt. robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow what's the punishment.

bnchrch 2y ago
Exactly. robots.txt is useless and those that think its useful for preventing unwanted crawling are clueless
- bdcravens 2y ago
  It's a "Keep off the grass" sign. The polite will obey, the ones who are truly the problem will not.
  - immibis 2y ago
    So your site ends up not being findable by Google, and missing from the Wayback Machine once it's dead, but it does nothing for LLMs.
kilian 2y ago
There is a proposal for that too: https://site.spawning.ai/spawning-ai-txt https://site.spawning.ai/spawning-ai-txt but it's wholly unclear if AI companies actually do something with this or if it's just wishful thinking... Some AI companies follow robots.txt (OpenAI and Google, for example) but others ignore it. There's also other limitations around using robots.txt to sole this problem: https://searchengineland.com/robots-txt-new-meta-tag-llm-ai-429510 https://searchengineland.com/robots-txt-new-meta-tag-llm-ai-...