4 ms·

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.

• mrweasel 2y ago
  > if you create an llms.txt you're not looking to manipulate anyone You don't know me :-) My suggestion is that someone might want taint the data that goes into an LLM. Let's say you have a website with guides, examples and tips and tricks for writing bash. What would prevent you from pointing the LLMs to separate content which would contain broken examples and code with a number of security issues, because you long term would want to exploit the code generated by the LLMs.
  • idf00 2y ago
    llms.txt doesn't seem to make it any easier or harder to do that.
    • sodality2 2y ago
      ...it gives you a version only the LLM will see, as opposed to having to identify the visitor as a scraper or human and determining if they get the good or bad version.
      • deleted 2y ago
        [deleted]
• msp26 2y ago
  > presumably if you create an llms.txt you're not looking to manipulate anyone I don't think the public sentiment around scrapers and LLMs is that friendly. I personally think scraping is very important and it allows for smaller players to compete (as long as you're careful about the number of requests).