3 ms·

Wouldn't this open up for manipulating LLMs? You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares n

Wouldn't this open up for manipulating LLMs? You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares no resemblance to the actual content of the page. If you really care about your content being picked up by the scrapers, why not structure it better? Most of the LLMs are pretty much black boxes, so we don't really know what a better structure would look like, but I would make the guess that involves simplifying your HTML and removing irrelevant tokens.

jph00 2y ago
llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.
- mrweasel 2y ago
  > if you create an llms.txt you're not looking to manipulate anyone You don't know me :-) My suggestion is that someone might want taint the data that goes into an LLM. Let's say you have a website with guides, examples and tips and tricks for writing bash. What would prevent you from pointing the LLMs to separate content which would contain broken examples and code with a number of security issues, because you long term would want to exploit the code generated by the LLMs.
  - idf00 2y ago
    llms.txt doesn't seem to make it any easier or harder to do that.
    - sodality2 2y ago
      ...it gives you a version only the LLM will see, as opposed to having to identify the visitor as a scraper or human and determining if they get the good or bad version.
      - deleted 2y ago
        [deleted]
- msp26 2y ago
  > presumably if you create an llms.txt you're not looking to manipulate anyone I don't think the public sentiment around scrapers and LLMs is that friendly. I personally think scraping is very important and it allows for smaller players to compete (as long as you're careful about the number of requests).

jph00 2y ago

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.

mrweasel 2y ago
> if you create an llms.txt you're not looking to manipulate anyone You don't know me :-) My suggestion is that someone might want taint the data that goes into an LLM. Let's say you have a website with guides, examples and tips and tricks for writing bash. What would prevent you from pointing the LLMs to separate content which would contain broken examples and code with a number of security issues, because you long term would want to exploit the code generated by the LLMs.
- idf00 2y ago
  llms.txt doesn't seem to make it any easier or harder to do that.
  - sodality2 2y ago
    ...it gives you a version only the LLM will see, as opposed to having to identify the visitor as a scraper or human and determining if they get the good or bad version.
    - deleted 2y ago
      [deleted]
msp26 2y ago
> presumably if you create an llms.txt you're not looking to manipulate anyone I don't think the public sentiment around scrapers and LLMs is that friendly. I personally think scraping is very important and it allows for smaller players to compete (as long as you're careful about the number of requests).