3 ms·
From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html): "The problem this
by jph00 2y ago
From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html):
"The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you:
1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form?
2. Selectively include external links in addition to the sitemap?
3. For specific domains like software documentation should you also try to include all the source code?
Site authors know best, and can provide a list of content that an LLM should use."
(There's quite a bit more info there that answers this question in more detail.)
- imjonse 2y agoI agree site authors should be able to tell what content they would like to be used for LLM training (even though that opinion will likely be ignored by LLM training scrapers), but the format of it is really up to those gathering and cleaning the data. It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace. It is also something LLM scrapers would need to validate/check/reformat anyway to protect from errors/trolling/poisoning of the data since even if most authors would provide curated info, not all will.