8 ms·
Hi Jeremy here. Nice to see this on HN. To explain the reasoning for this proposal, by way of an example: I recently released FastHTML, a small library for cre
by jph00 2y ago
Hi Jeremy here. Nice to see this on HN.
To explain the reasoning for this proposal, by way of an example: I recently released FastHTML, a small library for creating hypermedia applications, and by far the most common concern I've received from potential users is that language models aren't able to help use it, since it was created after the knowledge cutoff of current models.
IDEs like Cursor let you add docs to the model context, which is a great solution to this issue -- except what docs should you add? The idea is that if you, as a site creator, want to make it easier for systems like Cursor to use your docs, then you can provide a small text file linking to the AI-friendly documentation you think is most likely to be helpful in the context window.
Of course, these systems already are perfectly capable of doing their own automated scraping, but the results aren't that great. They don't really know what's needed to be in context to get the key foundational information, and some of that information might be on external sites anyway. I've found I get dramatically better results by carefully curating the context for my prompts for each system I use, and it seems like a waste of time for everyone to redo the same work of this curation, rather than the site owner doing it once for every visitor that needs it. I've also found this very useful with Claude Projects.
llms.txt isn't really designed to help with scraping; it's designed to help end-users use the information on web sites with the help of AI, for web-site owners interested in doing that. It's orthogonal to robots.txt, which is used to let bots know what they may and may not access.
(If folks feel like this proposal is helpful, then it might be worth registering with /.well-known/. Since the RFC for that says "Applications that wish to mint new well-known URIs MUST register them", and I don't even know if people are interested in this, it felt a bit soon to be registering it now.)
- crowcroft 2y agoI understand the problem, but I'm not convinced this solution would do much to solve the problem? 1. LLMs give this doc special preference and SEO type optimisation will run rampant by brands. 2. LLMs crawl this as just another page, and then you need to ask yourself why isn't this context already on the website?
- catchmost 2y agoI do agree with the other commenters about this being better solved with a <link rel="llm"> or just an Accept: text/markdown; profile=llm header. It's not given that a site only contains a single "thing" that LLMs are interested in. To continue your dev-doc example, many projects use github instead of their own website. Github's /llms.txt wouldn't contain anything at all about your FastHTML project, but rather instructions on how to use GitHub. That is not useful for people who asked Cursor about your library. Slightly off topic: An alternative approach to making sites more accessible to LLMs would be to revive the original interpretation of REST (markup with affordances for available actions).