4 ms·

What problem does this solve?

dylan604 2y ago
It solves the problem of the cat&mouse game of LLMs updating their scrapers by making site owners provide them the data in a format the LLMs have developed around already. You're clearly looking at this from the incorrect point of view. Silly human. Think like a bot. --The bot makers
- jph00 2y ago
  This proposal isn’t designed to help training. It’s designed to help end-users.
tbrownaw 2y ago
I think the idea is that LLMs aren't actually that good, so adding a semi-machine-readable version of your site can make it easier for them to surface your work to their own users.
- TZubiri 2y ago
  Html is highly machine readable. That's how crawlers like google and yahoo and browsers like chrome and netscape can work.
jph00 2y ago
From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html): "The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you: 1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form? 2. Selectively include external links in addition to the sitemap? 3. For specific domains like software documentation should you also try to include all the source code? Site authors know best, and can provide a list of content that an LLM should use." (There's quite a bit more info there that answers this question in more detail.)
- imjonse 2y ago
  I agree site authors should be able to tell what content they would like to be used for LLM training (even though that opinion will likely be ignored by LLM training scrapers), but the format of it is really up to those gathering and cleaning the data. It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace. It is also something LLM scrapers would need to validate/check/reformat anyway to protect from errors/trolling/poisoning of the data since even if most authors would provide curated info, not all will.
  - idf00 2y ago
    It's not to help people train models. It's for end-users to use in an LLM context (like Claude projects or cursor) to help them use your tool better.
    - imjonse 2y ago
      My bad, I did not read it carefully.
randomdata 2y ago
It tries to solve the problem of LLMs not having necessary context (because information you require was created after last training period, for example) by offering a document optimized for copying/pasting that you can include in your prompt, RAG-style.
- TZubiri 2y ago
  So the problem is for llms yet the tool is for site owners? Why don't we make a tool that solves poverty by taxing the rich?
  - randomdata 2y ago
    > So the problem is for llms yet the tool is for site owners? The problem is that of end users, and the tool is an attempt to help them with their problem. It does require cooperation with site owners, yes, but when a site exists to help the end user... > Why don't we make a tool that solves poverty by taxing the rich? Well, for one, there is not nearly enough utilized resources in the world to solve poverty. Taxing everything we can get our hands on would still only provide a fraction of what would be needed to solve poverty. As things sit today, it is mathematically impossible to solve poverty. There is all kinds of unutilized resources, namely human capital, that could potentially see an end to poverty if fully utilized, but you will never tax your way into utilizing unutilzed resources. A tool to unlock those resources would be useful, and, indeed, there are efforts underway to try and develop those tools, but we don't yet have the technology. It turns out developing such a tool is way harder than casually proposing that we agree to name a file `llms.txt`.
    - TZubiri 2y ago
      Also very cute of you to assume that llms are still being trained on websites. Or that the crib of software (california) with elite engineers (openai comp averages 900k/yr) needs help with a task that indians can do for 3 bucks an hour (web scraping)
      - 2y ago

It solves the problem of the cat&mouse game of LLMs updating their scrapers by making site owners provide them the data in a format the LLMs have developed around already. You're clearly looking at this from the incorrect point of view. Silly human. Think like a bot. --The bot makers

jph00 2y ago
This proposal isn’t designed to help training. It’s designed to help end-users.

tbrownaw 2y ago

I think the idea is that LLMs aren't actually that good, so adding a semi-machine-readable version of your site can make it easier for them to surface your work to their own users.

TZubiri 2y ago
Html is highly machine readable. That's how crawlers like google and yahoo and browsers like chrome and netscape can work.

jph00 2y ago

From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html): "The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you: 1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form? 2. Selectively include external links in addition to the sitemap? 3. For specific domains like software documentation should you also try to include all the source code? Site authors know best, and can provide a list of content that an LLM should use." (There's quite a bit more info there that answers this question in more detail.)

imjonse 2y ago
I agree site authors should be able to tell what content they would like to be used for LLM training (even though that opinion will likely be ignored by LLM training scrapers), but the format of it is really up to those gathering and cleaning the data. It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace. It is also something LLM scrapers would need to validate/check/reformat anyway to protect from errors/trolling/poisoning of the data since even if most authors would provide curated info, not all will.
- idf00 2y ago
  It's not to help people train models. It's for end-users to use in an LLM context (like Claude projects or cursor) to help them use your tool better.
  - imjonse 2y ago
    My bad, I did not read it carefully.

randomdata 2y ago

It tries to solve the problem of LLMs not having necessary context (because information you require was created after last training period, for example) by offering a document optimized for copying/pasting that you can include in your prompt, RAG-style.

TZubiri 2y ago
So the problem is for llms yet the tool is for site owners? Why don't we make a tool that solves poverty by taxing the rich?
- randomdata 2y ago
  > So the problem is for llms yet the tool is for site owners? The problem is that of end users, and the tool is an attempt to help them with their problem. It does require cooperation with site owners, yes, but when a site exists to help the end user... > Why don't we make a tool that solves poverty by taxing the rich? Well, for one, there is not nearly enough utilized resources in the world to solve poverty. Taxing everything we can get our hands on would still only provide a fraction of what would be needed to solve poverty. As things sit today, it is mathematically impossible to solve poverty. There is all kinds of unutilized resources, namely human capital, that could potentially see an end to poverty if fully utilized, but you will never tax your way into utilizing unutilzed resources. A tool to unlock those resources would be useful, and, indeed, there are efforts underway to try and develop those tools, but we don't yet have the technology. It turns out developing such a tool is way harder than casually proposing that we agree to name a file `llms.txt`.
  - TZubiri 2y ago
    Also very cute of you to assume that llms are still being trained on websites. Or that the crib of software (california) with elite engineers (openai comp averages 900k/yr) needs help with a task that indians can do for 3 bucks an hour (web scraping)
    - 2y ago