4 ms·

Yes! Please standardize the web into simple hypertext so “LLMs can use it”. I promise I won’t build any tools to read it without the ads, tracking, and JavaScri

by caseyy 2y ago

Yes! Please standardize the web into simple hypertext so “LLMs can use it”. I promise I won’t build any tools to read it without the ads, tracking, and JavaScript client side garbage myself. I will not partake in any such efforts to surf the web as it was intended be, before its commercialization and commodification. No, sir, I could never!

Eisenstein 2y ago
The problem I see is that people will develop two versions of websites -- the LLM version, optimized to give the model a good impression of their products or services and get them into the training data, and the human version (with sub-versions for mobile and such) which will be SEOd to hell. No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race until * the incentives are removed * the process is made so technically complex or expensive that only a few players can profit from them * it is regulated such that people can make money doing other things which have better risk/reward * most people just avoid the whole ecosystem because it becomes a cesspool
- ssl-3 2y ago
  A brief guide on avoiding this pitfall, for search engine and LLM operators: Step 1: Punish ranking and visibility of sites whose llms.txt differs from a random sampling of actual web (HTML) content. Step 2: There is no step 2.
  - Retr0id 2y ago
    If you can reliably machine-diff format A and format B, there's no need for two different formats in the first place.
    - blharr 2y ago
      Right. And so you'll end up punishing some sites that do it perfectly, while other sites that game the algorithm game first priority.
  - Eisenstein 2y ago
    Brief guide for site optimizers: 1. Figure out how to embed content that only LLMs see which affect their output 2. Wait for that to stop working 3. Innovate another way to get past new technical problem
  - mystified5016 2y ago
    > No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race But maybe this time the naive technical solution will work
    - ssl-3 2y ago
      Everything is temporary. You, me, LLMs, Google, humanity, Earth, Sol... We can choose to carry on and perform [what we think are] improvements and make the best of it, or we can choose to cash it in early and simply give up.
nerdix 2y ago
I wish this were true too but it doesn't appear that llms.txt is meant to be a markdown version of the site's content based on the example on the site and after peeking at a few llms.txt files on some of the sites in the directory. It looks like its text meant to be fed into the llm as a system prompt specific to the site. The most simple ones just look like a sitemap restricted to documentation: https://www.activepieces.com/docs/llms.txt https://www.activepieces.com/docs/llms.txt Some interesting stuff is in some of them. Like this one that prompts the LLM to explain to the user the ethical issues of using AI agents along with a disclaimer: https://boehs.org/llms.txt https://boehs.org/llms.txt That site appears to be someone's blog and they don't seem like big fans of LLMs. https://boehs.org/node/llms-destroying-internet https://boehs.org/node/llms-destroying-internet A pretty clever use of llms.txt.

by caseyy 2y ago

Eisenstein 2y ago

The problem I see is that people will develop two versions of websites -- the LLM version, optimized to give the model a good impression of their products or services and get them into the training data, and the human version (with sub-versions for mobile and such) which will be SEOd to hell. No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race until * the incentives are removed * the process is made so technically complex or expensive that only a few players can profit from them * it is regulated such that people can make money doing other things which have better risk/reward * most people just avoid the whole ecosystem because it becomes a cesspool

ssl-3 2y ago
A brief guide on avoiding this pitfall, for search engine and LLM operators: Step 1: Punish ranking and visibility of sites whose llms.txt differs from a random sampling of actual web (HTML) content. Step 2: There is no step 2.
- Retr0id 2y ago
  If you can reliably machine-diff format A and format B, there's no need for two different formats in the first place.
  - blharr 2y ago
    Right. And so you'll end up punishing some sites that do it perfectly, while other sites that game the algorithm game first priority.
- Eisenstein 2y ago
  Brief guide for site optimizers: 1. Figure out how to embed content that only LLMs see which affect their output 2. Wait for that to stop working 3. Innovate another way to get past new technical problem
- mystified5016 2y ago
  > No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race But maybe this time the naive technical solution will work
  - ssl-3 2y ago
    Everything is temporary. You, me, LLMs, Google, humanity, Earth, Sol... We can choose to carry on and perform [what we think are] improvements and make the best of it, or we can choose to cash it in early and simply give up.

nerdix 2y ago

I wish this were true too but it doesn't appear that llms.txt is meant to be a markdown version of the site's content based on the example on the site and after peeking at a few llms.txt files on some of the sites in the directory. It looks like its text meant to be fed into the llm as a system prompt specific to the site. The most simple ones just look like a sitemap restricted to documentation: https://www.activepieces.com/docs/llms.txt https://www.activepieces.com/docs/llms.txt Some interesting stuff is in some of them. Like this one that prompts the LLM to explain to the user the ethical issues of using AI agents along with a disclaimer: https://boehs.org/llms.txt https://boehs.org/llms.txt That site appears to be someone's blog and they don't seem like big fans of LLMs. https://boehs.org/node/llms-destroying-internet https://boehs.org/node/llms-destroying-internet A pretty clever use of llms.txt.