4 ms·

llms.txt makes a lot of sense, especially for LLMs to interact with http APIs autonomously. Seems like you could set a LLM loose and like the Google Bot have i

llms.txt makes a lot of sense, especially for LLMs to interact with http APIs autonomously. Seems like you could set a LLM loose and like the Google Bot have it start converting all html pages into llms.txt. Man, the future is crazy.

llms-txt 1y ago
[dead]
nothrabannosir 1y ago
Couldn’t believe my eyes. The www is truly bankrupt. If anyone has a browser plugin which automatically redirects to llms.txt sign me up. Website too confusing for humans? Add more design, modals, newsletter pop ups, cookie banners, ads, … Website too confusing for LLMs? Add an accessible, clean, ad-free, concise, high entropy, plain text summary of your website. Make sure to hide it from the humans! PS: it should be /.well-known/llms.txt but that feels futile at this point.. PPS: I enjoyed the talk, thanks.
- andrethegiant 1y ago
  > If anyone has a browser plugin which automatically redirects to llms.txt sign me up. Not a browser plugin, but you can prefix URLs with `pure.md/` to get the pure markdown of that page. It's not quite a 1:1 to llms.txt as it doesn't explain the entire domain, but works well for one-off pages. [disclaimer: I'm the maintainer]
  - FergusArgyll 1y ago
    I've been actually using it for my own consumption (I am not an llm...) It's great! thanks
- jph00 1y ago
  The next version of the llms.txt proposal will allow an llms.txt file to be added at any level of a path, which isn't compatible with /.well-known. (I'm the creator of the llms.txt proposal.)
  - nothrabannosir 1y ago
    [flagged]
    - dang 1y ago
      "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - nothrabannosir 1y ago
        Fair
        nothrabannosir 1y ago
        PS apologies to jph00. I still believe what I believe but I should have phrased it differently or not at all. Good luck on your endeavors either way.
  - achempion 1y ago
    Even with this future approach, it still can live under the `/.well-known`, think of `/.well-known/llm/<mirrored path>` or `/.well-known/llm.json` with key/value mappings.
  - andrethegiant 1y ago
    Doesn’t this conflict with the original proposal of appending .md to any resource, e.g. /foo/bar.html.md? Or why not tell servers to respond to the Accept header when it’s set to text/markdown?
- alightsoul 1y ago
  The web started dying with mobile social media apps, in which hyperlinks are a poor UX choice. Then again with SEO banning outlinks. Now this. The web of interconnected pages that was the World Wide Web is dead. Not on social media? No one sees you. Run a website? more bots than humans. Unless you sell something on the side with the website it's not profitable. Hyperlinking to other websites is dead. Gen Alpha doesn't know what a web page is and if they do, it's for stuff like neocities aka as a curiosity or art form only. Not as a source of information anymore. I don't blame them. Apps (social media apps) have less friction than web sites but have a higher barrier for people to create. We are going back to pre World Wide Web days in a way, kind of like Bulletin Board Systems on dial up without hyperlinking, and centralized (social media) Some countries mostly ones with few technical people llike the ones in Central America have moved away from the web almost entirely and into social media like Instagram. Due to the death of the web, google search and friends now rely mostly on matching queries with titles now so just like before the internet you have to know people to learn new stuff or wait for an algorithm to show it to you or someone to comment it online or forcefully enroll in a university. Maybe that's why search results have declined and poeple search using ChatGPT or maybe perplexity. Scholarly search engines are a bit better but frankly irrelevant for most poeple. Now I understand why Google established their own DNS server at 8.8.8.8. If you have a directory of all domains on DNS, you can still index sites without hyperlinks between them, even if the web dies. They saw it coming.
  - zelphirkalt 1y ago
    This. Only recently I realized, that in China for example many young people do not have a browser on their phone. Everything they do is via messenger "mini apps", which I can't imagine to be easier to create than a website. At every restaurant you get a QR code to scan, to download a so called mini app, which is a website in disguise. They run (un)social media apps like Tiktok or Xiaohongshu as their information input. I am not even sure they still use Baidu or something else as a search engine. Maybe that is another app, but they don't have a browser installed to go to a search engine's website. I think it is an extremely debilitating situation and it results in people not even knowing what a website is. What it consists of, or how one could possibly make one oneself. They would have to go straight to app development and have it in big tech's stores, in order to make anything their peers could see or use.
practal 1y ago
If you have different representations of the same thing (llms.txt / HTML), how do you know it is actually equivalent to each other? I am wondering if there are scenarios where webpage publishers would be interested in gaming this.
- andrethegiant 1y ago
  <link rel="alternate" /> is a standards-friendly way to semantically represent the same content in a different format
  - diggan 1y ago
    Also HTTP headers Accept/Content-Type which in theory could let you serve HTML, XML and JSON all under the same URL/URI but depending on Accept values.
- jph00 1y ago
  That's not what llms.txt is. You can just use a regular markdown URL or similar for that. llms.txt is a description for an LLM of how to find the information on your site needed for an LLM to use your product or service effectively.

llms-txt 1y ago

[dead]

nothrabannosir 1y ago

Couldn’t believe my eyes. The www is truly bankrupt. If anyone has a browser plugin which automatically redirects to llms.txt sign me up. Website too confusing for humans? Add more design, modals, newsletter pop ups, cookie banners, ads, … Website too confusing for LLMs? Add an accessible, clean, ad-free, concise, high entropy, plain text summary of your website. Make sure to hide it from the humans! PS: it should be /.well-known/llms.txt but that feels futile at this point.. PPS: I enjoyed the talk, thanks.

andrethegiant 1y ago
> If anyone has a browser plugin which automatically redirects to llms.txt sign me up. Not a browser plugin, but you can prefix URLs with `pure.md/` to get the pure markdown of that page. It's not quite a 1:1 to llms.txt as it doesn't explain the entire domain, but works well for one-off pages. [disclaimer: I'm the maintainer]
- FergusArgyll 1y ago
  I've been actually using it for my own consumption (I am not an llm...) It's great! thanks
jph00 1y ago
The next version of the llms.txt proposal will allow an llms.txt file to be added at any level of a path, which isn't compatible with /.well-known. (I'm the creator of the llms.txt proposal.)
- nothrabannosir 1y ago
  [flagged]
  - dang 1y ago
    "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - nothrabannosir 1y ago
      Fair
      - nothrabannosir 1y ago
        PS apologies to jph00. I still believe what I believe but I should have phrased it differently or not at all. Good luck on your endeavors either way.
- achempion 1y ago
  Even with this future approach, it still can live under the `/.well-known`, think of `/.well-known/llm/<mirrored path>` or `/.well-known/llm.json` with key/value mappings.
- andrethegiant 1y ago
  Doesn’t this conflict with the original proposal of appending .md to any resource, e.g. /foo/bar.html.md? Or why not tell servers to respond to the Accept header when it’s set to text/markdown?
alightsoul 1y ago
The web started dying with mobile social media apps, in which hyperlinks are a poor UX choice. Then again with SEO banning outlinks. Now this. The web of interconnected pages that was the World Wide Web is dead. Not on social media? No one sees you. Run a website? more bots than humans. Unless you sell something on the side with the website it's not profitable. Hyperlinking to other websites is dead. Gen Alpha doesn't know what a web page is and if they do, it's for stuff like neocities aka as a curiosity or art form only. Not as a source of information anymore. I don't blame them. Apps (social media apps) have less friction than web sites but have a higher barrier for people to create. We are going back to pre World Wide Web days in a way, kind of like Bulletin Board Systems on dial up without hyperlinking, and centralized (social media) Some countries mostly ones with few technical people llike the ones in Central America have moved away from the web almost entirely and into social media like Instagram. Due to the death of the web, google search and friends now rely mostly on matching queries with titles now so just like before the internet you have to know people to learn new stuff or wait for an algorithm to show it to you or someone to comment it online or forcefully enroll in a university. Maybe that's why search results have declined and poeple search using ChatGPT or maybe perplexity. Scholarly search engines are a bit better but frankly irrelevant for most poeple. Now I understand why Google established their own DNS server at 8.8.8.8. If you have a directory of all domains on DNS, you can still index sites without hyperlinks between them, even if the web dies. They saw it coming.
- zelphirkalt 1y ago
  This. Only recently I realized, that in China for example many young people do not have a browser on their phone. Everything they do is via messenger "mini apps", which I can't imagine to be easier to create than a website. At every restaurant you get a QR code to scan, to download a so called mini app, which is a website in disguise. They run (un)social media apps like Tiktok or Xiaohongshu as their information input. I am not even sure they still use Baidu or something else as a search engine. Maybe that is another app, but they don't have a browser installed to go to a search engine's website. I think it is an extremely debilitating situation and it results in people not even knowing what a website is. What it consists of, or how one could possibly make one oneself. They would have to go straight to app development and have it in big tech's stores, in order to make anything their peers could see or use.

practal 1y ago

If you have different representations of the same thing (llms.txt / HTML), how do you know it is actually equivalent to each other? I am wondering if there are scenarios where webpage publishers would be interested in gaming this.

andrethegiant 1y ago
<link rel="alternate" /> is a standards-friendly way to semantically represent the same content in a different format
- diggan 1y ago
  Also HTTP headers Accept/Content-Type which in theory could let you serve HTML, XML and JSON all under the same URL/URI but depending on Accept values.
jph00 1y ago
That's not what llms.txt is. You can just use a regular markdown URL or similar for that. llms.txt is a description for an LLM of how to find the information on your site needed for an LLM to use your product or service effectively.