4 ms·

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was desig

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is needed. I think it might be more useful to define script type=llm that would expose function calling to LLMs embedded in browsers.

Joker_vD 2y ago
If only this RFC was well-known among the people who actually put stuff out on the Web.
- seeknotfind 2y ago
  How about an LLM agent which automatically finds inconsistencies in RFCs.
- russellbeattie 2y ago
  If only that RFC didn't make it a hidden directory. I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.
  - CuriousCosmic 2y ago
    The reason is because it's supposed to be a standard folder that isn't in use accidentally for other purposes. It's exceedingly unlikely that a website is going to just happen to make content available in a hidden directory path without it being created by automated tooling (which would likely be aware of such standards). The entire point is to avoid adopting a path that people already publicly use for something else. A hidden directory is the best way to do that.
  - ftmch 2y ago
    Also "well-known" was always such an awkward name to me.
    - tbrownaw 2y ago
      It's a joke about things being described as needing to be put at "well known" paths.
  - darrenf 2y ago
    > If only that RFC didn't make it a hidden directory. There's zero guidance on configuring how URIs under `/.well-known/` should be served at all, is there? They just reserve/sandbox the initial path component for the URI schemes which support it. That's it. It's the developers' choice to implement it as a directory - hidden or otherwise - on a filesystem; neither RFC says they SHOULD or MUST be served in such a way. (The updated RFC says "e.g., on a filesystem" in section 4.1, and mentions directories in section 4.4 in a way that, to my eyes, pretty much recommends against making it hidden)
  - teddyh 2y ago
    A URL path is not a directory path; there is no reason to assume that a path must be served by a directory by the same name. I mean, do you assume that there somewhere exists an actual machine with its Unix hostname set to “news.ycombinator.com”?
    - baby_souffle 2y ago
      > there is no reason to assume that a path must be served by a directory by the same name Other than static sites, sure.
      - bawolff 2y ago
        Even in static sites, rewrite rules are a thing.
      - teddyh 2y ago
        Even static sites might be served out of a simple hash table in RAM. A URL path is not a file path: <https://www.w3.org/Provider/Style/URI https://www.w3.org/Provider/Style/URI>
- arthurcolle 2y ago
  OpenAI was good about using well known for plugins
  - jo32 2y ago
    Any example?
    - nl 2y ago
      Look in the URL field in all the files in https://github.com/cfortuner/wellknown/tree/main/plugins https://github.com/cfortuner/wellknown/tree/main/plugins Eg: https://www.wolframalpha.com/.well-known/apispec.json https://www.wolframalpha.com/.well-known/apispec.json
    - arthurcolle 2y ago
      they did a good attempt but I think they just realized (at same pace as people like me) that you can just grab all this shit from OpenAPI and not worry about defining custom format
- bawolff 2y ago
  I think it is. Lots of actually used standards use it. I see it in the wild all the time. Certainly more well used than content negotiation which grandparent also mentioned.
- nl 2y ago
  It's very broadly used. For example: - Apple uses it for their app to website association files - OpenID Connect uses it for connection discovery - security.txt is usually served from .well-known - JSON Web Tokens uses it for Web Key Sets to verify public keys - LetsEncrypt uses it for its ACME HTTPS verification protocol. There's probably more, but these are ones I've personally used.
samstave 2y ago
What if each LLM had to register and tell you when/where/what it scraped/ingested from your site/page/url? And you could look at whatever your LLM trigger log looks like, and have a DELETE_ME link. (right_to_be_un_vectorized)
CuriousCosmic 2y ago
Fwiw the active RFC is 8615 as RFC5785 is obsolete. https://datatracker.ietf.org/doc/html/rfc8615 https://datatracker.ietf.org/doc/html/rfc8615
- pasiaj 2y ago
  Having two different RFCs on this is clearly causing confusion! I think I'll create a new RFC to supersede both to clear up the situation.
KTibow 2y ago
It's based on the series of robots.txt, humans.txt, security.txt, etc
- JimDabell 2y ago
  Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as soon as they were announced and ignored people).
- 13hunteo 2y ago
  FWIW, security.txt is placed in .well-known/
Tepix 2y ago
I think the whole idea of extra instructions required for LLMs is unnecessary. A decent LLM should be able to handle browsing the site, if needed it can use the sitemap. It can hopefully also figure out what the various sections are about.
- sbergot 2y ago
  I agree. There are enough standard places to put metadata in a website.
- crimsoneer 2y ago
  But why would we waste that many tokens on objects that LLms dont care about? Humans appreciate beauty. LLMs do not. Why are we wasting effort?
  - anonzzzies 2y ago
    > Humans appreciate beauty Some humans do for websites; I personally couldn't care less and I find it often just annoying / in the way. I wish all sites where just black on white or the reverse and with clear interaction elements (including for saas sites). I welcome the near future where I can say; 'show me all important sentry issues, ah yes, make an issue in github to to fix this one and just make the rest resolved' instead of having to click through a myriad of useless and often confusing 'UX' and 'beauty' just to do things. Non saas sites I just visit to read so I immediately ram the reader-mode button which shows the site indeed as I want.
    - dotancohen 2y ago
      I wish that the website would provide the content, and my user agent would provide my preferred styling e.g. fonts, colours, line spacing, etc.
- addandsubtract 2y ago
  Anyone else remember when websites had sitemaps? I miss those days.
deleted 2y ago
[deleted]
lolinder 2y ago
OP replied to this complaint in a top-level comment: https://news.ycombinator.com/item?id=41442092 https://news.ycombinator.com/item?id=41442092 They're correct that the RFC technically requires registration, and looking through the existing list of contact information for registrations I'd be likewise intimidated to attempt to register something that's experimental.

Joker_vD 2y ago

If only this RFC was well-known among the people who actually put stuff out on the Web.

seeknotfind 2y ago
How about an LLM agent which automatically finds inconsistencies in RFCs.
russellbeattie 2y ago
If only that RFC didn't make it a hidden directory. I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.
- CuriousCosmic 2y ago
  The reason is because it's supposed to be a standard folder that isn't in use accidentally for other purposes. It's exceedingly unlikely that a website is going to just happen to make content available in a hidden directory path without it being created by automated tooling (which would likely be aware of such standards). The entire point is to avoid adopting a path that people already publicly use for something else. A hidden directory is the best way to do that.
- ftmch 2y ago
  Also "well-known" was always such an awkward name to me.
  - tbrownaw 2y ago
    It's a joke about things being described as needing to be put at "well known" paths.
- darrenf 2y ago
  > If only that RFC didn't make it a hidden directory. There's zero guidance on configuring how URIs under `/.well-known/` should be served at all, is there? They just reserve/sandbox the initial path component for the URI schemes which support it. That's it. It's the developers' choice to implement it as a directory - hidden or otherwise - on a filesystem; neither RFC says they SHOULD or MUST be served in such a way. (The updated RFC says "e.g., on a filesystem" in section 4.1, and mentions directories in section 4.4 in a way that, to my eyes, pretty much recommends against making it hidden)
- teddyh 2y ago
  A URL path is not a directory path; there is no reason to assume that a path must be served by a directory by the same name. I mean, do you assume that there somewhere exists an actual machine with its Unix hostname set to “news.ycombinator.com”?
  - baby_souffle 2y ago
    > there is no reason to assume that a path must be served by a directory by the same name Other than static sites, sure.
    - bawolff 2y ago
      Even in static sites, rewrite rules are a thing.
    - teddyh 2y ago
      Even static sites might be served out of a simple hash table in RAM. A URL path is not a file path: <https://www.w3.org/Provider/Style/URI https://www.w3.org/Provider/Style/URI>
arthurcolle 2y ago
OpenAI was good about using well known for plugins
- jo32 2y ago
  Any example?
  - nl 2y ago
    Look in the URL field in all the files in https://github.com/cfortuner/wellknown/tree/main/plugins https://github.com/cfortuner/wellknown/tree/main/plugins Eg: https://www.wolframalpha.com/.well-known/apispec.json https://www.wolframalpha.com/.well-known/apispec.json
  - arthurcolle 2y ago
    they did a good attempt but I think they just realized (at same pace as people like me) that you can just grab all this shit from OpenAPI and not worry about defining custom format
bawolff 2y ago
I think it is. Lots of actually used standards use it. I see it in the wild all the time. Certainly more well used than content negotiation which grandparent also mentioned.
nl 2y ago
It's very broadly used. For example: - Apple uses it for their app to website association files - OpenID Connect uses it for connection discovery - security.txt is usually served from .well-known - JSON Web Tokens uses it for Web Key Sets to verify public keys - LetsEncrypt uses it for its ACME HTTPS verification protocol. There's probably more, but these are ones I've personally used.

samstave 2y ago

What if each LLM had to register and tell you when/where/what it scraped/ingested from your site/page/url? And you could look at whatever your LLM trigger log looks like, and have a DELETE_ME link. (right_to_be_un_vectorized)

CuriousCosmic 2y ago

Fwiw the active RFC is 8615 as RFC5785 is obsolete. https://datatracker.ietf.org/doc/html/rfc8615 https://datatracker.ietf.org/doc/html/rfc8615

pasiaj 2y ago
Having two different RFCs on this is clearly causing confusion! I think I'll create a new RFC to supersede both to clear up the situation.

KTibow 2y ago

It's based on the series of robots.txt, humans.txt, security.txt, etc

JimDabell 2y ago
Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as soon as they were announced and ignored people).
13hunteo 2y ago
FWIW, security.txt is placed in .well-known/

Tepix 2y ago

I think the whole idea of extra instructions required for LLMs is unnecessary. A decent LLM should be able to handle browsing the site, if needed it can use the sitemap. It can hopefully also figure out what the various sections are about.

sbergot 2y ago
I agree. There are enough standard places to put metadata in a website.
crimsoneer 2y ago
But why would we waste that many tokens on objects that LLms dont care about? Humans appreciate beauty. LLMs do not. Why are we wasting effort?
- anonzzzies 2y ago
  > Humans appreciate beauty Some humans do for websites; I personally couldn't care less and I find it often just annoying / in the way. I wish all sites where just black on white or the reverse and with clear interaction elements (including for saas sites). I welcome the near future where I can say; 'show me all important sentry issues, ah yes, make an issue in github to to fix this one and just make the rest resolved' instead of having to click through a myriad of useless and often confusing 'UX' and 'beauty' just to do things. Non saas sites I just visit to read so I immediately ram the reader-mode button which shows the site indeed as I want.
  - dotancohen 2y ago
    I wish that the website would provide the content, and my user agent would provide my preferred styling e.g. fonts, colours, line spacing, etc.
addandsubtract 2y ago
Anyone else remember when websites had sitemaps? I miss those days.

deleted 2y ago

[deleted]

lolinder 2y ago

OP replied to this complaint in a top-level comment: https://news.ycombinator.com/item?id=41442092 https://news.ycombinator.com/item?id=41442092 They're correct that the RFC technically requires registration, and looking through the existing list of contact information for registrations I'd be likewise intimidated to attempt to register something that's experimental.