5 ms·

I really vehemently disagree with the 'feedforward, tolerance, feedback' pattern. Protocols and standards like HTML built around "be liberal with what you acce

I really vehemently disagree with the 'feedforward, tolerance, feedback' pattern. Protocols and standards like HTML built around "be liberal with what you accept" have turned out to be a real nightmare. Best-guessing the intent of your caller is a path to subtle bugs and behavior that's difficult to reason about. If the LLM isn't doing a good job calling your api, then make the LLM get smarter or rebuild the api, don't make the API looser.

arscan 1y ago
> Protocols and standards like HTML built around "be liberal with what you accept" have turned out to be a real nightmare. This feels a bit like the setup to the “But you have heard of me” joke in Pirates of the Caribbean [2003].
- paulddraper 1y ago
  Or "There are only two kinds of languages: the ones people complain about and the ones nobody uses."
  - FridgeSeal 1y ago
    Sure, this holds for any government “current state” but it shouldn’t be used as an excuse to _keep doing dumb things_.
mort96 1y ago
I'm not sure it's possible to have a technology that's user-facing with multiple competing implementations, and not also, in some way, "liberal in what it accepts". Back when XHTML was somewhat hype and there were sites which actually used it, I recall being met with a big fat "XML parse error" page on occasion. If XHTML really took off (as in a significant majority of web pages were XHTML), those XML parse error pages would become way more common, simply because developers sometimes write bugs and many websites are server-generated with dynamic content. I'm 100% convinced that some browser would decide to implement special rules in their XML parser to try to recover from errors. And then, that browser would have a significant advantage in the market; users would start to notice, "sites which give me an XML Parse Error in Firefox work well in Chrome, so I'll switch to Chrome". And there you have the exact same problem as HTML, even though the standard itself is strict. The magical thing of HTML is that they managed to make a standard, HTML 5, which incorporates most of the special case rules as implemented by browsers. As such, all browsers would be lenient, but they'd all be lenient in the same way. A strict standard which mandates e.g "the document MUST be valid XML" results in implementations which are lenient, but they're lenient in different ways. HTML should arguably have been specified to be lenient from the start. Making a lenient standard from scratch is probably easier than trying to standardize commonalities between many differently-lenient implementations of a strict standard like what HTML had to do.
- lucideer 1y ago
  History has gone the way it went & we have HTML now, there's not much point harking back, but I still find it very odd that people today - with the wisdom of foresight - believe that the world opting for HTML & abandoning XHTML was the sensible choice. It seems odd to me that it's not seen as one of those "worse winning out" stories in the history of technology, like betamax. The main argument about XHTML not being "lenient" always centred around client UX of error display - Chrome even went on to actually implement a user-friendly partial-parse/partial-render handling of XHTML files that literally solved everyone's complaints via UI design without any spec changes but by this stage it was already too late. The whole story of why we went with HTML is somewhat hilarious: 1 guy wrote an ill informed blog post bitching about XHTML, generated a lot of hype, made zero concrete proposals to solve its problems, & then somehow convinced major browser makers (his current & former employers) to form an undemocratic rival group to the W3C, in which he was appointed dictator. An absolutely bizarre story for the ages, I do wish it was documented better but alas most of the resources around it were random dev blogs that link rotted.
  - integralid 1y ago
    >The whole story of Is that really the story? I think it was more like "backward compatible solution soon about more pure, theoretically better solution" There's enormous non-xhtml legacy than nobody wanted to port. And tooling back in the day didn't make it easy to write correct xhtml. Also like it or not, HTML is still written by humans sometimes, and they don't like parser blowing up because of a minor problem. Especially since such problems are often detected late, and a page which displays slightly wrong is much better outcome than the page blowing up.
    - throwaway346434 1y ago
      More or less? https://en.m.wikipedia.org/wiki/WHATWG https://en.m.wikipedia.org/wiki/WHATWG is fairly neutral. As someone in userland at the time on the other side of it, it was all a bit nuts. IE we got new standards invented out of thin air - https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Microdata https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Mic... - which ignored what hundreds had worked on before, which seemed to be driven by one person controlling the "standard" making it up as they went along. Microformats and RDFa were the more widely adopted solutions at the time, had a lot of design and thought put into them, worked with HTML4 (but thrived if used with xhtml), etc etc. JSON-LD/schema.org has now filled the niche and arguably it's a lot better for devs, but imagine how much better the "AI web UX" would be now if we'd just standardised earlier on one and stuck with it for those years? This is the main area where I saw the behaviour on display, where I interacted most. So the original comment feels absolutely in line with my recollections. I love bits of HTML5, but the way it congealed into reality isn't one of them.
wvenable 1y ago
HTML being lenient is what made progressive enhancement possible -- right down the original <img> tag. The web would not have existed at all if HTML was strict right from the start.
- arccy 1y ago
  That's poor reasoning. The web now counts as strict but still extensible: you just have to clearly define how to handle unknown input. The web treats all unknowns as a div.
  - wvenable 1y ago
    > you just have to clearly define how to handle unknown input. That is being lenient. Allowing any unknown input is being lenient in what you accept. Not allowing unknown input at all is being strict.
- dathinab 1y ago
  > progressive enhancement possible no not at all extensible isn't the same as lenient having a Content-Type header where you can put in new media types (e.g. for images) once browsers support it is extensibility sniffing the media type instead of strictly relying on the Content-Type header is leniency and had been the source of a lot of security vulnerabilities over the years or having new top level JS object exposing new APIs is extensibility but allowing overriding the prototypes of fundamental JS objects (i.e. Array.prototype) turned out to be a terrible idea associated with multiple security issues (like idk. ~10 years ago, hence why it now is read only) same for SAML, its use of XML made it extensible, but they way it leniently encoded XML for signing happened to be a security nightmare or OAuth2 which is very extensible, but it being too lenient in what you can combine how was the source of many early security incidents and is still source of incompatibilities today (but OAuth2 is anyway a mess)
  - wvenable 1y ago
    > no not at all extensible isn't the same as lenient I never said it was. But lenient provides for extensibility that isn't planned for. The entire evolution of the web is based on that. Standards that were too strict or too inflexible have been long forgotten by history. That's not to say that isn't the source of security vulnerabilities and bugs but that doesn't negate the point.
dathinab 1y ago
oh yes so true, but I would generalize it to "to flexible" - content type sniffing spawned a whole class of attacks, and should have been unnecessary - a ton of historic security issues where related to html parsing being too flexible, or some JS parts being to flexible (e.g. Array prototype override) - or login flows being too flexible creating a easy to overlook way to bypass (part of) login checks - or look at the mess OAuth2/OIDC had been for years because they insisted to over-enginer it and how especially it being liberal about quite many parts lead to more then one or two big security incidents - (more then strictly needed) cipher flexibility is by now widely accepted to have been an anti pattern - or how so much theoretically okay but "old" security tech is such a pain to use because it was made to be supper tolerant to everything, like every use case imaginable, every combination of parameters, every kind of partial uninterpretable parts (I'm looking at you ASN.1, X509 certs and many old CA software, theoretically really not bad designed, practically such a pain). And sure you also can be too strict, high cipher flexibility being an anti-pattern was incorporated into TLS 1.3. But TLS still needs some cipher flexibility, so they fund a compromise of (oversimplified) you can choose 1 of 5 cipher suites but can't change any parameter of that suites. Just today I read an article (at work, I don't have the link at hand) about some so hypothetical but practically probably doable (with a bunch of more work) scenarios to trick very flexible multi step agents into leaking your secrets. The core approach was that they found a way to have a relative small snippet of text which if it end up in the context has a high chance to basically override the whole context with just your instruction (quite a bit oversimplified). In turn if you can sneak it into someones queries (e.g. you GTP model is allowed to read you mails and it's in a mail send to you) you can then trick the multi step model to grab a secret from your computer (because the agents often run with user permissions) and send it to you (by e.g. instrumenting the agent to scan a website under an url which happens to now contain the secret). Its a bit hypothetical, its hard to pull of, but it's very well in the realm of possibility due to how content and instructions are on a very fundamental level not cleanly separated (I mean AI vendors do try, but so far that never worked reliable it's in the end all the same input).

arscan 1y ago

> Protocols and standards like HTML built around "be liberal with what you accept" have turned out to be a real nightmare. This feels a bit like the setup to the “But you have heard of me” joke in Pirates of the Caribbean [2003].

paulddraper 1y ago
Or "There are only two kinds of languages: the ones people complain about and the ones nobody uses."
- FridgeSeal 1y ago
  Sure, this holds for any government “current state” but it shouldn’t be used as an excuse to _keep doing dumb things_.

mort96 1y ago

I'm not sure it's possible to have a technology that's user-facing with multiple competing implementations, and not also, in some way, "liberal in what it accepts". Back when XHTML was somewhat hype and there were sites which actually used it, I recall being met with a big fat "XML parse error" page on occasion. If XHTML really took off (as in a significant majority of web pages were XHTML), those XML parse error pages would become way more common, simply because developers sometimes write bugs and many websites are server-generated with dynamic content. I'm 100% convinced that some browser would decide to implement special rules in their XML parser to try to recover from errors. And then, that browser would have a significant advantage in the market; users would start to notice, "sites which give me an XML Parse Error in Firefox work well in Chrome, so I'll switch to Chrome". And there you have the exact same problem as HTML, even though the standard itself is strict. The magical thing of HTML is that they managed to make a standard, HTML 5, which incorporates most of the special case rules as implemented by browsers. As such, all browsers would be lenient, but they'd all be lenient in the same way. A strict standard which mandates e.g "the document MUST be valid XML" results in implementations which are lenient, but they're lenient in different ways. HTML should arguably have been specified to be lenient from the start. Making a lenient standard from scratch is probably easier than trying to standardize commonalities between many differently-lenient implementations of a strict standard like what HTML had to do.

lucideer 1y ago
History has gone the way it went & we have HTML now, there's not much point harking back, but I still find it very odd that people today - with the wisdom of foresight - believe that the world opting for HTML & abandoning XHTML was the sensible choice. It seems odd to me that it's not seen as one of those "worse winning out" stories in the history of technology, like betamax. The main argument about XHTML not being "lenient" always centred around client UX of error display - Chrome even went on to actually implement a user-friendly partial-parse/partial-render handling of XHTML files that literally solved everyone's complaints via UI design without any spec changes but by this stage it was already too late. The whole story of why we went with HTML is somewhat hilarious: 1 guy wrote an ill informed blog post bitching about XHTML, generated a lot of hype, made zero concrete proposals to solve its problems, & then somehow convinced major browser makers (his current & former employers) to form an undemocratic rival group to the W3C, in which he was appointed dictator. An absolutely bizarre story for the ages, I do wish it was documented better but alas most of the resources around it were random dev blogs that link rotted.
- integralid 1y ago
  >The whole story of Is that really the story? I think it was more like "backward compatible solution soon about more pure, theoretically better solution" There's enormous non-xhtml legacy than nobody wanted to port. And tooling back in the day didn't make it easy to write correct xhtml. Also like it or not, HTML is still written by humans sometimes, and they don't like parser blowing up because of a minor problem. Especially since such problems are often detected late, and a page which displays slightly wrong is much better outcome than the page blowing up.
  - throwaway346434 1y ago
    More or less? https://en.m.wikipedia.org/wiki/WHATWG https://en.m.wikipedia.org/wiki/WHATWG is fairly neutral. As someone in userland at the time on the other side of it, it was all a bit nuts. IE we got new standards invented out of thin air - https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Microdata https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Mic... - which ignored what hundreds had worked on before, which seemed to be driven by one person controlling the "standard" making it up as they went along. Microformats and RDFa were the more widely adopted solutions at the time, had a lot of design and thought put into them, worked with HTML4 (but thrived if used with xhtml), etc etc. JSON-LD/schema.org has now filled the niche and arguably it's a lot better for devs, but imagine how much better the "AI web UX" would be now if we'd just standardised earlier on one and stuck with it for those years? This is the main area where I saw the behaviour on display, where I interacted most. So the original comment feels absolutely in line with my recollections. I love bits of HTML5, but the way it congealed into reality isn't one of them.

wvenable 1y ago

HTML being lenient is what made progressive enhancement possible -- right down the original <img> tag. The web would not have existed at all if HTML was strict right from the start.

arccy 1y ago
That's poor reasoning. The web now counts as strict but still extensible: you just have to clearly define how to handle unknown input. The web treats all unknowns as a div.
- wvenable 1y ago
  > you just have to clearly define how to handle unknown input. That is being lenient. Allowing any unknown input is being lenient in what you accept. Not allowing unknown input at all is being strict.
dathinab 1y ago
> progressive enhancement possible no not at all extensible isn't the same as lenient having a Content-Type header where you can put in new media types (e.g. for images) once browsers support it is extensibility sniffing the media type instead of strictly relying on the Content-Type header is leniency and had been the source of a lot of security vulnerabilities over the years or having new top level JS object exposing new APIs is extensibility but allowing overriding the prototypes of fundamental JS objects (i.e. Array.prototype) turned out to be a terrible idea associated with multiple security issues (like idk. ~10 years ago, hence why it now is read only) same for SAML, its use of XML made it extensible, but they way it leniently encoded XML for signing happened to be a security nightmare or OAuth2 which is very extensible, but it being too lenient in what you can combine how was the source of many early security incidents and is still source of incompatibilities today (but OAuth2 is anyway a mess)
- wvenable 1y ago
  > no not at all extensible isn't the same as lenient I never said it was. But lenient provides for extensibility that isn't planned for. The entire evolution of the web is based on that. Standards that were too strict or too inflexible have been long forgotten by history. That's not to say that isn't the source of security vulnerabilities and bugs but that doesn't negate the point.

dathinab 1y ago

oh yes so true, but I would generalize it to "to flexible" - content type sniffing spawned a whole class of attacks, and should have been unnecessary - a ton of historic security issues where related to html parsing being too flexible, or some JS parts being to flexible (e.g. Array prototype override) - or login flows being too flexible creating a easy to overlook way to bypass (part of) login checks - or look at the mess OAuth2/OIDC had been for years because they insisted to over-enginer it and how especially it being liberal about quite many parts lead to more then one or two big security incidents - (more then strictly needed) cipher flexibility is by now widely accepted to have been an anti pattern - or how so much theoretically okay but "old" security tech is such a pain to use because it was made to be supper tolerant to everything, like every use case imaginable, every combination of parameters, every kind of partial uninterpretable parts (I'm looking at you ASN.1, X509 certs and many old CA software, theoretically really not bad designed, practically such a pain). And sure you also can be too strict, high cipher flexibility being an anti-pattern was incorporated into TLS 1.3. But TLS still needs some cipher flexibility, so they fund a compromise of (oversimplified) you can choose 1 of 5 cipher suites but can't change any parameter of that suites. Just today I read an article (at work, I don't have the link at hand) about some so hypothetical but practically probably doable (with a bunch of more work) scenarios to trick very flexible multi step agents into leaking your secrets. The core approach was that they found a way to have a relative small snippet of text which if it end up in the context has a high chance to basically override the whole context with just your instruction (quite a bit oversimplified). In turn if you can sneak it into someones queries (e.g. you GTP model is allowed to read you mails and it's in a mail send to you) you can then trick the multi step model to grab a secret from your computer (because the agents often run with user permissions) and send it to you (by e.g. instrumenting the agent to scan a website under an url which happens to now contain the secret). Its a bit hypothetical, its hard to pull of, but it's very well in the realm of possibility due to how content and instructions are on a very fundamental level not cleanly separated (I mean AI vendors do try, but so far that never worked reliable it's in the end all the same input).