3 ms·

The IETF title makes it sound like developers everywhere were using Postel's words as guidance. Instead, relaxed or permissive parsing of files/protocols/API e

by jasode 9y ago

The IETF title makes it sound like developers everywhere were using Postel's words as guidance. Instead, relaxed or permissive parsing of files/protocols/API even though they have well-documented specifications is an unavoidable emergent phenomenon. The various examples across domains is fascinating: - HTML (missing tags, unpaired tags, etc). Developers may prefer to strictly parse HTML and any non-compliance results in a "page error". But the websurfers want that page data so developers end up guessing the HTML authors intentions and renders the page. - PDF (malformed pdf files that Adobe can read). Lots of utilities out there create bad pdf files and Adobe Acrobat has lots of workarounds to read them. Even though Adobe Inc controls the pdf specification, even they succumb to the pressure of adding workarounds to their parser to render broken pdf files. To add to the insanity, all 3rd-party industrial-strength pdf parsers end up copying Adobe Acrobat's behavior to parse broken pdf files! - Win32 API. Programmers out in the wild will notice an undocumented behavior of an API that's not in the contract and start to depend on it. When a new version of Windows is released that removes the undocumented behavior and breaks the 3rd-party app, the customer blames the "Microsoft Windows upgrade" and not Quicken or videogame company. Raymond Chen has written several articles on Microsoft adding "shims" to Windows codebase to help "guarantee compatibility" for misbehaving apps incorrectly using the Win32 API. If the 3rd-party app is important enough to consumers that it prevents them from upgrading Windows (in other words -- pay Microsoft money), it means Microsoft will bend over backwards to accommodate the badly-written code from the vendor. The external forces are too great to enforce perfect discipline of strict parsing across all parties. Even the big proprietary companies like Microsoft and Adobe can't enforce their own standards-compliant parsing. An open standard from IETF would have no chance at all. We'd like to think of a "computing standard" as some inviolable contract but history has shown it is actually an organic (and unspoken) "social" agreement. This is unavoidable unless we all agree to have all software "approved" by a central authority before anyone can download or use it.

dozzie 9y ago
> Developers may prefer to strictly parse HTML and any non-compliance results in a "page error". But the websurfers want that page data [...] Note that similar thing was happening with SSL/TLS connections for a long time: developers may prefer to strictly verify X.509 certificates, but websurfers want that page data. The result of developers bowing to accepting invalid certificates was hurting everybody. Nowadays the situation is much better, because browser developers made visiting sites with self-signed, expired, or incorrectly named certificates significantly harder.
- jasode 9y ago
 >developers may prefer to strictly verify X.509 certificates, but websurfers want that page data. I categorize that scenario in a different bucket because novice users don't understand the security implications of broken TLS. So, the users don't want that data but they don't know it. The developers in this case are looking out for the user. (Same security situation as developers helping the user by restricting cross-site scripting or address bar hijacking.) To me, that's not the same as the forgiving parsers of HTML and PDF. Whether the HTML is missing closing tags like this: This is paragraph 1. This is paragraph 2. Or is 100% compliant like this: This is paragraph 1. This is paragraph 2. ... the websurfer just wants the page rendered in both cases.
 - dozzie 9y ago
 > To me, that's not the same as the forgiving parsers of HTML and PDF. Oh, of course it's not the same. The point is, in both cases users want strictness, but don't know that they want. Users want web pages that are easy for their browsers to parse; "want" to different degree, yes, even an indirect "want", but still. If browsers fail to display web page with invalid markup, developers are forced to fix their sh&t instead of adding another exception to an already big pile of "steaming let's not" in the standard. And we had a precedent of going down this way with X.509 certificates. I think it's a pity that the way is too hard for something not as critical as SSL/TLS.
 - inimino 9y ago
 Incidentally, both are 100% compliant, since closing tags on some elements have always been optional in HTML.
 - anowicki 9y ago
 Your first example is also probably 100% compliant HTML (it depends on what tag is second immediately followed by): https://www.w3.org/TR/html/syntax.html#optional-tags https://www.w3.org/TR/html/syntax.html#optional-tags
im3w1l 9y ago
It may also be easier to write a lax parser. Say some spec says a file must start with a magic number. Do you verify the number or just skip ahead? Say there is some metadata in the file you aren't interested in. Do you verify that it's valid before ignoring it?

by jasode 9y ago

dozzie 9y ago

> Developers may prefer to strictly parse HTML and any non-compliance results in a "page error". But the websurfers want that page data [...] Note that similar thing was happening with SSL/TLS connections for a long time: developers may prefer to strictly verify X.509 certificates, but websurfers want that page data. The result of developers bowing to accepting invalid certificates was hurting everybody. Nowadays the situation is much better, because browser developers made visiting sites with self-signed, expired, or incorrectly named certificates significantly harder.

jasode 9y ago
>developers may prefer to strictly verify X.509 certificates, but websurfers want that page data. I categorize that scenario in a different bucket because novice users don't understand the security implications of broken TLS. So, the users don't want that data but they don't know it. The developers in this case are looking out for the user. (Same security situation as developers helping the user by restricting cross-site scripting or address bar hijacking.) To me, that's not the same as the forgiving parsers of HTML and PDF. Whether the HTML is missing closing tags like this: This is paragraph 1. This is paragraph 2. Or is 100% compliant like this: This is paragraph 1. This is paragraph 2. ... the websurfer just wants the page rendered in both cases.
- dozzie 9y ago
 > To me, that's not the same as the forgiving parsers of HTML and PDF. Oh, of course it's not the same. The point is, in both cases users want strictness, but don't know that they want. Users want web pages that are easy for their browsers to parse; "want" to different degree, yes, even an indirect "want", but still. If browsers fail to display web page with invalid markup, developers are forced to fix their sh&t instead of adding another exception to an already big pile of "steaming let's not" in the standard. And we had a precedent of going down this way with X.509 certificates. I think it's a pity that the way is too hard for something not as critical as SSL/TLS.
- inimino 9y ago
 Incidentally, both are 100% compliant, since closing tags on some elements have always been optional in HTML.
- anowicki 9y ago
 Your first example is also probably 100% compliant HTML (it depends on what tag is second immediately followed by): https://www.w3.org/TR/html/syntax.html#optional-tags https://www.w3.org/TR/html/syntax.html#optional-tags

im3w1l 9y ago

It may also be easier to write a lax parser. Say some spec says a file must start with a magic number. Do you verify the number or just skip ahead? Say there is some metadata in the file you aren't interested in. Do you verify that it's valid before ignoring it?