6 ms·

You can't in general case. But you can in lots of typical cases. Actually real world HTML usually can't be parsed by any strict parser, as it's not valid. It's

You can't in general case. But you can in lots of typical cases. Actually real world HTML usually can't be parsed by any strict parser, as it's not valid. It's just a machine-generated text which pretends to be similar to HTML. So extracting some bits of information with regexes often works.

lifthrasiir 6y ago
I believe you really meant that you are frequently dealing with HTML which structure is already known in advance, not the general HTML. Because... > [...] real world HTML usually can't be parsed by any strict parser [...] There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. Just that you don't always need the full HTML parser to do your job. [1] https://html.spec.whatwg.org/multipage/parsing.html https://html.spec.whatwg.org/multipage/parsing.html
- tomashubelbauer 6y ago
  I think this is only true for HTML5, but previous versions of HTML supposedly weren't specced well enough to write a prefect parser. Fixing this was one of the goals of the HTML5 revision if I'm not mistaken.
  - lifthrasiir 6y ago
    You are correct, but I don't think they are even slightly relevant in 2021.
  - bawolff 6y ago
    Web browsers exist, therefore its possible to parse html, even earlier versions. You're right that the de jure spec did not match de facto html, and browsers didn't neccesarily agree with each other. But that's always true. GCC has language extensions that aren't part of the c spec, but you wouldn't say that c is impossible to parse. Old html may have taken it up to 11, but its not fair to say its impossible to parse.
    - chrismorgan 6y ago
      Perl is famously unparseable: it’s impossible to determine a parse tree without executing the code. (HTML, however, was never unparseable, merely insufficiently defined.)
    - uuidgen 6y ago
      No, not really. The browsers did guess a lot and did standard-deviating parsing because the typical uses were wrong and they had to work. Nobody would switch to a new browser that doesn't work with existing pages. Modern example - mXSS. Even though modern html have to be valid xml the browser will, instead of giving an error when served invalid html, transform what's given to make it standard-compliant.
      - bawolff 6y ago
        Modern html by definition is not valid XML, unless you are using the xml serialization of html5 which isn't really teccomended and nobody does. Really no version of the official html spec was valid xml other than XHTML which was never particularly popular. But i don't really see your point. An implementation having a different idea how to parse html than you think is correct is not the same thing as something being unparsable. Its a tautology that if there exists a computer progran to parse something than it is possible to parse it with a computer program.
  - anoncake 6y ago
    Previous versions of HTML were based on SGML. You can write a perfect SGML parser, but the developers of web browsers couldn't be bothered.
    - erik_seaberg 6y ago
      “Worse is better” strikes again. In the early days of markup hand written by amateurs, users preferred the browsers that tolerated mistakes. Now the language is defined by a common error handling procedure, instead of a grammar.
- karatinversion 6y ago
  > There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. I believe GP was alluding to the fact that many actual resources that declare themselves HTML are not spec conformant, and thus can’t be parsed by a parser that only accepts valid HTML.
  - lifthrasiir 6y ago
    The distinction between "valid" and "invalid" HTML used to matter once upon a time, but it no longer does at least for agents (authors can still benefit from error-free HTMLs because errors can distort their intents). Pretty much every string can be parsed to HTML since HTML5 and all errors are non-fatal, so many modern HTML parsers default to ignore errors. There are parsers that can be configured to abort on any error, but I don't think the GP intended that.
    - tannhaeuser 6y ago
      True, and it's worth noting that since WHATWG HTML 5 has usurped HTML and taken it ad absurdum, WHATWG's parsing spec isn't actually useful nor representative at all of what people usually think HTML is. Nor do people have to follow WHATWG's (= bunch of Chrome developers) idea of HTML anymore than WHATWG did follow other's.
      - detaro 6y ago
        WHATWG is not a "bunch of Chrome developers", and if you want to understand what a browser does with HTML, it's the place to look. "HTML, but not the HTML web browsers mean" is a fairly niche concern.
      - chrismorgan 6y ago
        WHATWG’s HTML spec is the only thing that matters when considering what HTML is, because it’s what every browser uses, which is the primary target of HTML.
      - dragonwriter 6y ago
retox 6y ago
Exactly my experience on this. In a past life I've had to parse valid HTML that was generated by a forum system; the user submitted something akin to bbcode [b]this sort of thing[/b] that was pre-parsed and converted to valid HTML, and then I had to parse that fragment again after the fact. Given the constraints it's entirely possible to parse (a subset of) irregular grammar with regular expressions. Asking a question along those lines on SO would have have only elicited responses that I/someone was doing it wrong. I won't argue that it was or wasn't the wrong to do, but you don't always get to pick your client.
syrrim 6y ago
The html parser spec defines what every sequence of bytes should parse into. It defines certain such sequences as containing "errors", but it still defines exactly how they should be parsed. There is no invalid html. Every browser follows the spec, so every browser will parse the same html to the same thing. This is true even if the html contains "errors". The only checking most html receives is to make sure it renders correctly in a browser. If you are writing your own parser, you likely want it to do the same thing as every browser. In that case, you should use a parser that conforms to the spec.

lifthrasiir 6y ago

I believe you really meant that you are frequently dealing with HTML which structure is already known in advance, not the general HTML. Because... > [...] real world HTML usually can't be parsed by any strict parser [...] There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. Just that you don't always need the full HTML parser to do your job. [1] https://html.spec.whatwg.org/multipage/parsing.html https://html.spec.whatwg.org/multipage/parsing.html

tomashubelbauer 6y ago
I think this is only true for HTML5, but previous versions of HTML supposedly weren't specced well enough to write a prefect parser. Fixing this was one of the goals of the HTML5 revision if I'm not mistaken.
- lifthrasiir 6y ago
  You are correct, but I don't think they are even slightly relevant in 2021.
- bawolff 6y ago
  Web browsers exist, therefore its possible to parse html, even earlier versions. You're right that the de jure spec did not match de facto html, and browsers didn't neccesarily agree with each other. But that's always true. GCC has language extensions that aren't part of the c spec, but you wouldn't say that c is impossible to parse. Old html may have taken it up to 11, but its not fair to say its impossible to parse.
  - chrismorgan 6y ago
    Perl is famously unparseable: it’s impossible to determine a parse tree without executing the code. (HTML, however, was never unparseable, merely insufficiently defined.)
  - uuidgen 6y ago
    No, not really. The browsers did guess a lot and did standard-deviating parsing because the typical uses were wrong and they had to work. Nobody would switch to a new browser that doesn't work with existing pages. Modern example - mXSS. Even though modern html have to be valid xml the browser will, instead of giving an error when served invalid html, transform what's given to make it standard-compliant.
    - bawolff 6y ago
      Modern html by definition is not valid XML, unless you are using the xml serialization of html5 which isn't really teccomended and nobody does. Really no version of the official html spec was valid xml other than XHTML which was never particularly popular. But i don't really see your point. An implementation having a different idea how to parse html than you think is correct is not the same thing as something being unparsable. Its a tautology that if there exists a computer progran to parse something than it is possible to parse it with a computer program.
- anoncake 6y ago
  Previous versions of HTML were based on SGML. You can write a perfect SGML parser, but the developers of web browsers couldn't be bothered.
  - erik_seaberg 6y ago
    “Worse is better” strikes again. In the early days of markup hand written by amateurs, users preferred the browsers that tolerated mistakes. Now the language is defined by a common error handling procedure, instead of a grammar.
karatinversion 6y ago
> There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. I believe GP was alluding to the fact that many actual resources that declare themselves HTML are not spec conformant, and thus can’t be parsed by a parser that only accepts valid HTML.
- lifthrasiir 6y ago
  The distinction between "valid" and "invalid" HTML used to matter once upon a time, but it no longer does at least for agents (authors can still benefit from error-free HTMLs because errors can distort their intents). Pretty much every string can be parsed to HTML since HTML5 and all errors are non-fatal, so many modern HTML parsers default to ignore errors. There are parsers that can be configured to abort on any error, but I don't think the GP intended that.
  - tannhaeuser 6y ago
    True, and it's worth noting that since WHATWG HTML 5 has usurped HTML and taken it ad absurdum, WHATWG's parsing spec isn't actually useful nor representative at all of what people usually think HTML is. Nor do people have to follow WHATWG's (= bunch of Chrome developers) idea of HTML anymore than WHATWG did follow other's.
    - detaro 6y ago
      WHATWG is not a "bunch of Chrome developers", and if you want to understand what a browser does with HTML, it's the place to look. "HTML, but not the HTML web browsers mean" is a fairly niche concern.
    - chrismorgan 6y ago
      WHATWG’s HTML spec is the only thing that matters when considering what HTML is, because it’s what every browser uses, which is the primary target of HTML.
    - dragonwriter 6y ago

retox 6y ago

Exactly my experience on this. In a past life I've had to parse valid HTML that was generated by a forum system; the user submitted something akin to bbcode [b]this sort of thing[/b] that was pre-parsed and converted to valid HTML, and then I had to parse that fragment again after the fact. Given the constraints it's entirely possible to parse (a subset of) irregular grammar with regular expressions. Asking a question along those lines on SO would have have only elicited responses that I/someone was doing it wrong. I won't argue that it was or wasn't the wrong to do, but you don't always get to pick your client.

syrrim 6y ago

The html parser spec defines what every sequence of bytes should parse into. It defines certain such sequences as containing "errors", but it still defines exactly how they should be parsed. There is no invalid html. Every browser follows the spec, so every browser will parse the same html to the same thing. This is true even if the html contains "errors". The only checking most html receives is to make sure it renders correctly in a browser. If you are writing your own parser, you likely want it to do the same thing as every browser. In that case, you should use a parser that conforms to the spec.