13 ms·

You can't parse [X]HTML with regex (2009)

127 points by BerislavLopac 6y ago

vbezhenar 6y ago
You can't in general case. But you can in lots of typical cases. Actually real world HTML usually can't be parsed by any strict parser, as it's not valid. It's just a machine-generated text which pretends to be similar to HTML. So extracting some bits of information with regexes often works.
- lifthrasiir 6y ago
  I believe you really meant that you are frequently dealing with HTML which structure is already known in advance, not the general HTML. Because... > [...] real world HTML usually can't be parsed by any strict parser [...] There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. Just that you don't always need the full HTML parser to do your job. [1] https://html.spec.whatwg.org/multipage/parsing.html https://html.spec.whatwg.org/multipage/parsing.html
  - tomashubelbauer 6y ago
    I think this is only true for HTML5, but previous versions of HTML supposedly weren't specced well enough to write a prefect parser. Fixing this was one of the goals of the HTML5 revision if I'm not mistaken.
    - lifthrasiir 6y ago
      You are correct, but I don't think they are even slightly relevant in 2021.
    - bawolff 6y ago
      Web browsers exist, therefore its possible to parse html, even earlier versions. You're right that the de jure spec did not match de facto html, and browsers didn't neccesarily agree with each other. But that's always true. GCC has language extensions that aren't part of the c spec, but you wouldn't say that c is impossible to parse. Old html may have taken it up to 11, but its not fair to say its impossible to parse.
      - chrismorgan 6y ago
        Perl is famously unparseable: it’s impossible to determine a parse tree without executing the code. (HTML, however, was never unparseable, merely insufficiently defined.)
      - uuidgen 6y ago
        No, not really. The browsers did guess a lot and did standard-deviating parsing because the typical uses were wrong and they had to work. Nobody would switch to a new browser that doesn't work with existing pages. Modern example - mXSS. Even though modern html have to be valid xml the browser will, instead of giving an error when served invalid html, transform what's given to make it standard-compliant.
        bawolff 6y ago
        Modern html by definition is not valid XML, unless you are using the xml serialization of html5 which isn't really teccomended and nobody does. Really no version of the official html spec was valid xml other than XHTML which was never particularly popular. But i don't really see your point. An implementation having a different idea how to parse html than you think is correct is not the same thing as something being unparsable. Its a tautology that if there exists a computer progran to parse something than it is possible to parse it with a computer program.
    - anoncake 6y ago
      Previous versions of HTML were based on SGML. You can write a perfect SGML parser, but the developers of web browsers couldn't be bothered.
      - erik_seaberg 6y ago
        “Worse is better” strikes again. In the early days of markup hand written by amateurs, users preferred the browsers that tolerated mistakes. Now the language is defined by a common error handling procedure, instead of a grammar.
  - karatinversion 6y ago
    > There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. I believe GP was alluding to the fact that many actual resources that declare themselves HTML are not spec conformant, and thus can’t be parsed by a parser that only accepts valid HTML.
    - lifthrasiir 6y ago
      The distinction between "valid" and "invalid" HTML used to matter once upon a time, but it no longer does at least for agents (authors can still benefit from error-free HTMLs because errors can distort their intents). Pretty much every string can be parsed to HTML since HTML5 and all errors are non-fatal, so many modern HTML parsers default to ignore errors. There are parsers that can be configured to abort on any error, but I don't think the GP intended that.
      - tannhaeuser 6y ago
        True, and it's worth noting that since WHATWG HTML 5 has usurped HTML and taken it ad absurdum, WHATWG's parsing spec isn't actually useful nor representative at all of what people usually think HTML is. Nor do people have to follow WHATWG's (= bunch of Chrome developers) idea of HTML anymore than WHATWG did follow other's.
        detaro 6y ago
        WHATWG is not a "bunch of Chrome developers", and if you want to understand what a browser does with HTML, it's the place to look. "HTML, but not the HTML web browsers mean" is a fairly niche concern.
        chrismorgan 6y ago
        WHATWG’s HTML spec is the only thing that matters when considering what HTML is, because it’s what every browser uses, which is the primary target of HTML.
        dragonwriter 6y ago
- retox 6y ago
  Exactly my experience on this. In a past life I've had to parse valid HTML that was generated by a forum system; the user submitted something akin to bbcode [b]this sort of thing[/b] that was pre-parsed and converted to valid HTML, and then I had to parse that fragment again after the fact. Given the constraints it's entirely possible to parse (a subset of) irregular grammar with regular expressions. Asking a question along those lines on SO would have have only elicited responses that I/someone was doing it wrong. I won't argue that it was or wasn't the wrong to do, but you don't always get to pick your client.
- syrrim 6y ago
  The html parser spec defines what every sequence of bytes should parse into. It defines certain such sequences as containing "errors", but it still defines exactly how they should be parsed. There is no invalid html. Every browser follows the spec, so every browser will parse the same html to the same thing. This is true even if the html contains "errors". The only checking most html receives is to make sure it renders correctly in a browser. If you are writing your own parser, you likely want it to do the same thing as every browser. In that case, you should use a parser that conforms to the spec.
Doctor_Fegg 6y ago
"This post is locked to prevent inappropriate edits to its content. The post looks exactly as it is supposed to look - there are no problems with its content. Please do not flag it for our attention." Stack Overflow can be remarkably humourless at times.
- kroltan 6y ago
  That is a defense against the humourless people that tried to edit the post down into a more objective answer years after the fact: https://stackoverflow.com/posts/1732454/revisions?page=2 https://stackoverflow.com/posts/1732454/revisions?page=2
  - Doctor_Fegg 6y ago
    Yes, that was my point.
    - kroltan 6y ago
      Ah I see, I misunderstood your comment as referring to the quote's clinical wording. Sorry!
- captainmuon 6y ago
  Note that you cannot even vote on it, and it is marked CW, so it can doubly not give reputation. This reputation resentment makes me sad...
imedadel 6y ago
I've seen many people complain about StackOverflow, but this is the best example I've ever encountered. The question: How to match <p> <a href="foo"> The answers: rants about how RegEx is not suitable for parsing entire HTML. Only the 5th answer starts to actually answer the question.
- chrismorgan 6y ago
  “You’re asking the wrong question” is a valid response.
  - tomashubelbauer 6y ago
    Yes, but I wish people who like to assume and answer saying so would still answer the question they think is wrong. Context matters and I don't think you can determine that with certainly nearly as often as some people online like to think.
    - shawnz 6y ago
      In this case the answer is correct given the parameters of the question: There is no way to have a regex that only matches the things which OP wants to match, but not any of the things OP doesn't want to match. Given a specific situation, like a particular page or something, sure, regexes are still a possibility for solving the problem. The 2nd highest answer on the page details exactly that. So what is the problem? Is every single contributer obligated to artificially entertain the OP's preconceptions before giving the advice which they believe actually helps best? For example, if I were knowledgeable about XML but not regex, should I just not contribute in such a situation?
  - dgellow 6y ago
    Really, it depends the context. You might be aware that’s not something to generally do and still want to know the answer to the actual question.
  - anon946 6y ago
    If the question were about full validated parsing of HTML with a regex, then I'd agree that "You can't do that" might be part of a valid answer. But finding tags is not doing a full validating parse. Note that the set of valid C programs is not a context-free language. Yet it's common to use a context free-based approach to parsing. You just add additional code to handle the context-sensitive aspects (such as a symbol table).
gostsamo 6y ago
I get why all the complains against the top answer. At the same time one should appreciate its literary qualities in regard to structure and style.
llimos 6y ago
Would be interesting to know how many up and down votes HN is sending that answer.
- tomashubelbauer 6y ago
  The question is locked, so it cannot be voted on.
- daniel-thompson 6y ago
  The answer had 4440 upvotes and 27 downvotes at the time it was locked (click on it to reveal the breakdown, if you have sufficient SO "reputation").
zihotki 6y ago
Should 2011 (when answer was first provided) or 2009 (when question was posted) be added to the title?
- usrusr 6y ago
  Clearly it should be 20(?:09|11) On the other hand, the inclusion of [X] in the title is more than enough to establish the historical setting.
lifthrasiir 6y ago
I had once tweeted related quizzes. Can you guess parse trees (or reserializations) for following HTML fragments without invoking browsers? Assume that everything gets pasted right after the document body. 1. <a b="42>c">d 2. <a/b/c=d/e>f 3. <a/="42>b 4. <a x=&amp0>&amp0</a> 5. ad Really, don't try to answer and just use complaint HTML parsers.
- bhaak 6y ago
  What kind of HTML parser? A SGML one or a HTML5 one? I'm really sad that they didn't go with a XML base for HTML5.
  - lucideer 6y ago
    > I'm really sad that they didn't go with a XML base for HTML5. I'm really sad that they didn't evangelise an XML base for HTML5, and that many HTML5-ish tools don't explicitly support XML, but it's not strictly true that they didn't go for an XML base for HTML5[0][1] [0] https://html.spec.whatwg.org/multipage/xhtml.html https://html.spec.whatwg.org/multipage/xhtml.html [1] https://www.w3.org/TR/html-polyglot/ https://www.w3.org/TR/html-polyglot/
    - bhaak 6y ago
      That's just HTML5 rewritten as a well formed XML document. HTML5 does not describe well formed XML. That you can put the same information of a HTML5 document into a XML document doesn't help much if most of the HTML5 documents out there are not polygot.
      - chrismorgan 6y ago
        But the point is that XML syntax is still a thing, supported by all browsers (and it’s reasonable to expect that support to remain as long as HTML remains). See also https://html.spec.whatwg.org/multipage/introduction.html#html-vs-xhtml https://html.spec.whatwg.org/multipage/introduction.html#htm...: > When a document is transmitted with an XML MIME type, such as application/xhtml+xml, then it is treated as an XML document by web browsers, to be parsed by an XML processor. Authors are reminded that the processing for XML and HTML differs; in particular, even minor syntax errors will prevent a document labeled as XML from being rendered fully, whereas they would be ignored in the HTML syntax. HTML does use an XML base (elements, attributes, namespaces, &c.), it just doesn’t use an XML parser most of the time. But the XMLness is easily observed in various DOM APIs.
ricardo81 6y ago
I think we've all (mostly?) tried it. It really is the Wild West of the web when you're trying to parse other people's HTML, though. I've played around with this parser which is extremely quick. https://github.com/lexbor/lexbor https://github.com/lexbor/lexbor
8lall0 6y ago
TONY THE PONY.
The_rationalist 6y ago
Aren't html code highlighters using regex? Isn't vscode using TextMate regex for color highlithing?
- layer8 6y ago
  You can usually use regexes for tokenization, which is sufficient for syntax highlighting, but you generally can’t use regexes for parsing (nested structures).
- chrismorgan 6y ago
  Most code highlighters do. They’re normally close enough to accurate for highlighting purposes (though they will commonly have some uncommon constructs that they get wrong), but they tend to fall apart when you try to use that for much more; for example, indentation when you use regular expressions to parse your HTML tends to start falling apart if you take what XML users might consider “shortcuts” (such as omitting optional end tags).
  - The_rationalist 6y ago
    Valid HTML allow optional end tags? For example?
    - lifthrasiir 6y ago
      `<script>` is an usual example that you can't self-close and absolutely need to be followed by `</script>` in HTML5. In general though self-closing tag has no effect in HTML5 anyway, `<script>` is just an example where the usual heuristic specified by HTML5 doesn't help you at all (since it switches the lexer state).
    - deleted 6y ago
      [deleted]
    - chrismorgan 6y ago
      https://html.spec.whatwg.org/multipage/syntax.html#optional-tags https://html.spec.whatwg.org/multipage/syntax.html#optional-... describes which elements have optional start and end tags. This document is valid: <!doctype html> <title>Hello,</title> <p>World! <ul> <li>A <li>B </ul> <table> <tr> <th>C <td>D <td>E <tfoot> <tr> <th>F <td>G <td>H </table> And is precisely equivalent to this document in the canonical serialisation: <!DOCTYPE html><html><head><title>Hello,</title> </head><body><p>World! </p><ul> <li>A </li><li>B </li></ul> <table> <tbody><tr> <th>C </th><td>D </td><td>E </td></tr></tbody><tfoot> <tr> <th>F </th><td>G </th><td>H </td></tr></tfoot></table></body></html>
Corazoor 6y ago
Well, as stated that particuar answer is both right and wrong... Yes, you can not use "true" regular expressions to parse recursive structures. But the libraries that get used for regular expressions quite often include non-regular extensions (and confusingly call the resulting expressions still "regular"). Most notably, PCRE allows for recursive patterns via "(?R)". You can absolutely parse arbitrary HTML with it. In fact you can parse anything whith that, including binary formats. You just can't do it whithout recursively applying the same "regex" again and again... And precise error handling is basically impossible without writing a proper lexer anyway, since your regex won't (can't, really) tell you where it was thrown off. It either works or doesn't, the "why" is left to the program to figure out...
JackeJR 6y ago
Counter argument: Oh Yes You Can Use Regexes to Parse HTML! https://stackoverflow.com/questions/4231382/what-to-do-regular-expression-pattern-doesnt-match-anywhere-in-string/4234491#4234491 https://stackoverflow.com/questions/4231382/what-to-do-regul... Discussion: https://news.ycombinator.com/item?id=26357237 https://news.ycombinator.com/item?id=26357237
- harveywi 6y ago
  Do the Halting Problem next.
- lucideer 6y ago
  Counter-counter-argument, one of the comments underneath that answer: > what you have written is not really a regular expression (modern, regular, or otherwise), but rather a Perl program that uses regular expressions heavily. Does your post really support the claim that regular expressions can parse HTML correctly? Or is it more like evidence that Perl can parse HTML correctly?
csours 6y ago
To me, this typifies working with technology and programming. Computer programs only ever look like they are working, because they have not encountered problem data or conditions. Aka, it works on the happy path. Software engineering is how we balance how much of the unhappy path and corner cases we take care of, and how we handle them, imo.
rswskg 6y ago
Old, but gold.
taylodl 6y ago
2nd answer: "While arbitrary HTML with only a regex is impossible, it's sometimes appropriate to use them for parsing a limited, known set of HTML." This. So much this. Yes, you can't parse arbitrary, unknown XML with regex. But I don't find myself parsing arbitrary, unknown XML very often. Usually I know exactly what I'm expecting and if I can't find the information I need then it's a problem. Regex parsing is perfect for this scenario - and much, much faster. I created a regex parser for Java that even handles namespaces and relative paths. Can it parse every XML file? No - you can't parse XML with regex. But I can parse everything I need to parse - and if I can't? I can always fall back on full-featured XML parsers.
- dragontamer 6y ago
  I found a one-pass top-down "XML parser". Not like a proper SAX parser, no... the XML had to be specially formatted, almost like TOML. <whatever> <--- Parser parsed this as a new section <foo attrib=bar> <--- All "XML" had to be one-per-line </whatever> <---- parser ignored this It was an "XML parser" per se, but it really was just a linear one-pass parser that tricked me into thinking it was XML. So really, it was more like TOML (or .INI files) than like XML. But I guess the advantage of making it "bastard XML" instead of TOML is that maybe this worked with XML-editors or something. I dunno...
  - cbsks 6y ago
    I’ve written an XML parser like this for a toy project. I passed the XML through a prettifier first so that it was in a standard format. It wouldn’t work for every XML file, but it worked on the files that I needed it for. I have also had success searching through html files with grep after passing them through a prettifier. It’s ugly, but 90% of the time, it works every time!
    - dragontamer 6y ago
      The specific project I saw that did this had ALL of its configuration files in this "Bastard XML" format. Its ugly, but when 100% of your files you interpret match the one-per-line (and other clearly made-up rules), then it works 100% of the time!!
h2odragon 6y ago
This argument is common, and this is a good answer; but so often people aren't "parsing" XML but extracting a few bits of it and would have benefited from less cargo cult and more thought in the answer cited. As it is, I've seen this article used to scare people away from "can i make a game behave differently?" efforts that would have been trivial to do and likely given these people a gateway "i can try to be a programmer" experience.
- zeroimpl 6y ago
  This answer is right. The original question isn't asking anything about parsing, they are trying to search for specific tags, which is a great use for a regex. Yes, it will probably return some invalid matches unless you go out of your way to filter out comments and script tags, but odds are you don't need that. Applying regex's only really count as "parsing" when you are matching against an entire document. Searching is not parsing. Same argument applies if you apply grep to an html doc - I wonder why there's no posts about "you can't parse HTML with grep". I apply grep all the time to source code...
darkwater 6y ago
My 5yo daughter can already write more or less ok-ish but she has big problems reading that back, especially when she does spelling mistakes. I feel more or less the same when I write regular expressions.
goldsteinq 6y ago
You can't parse HTML with regex, but PCRE is not regex. I'm not sure if you can parse HTML with PCRE.
mumblemumble 6y ago
This pops up every so often, and it sort of irritates me every time. Partially because it's overly simplistic, but, even more so, because, while it's cute and humorous, it's not actually very good advice and it doesn't actually answer the question. No, you can't parse html with regex. But go look at the question. The author is just trying to detect some tags. That's not exactly parsing. It's true that there are some complications around things like "What if > appears in an attribute's value?" If you know your input well enough, or you don't need perfection, that might be a problem you can ignore. Alternatively, you can still use regex, if you use a sufficiently powerful regular expression tool. .NET's regular expressions, for example, have a concept of balancing groups that will let you do this. I would also point out that a lot of open source HTML parsing libraries are even more dangerous than regular expressions for parsing unknown HTML, because they use recursive descent. Where you have recursion, you have the potential for a stack overflow. With a regex library, you do have to be careful about catastrophic backtracking, but that's at least something you can usually handle in your own code, or, in the worst case, defend against with timeouts. A parser that's capable of blowing the call stack, and has been exposed to input from the Internet, though, is capable of taking down your process in a way you can't defend against in most languages. And it's difficult to patch up a parser like that without more-or-less rewriting it. I absolutely have had to deal with html handling code getting into situations like that in the past. Malicious input is real. So is plain old bad input. Reading the code before you use it is often a good idea.
indymike 6y ago
Funny thing. Email addresses need a rant like this too. Yes, you can parse 99% or so with a regex, but like HTML or XML, you really need an email address parser. RFC 2822 was designed to be parsed using string processing (in C no less) and requires some complexity that most regexes fail on. Here's a discussion about using the simpler, older RFC (822) and regex: https://stackoverflow.com/questions/20771794/mailrfc822address-regex https://stackoverflow.com/questions/20771794/mailrfc822addre...
- mumblemumble 6y ago
  For most purposes, if you're trying to use parsing to achieve email address validation perfection, you've already lost the battle. A valid email address typically isn't just a syntactically correct one; it's also one that can be used to get an email to the recipient. The only way to test that is to send an email and see if it gets to the recipient. Which is why it's much more common to see some minimal client-side validation that uses a simple regex that will (ideally) match all valid email addresses but only catch gross syntactic errors like typing # where you meant @, more for the sake of decent UX than anything else, and rely on asking the user to double-type their email address and sending an activation email to deal with finger-grained syntactic errors and the whole universe of non-syntactic errors.
  - indymike 6y ago
    # is a valid character in an email address. Regexes are a horrible idea to validate or parse emails and have been since the beginning. Most regexes will be too strict. The reason people use regexes on the client side is they copy pasta something they find on the internet. Incidentally, a great way to get emails delivered is to properly validate them. Sure typos are possible... Few things are a worse user experience than having a valid email rejected by a client side parser.
xg15 6y ago
Yes, it's hopeless to try and parse arbitrary HTML with regexes (or with anything really. I believe, the full HTML5 parsing algorithm is not even type-2, it's more or less turing-complete. In-the-wild HTML can also interact with scripting in all kinds of entertaining ways, to the point that a conforming HTML5 parser has to be able to execute javascript while parsing - and the javascript can inject additional tokens into the parser's input stream. It's possible to create a HTML5 document that only validates on every second thursday of the month.) However, if we know the document is well-formed(!) XHTML, shouldn't it be possible? This would mean the document is valid XML and XML was specifically designed to be regex-friendly, I believe. At least, out of my head, the only gotchas that have to be accounted for are comments and CDATA sections - those may contain arbitrary unescaped text, including angle brackets. However they also have unambigous start and end markers and can't be nested, so a regex could account for them. Attribute values should not be a problem, as angle brackets must be escaped inside those to be valid XML. I'm not sure about processing instructions and doctypes though.
- theandrewbailey 6y ago
  On my blog, I write my posts in markdown. After it's converted to HTML, I use regex to search and replace images (for high-res and alternative formats), and get the first paragraph (for a 'preview'). I've been doing this for years, so the 'never use regex over HTML' advice isn't holding up for me.
- chubot 6y ago
  Yes that's basically what I wrote about here: https://news.ycombinator.com/item?id=26359556 https://news.ycombinator.com/item?id=26359556 I listed 3 or 4 caveats. CDATA might be another one since I'm not handling those... I've never used it so I left it out. Actually I remember that regular languages weren't entirely enough; I think the .*? non-greedy matching was useful, e.g. for finding --> at the end of comments.
legec 6y ago
Where are all the comments gone ? (note : I mean the comments on StackOverflow, not the comments here in Hacker News ... )
deleted 6y ago
[deleted]
loxias 6y ago
This never gets old.
a1369209993 6y ago
You can't parse fully-general HTML with regex, but unless you're writing a web browser of something, that's not what you're trying to do; you're trying to parse the particular subset of HTML that happens to be emitted by this particular website that you got the HTML to be parsed from. And, much like the halting problem or integer factorization, despite the general case being difficult or impossible, the overwhelming majority of specific cases are easy.
dang 6y ago
If curious, past threads: RegEx match open tags except XHTML self-contained tags - https://news.ycombinator.com/item?id=14942060 https://news.ycombinator.com/item?id=14942060 - Aug 2017 (6 comments) You can't parse [X]HTML with regex - https://news.ycombinator.com/item?id=14155015 https://news.ycombinator.com/item?id=14155015 - April 2017 (1 comment) Why you can't parse HTML with regex - https://news.ycombinator.com/item?id=9728474 https://news.ycombinator.com/item?id=9728474 - June 2015 (2 comments) Can you parse html with regular expressions? - https://news.ycombinator.com/item?id=5264511 https://news.ycombinator.com/item?id=5264511 - Feb 2013 (2 comments) Can regular expressions parse HTML or not? - https://news.ycombinator.com/item?id=5257535 https://news.ycombinator.com/item?id=5257535 - Feb 2013 (23 comments) Regexes Parse XML Just Fine, Actually - https://news.ycombinator.com/item?id=3088402 https://news.ycombinator.com/item?id=3088402 - Oct 2011 (27 comments) Oh Yes You Can Use Regexes to Parse HTML - https://news.ycombinator.com/item?id=2741780 https://news.ycombinator.com/item?id=2741780 - July 2011 (77 comments) Why you should not parse (X)HTML with a Regexp - https://news.ycombinator.com/item?id=2423301 https://news.ycombinator.com/item?id=2423301 - April 2011 (5 comments) Stackoverflow, HTML by Regex, topmost answer - https://news.ycombinator.com/item?id=1487695 https://news.ycombinator.com/item?id=1487695 - July 2010 (37 comments) You Cannot Parse HTML with Regular Expressions - https://news.ycombinator.com/item?id=1274870 https://news.ycombinator.com/item?id=1274870 - April 2010 (7 comments) You can't parse [X]HTML with regex. - https://news.ycombinator.com/item?id=941401 https://news.ycombinator.com/item?id=941401 - Nov 2009 (1 comment) I'm surprised that most of these are so small. There must have been others? (I excluded the boring or empty ones.)
- pvg 6y ago
  There must have been others? Yog-Sothoth knows the gate. Yog-Sothoth is the gate. Yog-Sothoth is the key and guardian of the gate. Past, present, future, all are one in Yog-Sothoth. He knows where the Old Threads broke through of old, and where They shall break through again.

127 points by BerislavLopac 6y ago

You can't in general case. But you can in lots of typical cases. Actually real world HTML usually can't be parsed by any strict parser, as it's not valid. It's just a machine-generated text which pretends to be similar to HTML. So extracting some bits of information with regexes often works.

lifthrasiir 6y ago
I believe you really meant that you are frequently dealing with HTML which structure is already known in advance, not the general HTML. Because... > [...] real world HTML usually can't be parsed by any strict parser [...] There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. Just that you don't always need the full HTML parser to do your job. [1] https://html.spec.whatwg.org/multipage/parsing.html https://html.spec.whatwg.org/multipage/parsing.html
- tomashubelbauer 6y ago
  I think this is only true for HTML5, but previous versions of HTML supposedly weren't specced well enough to write a prefect parser. Fixing this was one of the goals of the HTML5 revision if I'm not mistaken.
  - lifthrasiir 6y ago
    You are correct, but I don't think they are even slightly relevant in 2021.
  - bawolff 6y ago
    Web browsers exist, therefore its possible to parse html, even earlier versions. You're right that the de jure spec did not match de facto html, and browsers didn't neccesarily agree with each other. But that's always true. GCC has language extensions that aren't part of the c spec, but you wouldn't say that c is impossible to parse. Old html may have taken it up to 11, but its not fair to say its impossible to parse.
    - chrismorgan 6y ago
      Perl is famously unparseable: it’s impossible to determine a parse tree without executing the code. (HTML, however, was never unparseable, merely insufficiently defined.)
    - uuidgen 6y ago
      No, not really. The browsers did guess a lot and did standard-deviating parsing because the typical uses were wrong and they had to work. Nobody would switch to a new browser that doesn't work with existing pages. Modern example - mXSS. Even though modern html have to be valid xml the browser will, instead of giving an error when served invalid html, transform what's given to make it standard-compliant.
      - bawolff 6y ago
        Modern html by definition is not valid XML, unless you are using the xml serialization of html5 which isn't really teccomended and nobody does. Really no version of the official html spec was valid xml other than XHTML which was never particularly popular. But i don't really see your point. An implementation having a different idea how to parse html than you think is correct is not the same thing as something being unparsable. Its a tautology that if there exists a computer progran to parse something than it is possible to parse it with a computer program.
  - anoncake 6y ago
    Previous versions of HTML were based on SGML. You can write a perfect SGML parser, but the developers of web browsers couldn't be bothered.
    - erik_seaberg 6y ago
      “Worse is better” strikes again. In the early days of markup hand written by amateurs, users preferred the browsers that tolerated mistakes. Now the language is defined by a common error handling procedure, instead of a grammar.
- karatinversion 6y ago
  > There is the literal standard for parsing HTML [1]. Any conformant implementation (and there are plenty) can of course parse the real world HTML by definition. I believe GP was alluding to the fact that many actual resources that declare themselves HTML are not spec conformant, and thus can’t be parsed by a parser that only accepts valid HTML.
  - lifthrasiir 6y ago
    The distinction between "valid" and "invalid" HTML used to matter once upon a time, but it no longer does at least for agents (authors can still benefit from error-free HTMLs because errors can distort their intents). Pretty much every string can be parsed to HTML since HTML5 and all errors are non-fatal, so many modern HTML parsers default to ignore errors. There are parsers that can be configured to abort on any error, but I don't think the GP intended that.
    - tannhaeuser 6y ago
      True, and it's worth noting that since WHATWG HTML 5 has usurped HTML and taken it ad absurdum, WHATWG's parsing spec isn't actually useful nor representative at all of what people usually think HTML is. Nor do people have to follow WHATWG's (= bunch of Chrome developers) idea of HTML anymore than WHATWG did follow other's.
      - detaro 6y ago
        WHATWG is not a "bunch of Chrome developers", and if you want to understand what a browser does with HTML, it's the place to look. "HTML, but not the HTML web browsers mean" is a fairly niche concern.
      - chrismorgan 6y ago
        WHATWG’s HTML spec is the only thing that matters when considering what HTML is, because it’s what every browser uses, which is the primary target of HTML.
      - dragonwriter 6y ago
retox 6y ago
Exactly my experience on this. In a past life I've had to parse valid HTML that was generated by a forum system; the user submitted something akin to bbcode [b]this sort of thing[/b] that was pre-parsed and converted to valid HTML, and then I had to parse that fragment again after the fact. Given the constraints it's entirely possible to parse (a subset of) irregular grammar with regular expressions. Asking a question along those lines on SO would have have only elicited responses that I/someone was doing it wrong. I won't argue that it was or wasn't the wrong to do, but you don't always get to pick your client.
syrrim 6y ago
The html parser spec defines what every sequence of bytes should parse into. It defines certain such sequences as containing "errors", but it still defines exactly how they should be parsed. There is no invalid html. Every browser follows the spec, so every browser will parse the same html to the same thing. This is true even if the html contains "errors". The only checking most html receives is to make sure it renders correctly in a browser. If you are writing your own parser, you likely want it to do the same thing as every browser. In that case, you should use a parser that conforms to the spec.

Doctor_Fegg 6y ago

"This post is locked to prevent inappropriate edits to its content. The post looks exactly as it is supposed to look - there are no problems with its content. Please do not flag it for our attention." Stack Overflow can be remarkably humourless at times.

kroltan 6y ago
That is a defense against the humourless people that tried to edit the post down into a more objective answer years after the fact: https://stackoverflow.com/posts/1732454/revisions?page=2 https://stackoverflow.com/posts/1732454/revisions?page=2
- Doctor_Fegg 6y ago
  Yes, that was my point.
  - kroltan 6y ago
    Ah I see, I misunderstood your comment as referring to the quote's clinical wording. Sorry!
captainmuon 6y ago
Note that you cannot even vote on it, and it is marked CW, so it can doubly not give reputation. This reputation resentment makes me sad...

imedadel 6y ago

I've seen many people complain about StackOverflow, but this is the best example I've ever encountered. The question: How to match <p> <a href="foo"> The answers: rants about how RegEx is not suitable for parsing entire HTML. Only the 5th answer starts to actually answer the question.

chrismorgan 6y ago
“You’re asking the wrong question” is a valid response.
- tomashubelbauer 6y ago
  Yes, but I wish people who like to assume and answer saying so would still answer the question they think is wrong. Context matters and I don't think you can determine that with certainly nearly as often as some people online like to think.
  - shawnz 6y ago
    In this case the answer is correct given the parameters of the question: There is no way to have a regex that only matches the things which OP wants to match, but not any of the things OP doesn't want to match. Given a specific situation, like a particular page or something, sure, regexes are still a possibility for solving the problem. The 2nd highest answer on the page details exactly that. So what is the problem? Is every single contributer obligated to artificially entertain the OP's preconceptions before giving the advice which they believe actually helps best? For example, if I were knowledgeable about XML but not regex, should I just not contribute in such a situation?
- dgellow 6y ago
  Really, it depends the context. You might be aware that’s not something to generally do and still want to know the answer to the actual question.
- anon946 6y ago
  If the question were about full validated parsing of HTML with a regex, then I'd agree that "You can't do that" might be part of a valid answer. But finding tags is not doing a full validating parse. Note that the set of valid C programs is not a context-free language. Yet it's common to use a context free-based approach to parsing. You just add additional code to handle the context-sensitive aspects (such as a symbol table).

gostsamo 6y ago

I get why all the complains against the top answer. At the same time one should appreciate its literary qualities in regard to structure and style.

llimos 6y ago

Would be interesting to know how many up and down votes HN is sending that answer.

tomashubelbauer 6y ago
The question is locked, so it cannot be voted on.
daniel-thompson 6y ago
The answer had 4440 upvotes and 27 downvotes at the time it was locked (click on it to reveal the breakdown, if you have sufficient SO "reputation").

zihotki 6y ago

Should 2011 (when answer was first provided) or 2009 (when question was posted) be added to the title?

usrusr 6y ago
Clearly it should be 20(?:09|11) On the other hand, the inclusion of [X] in the title is more than enough to establish the historical setting.

lifthrasiir 6y ago

I had once tweeted related quizzes. Can you guess parse trees (or reserializations) for following HTML fragments without invoking browsers? Assume that everything gets pasted right after the document body. 1. <a b="42>c">d 2. <a/b/c=d/e>f 3. <a/="42>b 4. <a x=&amp0>&amp0</a> 5. ad Really, don't try to answer and just use complaint HTML parsers.

bhaak 6y ago
What kind of HTML parser? A SGML one or a HTML5 one? I'm really sad that they didn't go with a XML base for HTML5.
- lucideer 6y ago
  > I'm really sad that they didn't go with a XML base for HTML5. I'm really sad that they didn't evangelise an XML base for HTML5, and that many HTML5-ish tools don't explicitly support XML, but it's not strictly true that they didn't go for an XML base for HTML5[0][1] [0] https://html.spec.whatwg.org/multipage/xhtml.html https://html.spec.whatwg.org/multipage/xhtml.html [1] https://www.w3.org/TR/html-polyglot/ https://www.w3.org/TR/html-polyglot/
  - bhaak 6y ago
    That's just HTML5 rewritten as a well formed XML document. HTML5 does not describe well formed XML. That you can put the same information of a HTML5 document into a XML document doesn't help much if most of the HTML5 documents out there are not polygot.
    - chrismorgan 6y ago
      But the point is that XML syntax is still a thing, supported by all browsers (and it’s reasonable to expect that support to remain as long as HTML remains). See also https://html.spec.whatwg.org/multipage/introduction.html#html-vs-xhtml https://html.spec.whatwg.org/multipage/introduction.html#htm...: > When a document is transmitted with an XML MIME type, such as application/xhtml+xml, then it is treated as an XML document by web browsers, to be parsed by an XML processor. Authors are reminded that the processing for XML and HTML differs; in particular, even minor syntax errors will prevent a document labeled as XML from being rendered fully, whereas they would be ignored in the HTML syntax. HTML does use an XML base (elements, attributes, namespaces, &c.), it just doesn’t use an XML parser most of the time. But the XMLness is easily observed in various DOM APIs.

ricardo81 6y ago

I think we've all (mostly?) tried it. It really is the Wild West of the web when you're trying to parse other people's HTML, though. I've played around with this parser which is extremely quick. https://github.com/lexbor/lexbor https://github.com/lexbor/lexbor

8lall0 6y ago

TONY THE PONY.

The_rationalist 6y ago

Aren't html code highlighters using regex? Isn't vscode using TextMate regex for color highlithing?

layer8 6y ago
You can usually use regexes for tokenization, which is sufficient for syntax highlighting, but you generally can’t use regexes for parsing (nested structures).
chrismorgan 6y ago
Most code highlighters do. They’re normally close enough to accurate for highlighting purposes (though they will commonly have some uncommon constructs that they get wrong), but they tend to fall apart when you try to use that for much more; for example, indentation when you use regular expressions to parse your HTML tends to start falling apart if you take what XML users might consider “shortcuts” (such as omitting optional end tags).
- The_rationalist 6y ago
  Valid HTML allow optional end tags? For example?
  - lifthrasiir 6y ago
    `<script>` is an usual example that you can't self-close and absolutely need to be followed by `</script>` in HTML5. In general though self-closing tag has no effect in HTML5 anyway, `<script>` is just an example where the usual heuristic specified by HTML5 doesn't help you at all (since it switches the lexer state).
  - deleted 6y ago
    [deleted]
  - chrismorgan 6y ago
    https://html.spec.whatwg.org/multipage/syntax.html#optional-tags https://html.spec.whatwg.org/multipage/syntax.html#optional-... describes which elements have optional start and end tags. This document is valid: <!doctype html> <title>Hello,</title> <p>World! <ul> <li>A <li>B </ul> <table> <tr> <th>C <td>D <td>E <tfoot> <tr> <th>F <td>G <td>H </table> And is precisely equivalent to this document in the canonical serialisation: <!DOCTYPE html><html><head><title>Hello,</title> </head><body><p>World! </p><ul> <li>A </li><li>B </li></ul> <table> <tbody><tr> <th>C </th><td>D </td><td>E </td></tr></tbody><tfoot> <tr> <th>F </th><td>G </th><td>H </td></tr></tfoot></table></body></html>

Corazoor 6y ago

Well, as stated that particuar answer is both right and wrong... Yes, you can not use "true" regular expressions to parse recursive structures. But the libraries that get used for regular expressions quite often include non-regular extensions (and confusingly call the resulting expressions still "regular"). Most notably, PCRE allows for recursive patterns via "(?R)". You can absolutely parse arbitrary HTML with it. In fact you can parse anything whith that, including binary formats. You just can't do it whithout recursively applying the same "regex" again and again... And precise error handling is basically impossible without writing a proper lexer anyway, since your regex won't (can't, really) tell you where it was thrown off. It either works or doesn't, the "why" is left to the program to figure out...

JackeJR 6y ago

Counter argument: Oh Yes You Can Use Regexes to Parse HTML! https://stackoverflow.com/questions/4231382/what-to-do-regular-expression-pattern-doesnt-match-anywhere-in-string/4234491#4234491 https://stackoverflow.com/questions/4231382/what-to-do-regul... Discussion: https://news.ycombinator.com/item?id=26357237 https://news.ycombinator.com/item?id=26357237

harveywi 6y ago
Do the Halting Problem next.
lucideer 6y ago
Counter-counter-argument, one of the comments underneath that answer: > what you have written is not really a regular expression (modern, regular, or otherwise), but rather a Perl program that uses regular expressions heavily. Does your post really support the claim that regular expressions can parse HTML correctly? Or is it more like evidence that Perl can parse HTML correctly?

csours 6y ago

To me, this typifies working with technology and programming. Computer programs only ever look like they are working, because they have not encountered problem data or conditions. Aka, it works on the happy path. Software engineering is how we balance how much of the unhappy path and corner cases we take care of, and how we handle them, imo.

rswskg 6y ago

Old, but gold.

taylodl 6y ago

2nd answer: "While arbitrary HTML with only a regex is impossible, it's sometimes appropriate to use them for parsing a limited, known set of HTML." This. So much this. Yes, you can't parse arbitrary, unknown XML with regex. But I don't find myself parsing arbitrary, unknown XML very often. Usually I know exactly what I'm expecting and if I can't find the information I need then it's a problem. Regex parsing is perfect for this scenario - and much, much faster. I created a regex parser for Java that even handles namespaces and relative paths. Can it parse every XML file? No - you can't parse XML with regex. But I can parse everything I need to parse - and if I can't? I can always fall back on full-featured XML parsers.

dragontamer 6y ago
I found a one-pass top-down "XML parser". Not like a proper SAX parser, no... the XML had to be specially formatted, almost like TOML. <whatever> <--- Parser parsed this as a new section <foo attrib=bar> <--- All "XML" had to be one-per-line </whatever> <---- parser ignored this It was an "XML parser" per se, but it really was just a linear one-pass parser that tricked me into thinking it was XML. So really, it was more like TOML (or .INI files) than like XML. But I guess the advantage of making it "bastard XML" instead of TOML is that maybe this worked with XML-editors or something. I dunno...
- cbsks 6y ago
  I’ve written an XML parser like this for a toy project. I passed the XML through a prettifier first so that it was in a standard format. It wouldn’t work for every XML file, but it worked on the files that I needed it for. I have also had success searching through html files with grep after passing them through a prettifier. It’s ugly, but 90% of the time, it works every time!
  - dragontamer 6y ago
    The specific project I saw that did this had ALL of its configuration files in this "Bastard XML" format. Its ugly, but when 100% of your files you interpret match the one-per-line (and other clearly made-up rules), then it works 100% of the time!!

h2odragon 6y ago

This argument is common, and this is a good answer; but so often people aren't "parsing" XML but extracting a few bits of it and would have benefited from less cargo cult and more thought in the answer cited. As it is, I've seen this article used to scare people away from "can i make a game behave differently?" efforts that would have been trivial to do and likely given these people a gateway "i can try to be a programmer" experience.

zeroimpl 6y ago
This answer is right. The original question isn't asking anything about parsing, they are trying to search for specific tags, which is a great use for a regex. Yes, it will probably return some invalid matches unless you go out of your way to filter out comments and script tags, but odds are you don't need that. Applying regex's only really count as "parsing" when you are matching against an entire document. Searching is not parsing. Same argument applies if you apply grep to an html doc - I wonder why there's no posts about "you can't parse HTML with grep". I apply grep all the time to source code...

darkwater 6y ago

My 5yo daughter can already write more or less ok-ish but she has big problems reading that back, especially when she does spelling mistakes. I feel more or less the same when I write regular expressions.

goldsteinq 6y ago

You can't parse HTML with regex, but PCRE is not regex. I'm not sure if you can parse HTML with PCRE.

mumblemumble 6y ago

This pops up every so often, and it sort of irritates me every time. Partially because it's overly simplistic, but, even more so, because, while it's cute and humorous, it's not actually very good advice and it doesn't actually answer the question. No, you can't parse html with regex. But go look at the question. The author is just trying to detect some tags. That's not exactly parsing. It's true that there are some complications around things like "What if > appears in an attribute's value?" If you know your input well enough, or you don't need perfection, that might be a problem you can ignore. Alternatively, you can still use regex, if you use a sufficiently powerful regular expression tool. .NET's regular expressions, for example, have a concept of balancing groups that will let you do this. I would also point out that a lot of open source HTML parsing libraries are even more dangerous than regular expressions for parsing unknown HTML, because they use recursive descent. Where you have recursion, you have the potential for a stack overflow. With a regex library, you do have to be careful about catastrophic backtracking, but that's at least something you can usually handle in your own code, or, in the worst case, defend against with timeouts. A parser that's capable of blowing the call stack, and has been exposed to input from the Internet, though, is capable of taking down your process in a way you can't defend against in most languages. And it's difficult to patch up a parser like that without more-or-less rewriting it. I absolutely have had to deal with html handling code getting into situations like that in the past. Malicious input is real. So is plain old bad input. Reading the code before you use it is often a good idea.

indymike 6y ago

Funny thing. Email addresses need a rant like this too. Yes, you can parse 99% or so with a regex, but like HTML or XML, you really need an email address parser. RFC 2822 was designed to be parsed using string processing (in C no less) and requires some complexity that most regexes fail on. Here's a discussion about using the simpler, older RFC (822) and regex: https://stackoverflow.com/questions/20771794/mailrfc822address-regex https://stackoverflow.com/questions/20771794/mailrfc822addre...

mumblemumble 6y ago
For most purposes, if you're trying to use parsing to achieve email address validation perfection, you've already lost the battle. A valid email address typically isn't just a syntactically correct one; it's also one that can be used to get an email to the recipient. The only way to test that is to send an email and see if it gets to the recipient. Which is why it's much more common to see some minimal client-side validation that uses a simple regex that will (ideally) match all valid email addresses but only catch gross syntactic errors like typing # where you meant @, more for the sake of decent UX than anything else, and rely on asking the user to double-type their email address and sending an activation email to deal with finger-grained syntactic errors and the whole universe of non-syntactic errors.
- indymike 6y ago
  # is a valid character in an email address. Regexes are a horrible idea to validate or parse emails and have been since the beginning. Most regexes will be too strict. The reason people use regexes on the client side is they copy pasta something they find on the internet. Incidentally, a great way to get emails delivered is to properly validate them. Sure typos are possible... Few things are a worse user experience than having a valid email rejected by a client side parser.

xg15 6y ago

Yes, it's hopeless to try and parse arbitrary HTML with regexes (or with anything really. I believe, the full HTML5 parsing algorithm is not even type-2, it's more or less turing-complete. In-the-wild HTML can also interact with scripting in all kinds of entertaining ways, to the point that a conforming HTML5 parser has to be able to execute javascript while parsing - and the javascript can inject additional tokens into the parser's input stream. It's possible to create a HTML5 document that only validates on every second thursday of the month.) However, if we know the document is well-formed(!) XHTML, shouldn't it be possible? This would mean the document is valid XML and XML was specifically designed to be regex-friendly, I believe. At least, out of my head, the only gotchas that have to be accounted for are comments and CDATA sections - those may contain arbitrary unescaped text, including angle brackets. However they also have unambigous start and end markers and can't be nested, so a regex could account for them. Attribute values should not be a problem, as angle brackets must be escaped inside those to be valid XML. I'm not sure about processing instructions and doctypes though.

theandrewbailey 6y ago
On my blog, I write my posts in markdown. After it's converted to HTML, I use regex to search and replace images (for high-res and alternative formats), and get the first paragraph (for a 'preview'). I've been doing this for years, so the 'never use regex over HTML' advice isn't holding up for me.
chubot 6y ago
Yes that's basically what I wrote about here: https://news.ycombinator.com/item?id=26359556 https://news.ycombinator.com/item?id=26359556 I listed 3 or 4 caveats. CDATA might be another one since I'm not handling those... I've never used it so I left it out. Actually I remember that regular languages weren't entirely enough; I think the .*? non-greedy matching was useful, e.g. for finding --> at the end of comments.

legec 6y ago

Where are all the comments gone ? (note : I mean the comments on StackOverflow, not the comments here in Hacker News ... )

deleted 6y ago

[deleted]

loxias 6y ago

This never gets old.

a1369209993 6y ago

You can't parse fully-general HTML with regex, but unless you're writing a web browser of something, that's not what you're trying to do; you're trying to parse the particular subset of HTML that happens to be emitted by this particular website that you got the HTML to be parsed from. And, much like the halting problem or integer factorization, despite the general case being difficult or impossible, the overwhelming majority of specific cases are easy.

dang 6y ago

If curious, past threads: RegEx match open tags except XHTML self-contained tags - https://news.ycombinator.com/item?id=14942060 https://news.ycombinator.com/item?id=14942060 - Aug 2017 (6 comments) You can't parse [X]HTML with regex - https://news.ycombinator.com/item?id=14155015 https://news.ycombinator.com/item?id=14155015 - April 2017 (1 comment) Why you can't parse HTML with regex - https://news.ycombinator.com/item?id=9728474 https://news.ycombinator.com/item?id=9728474 - June 2015 (2 comments) Can you parse html with regular expressions? - https://news.ycombinator.com/item?id=5264511 https://news.ycombinator.com/item?id=5264511 - Feb 2013 (2 comments) Can regular expressions parse HTML or not? - https://news.ycombinator.com/item?id=5257535 https://news.ycombinator.com/item?id=5257535 - Feb 2013 (23 comments) Regexes Parse XML Just Fine, Actually - https://news.ycombinator.com/item?id=3088402 https://news.ycombinator.com/item?id=3088402 - Oct 2011 (27 comments) Oh Yes You Can Use Regexes to Parse HTML - https://news.ycombinator.com/item?id=2741780 https://news.ycombinator.com/item?id=2741780 - July 2011 (77 comments) Why you should not parse (X)HTML with a Regexp - https://news.ycombinator.com/item?id=2423301 https://news.ycombinator.com/item?id=2423301 - April 2011 (5 comments) Stackoverflow, HTML by Regex, topmost answer - https://news.ycombinator.com/item?id=1487695 https://news.ycombinator.com/item?id=1487695 - July 2010 (37 comments) You Cannot Parse HTML with Regular Expressions - https://news.ycombinator.com/item?id=1274870 https://news.ycombinator.com/item?id=1274870 - April 2010 (7 comments) You can't parse [X]HTML with regex. - https://news.ycombinator.com/item?id=941401 https://news.ycombinator.com/item?id=941401 - Nov 2009 (1 comment) I'm surprised that most of these are so small. There must have been others? (I excluded the boring or empty ones.)

pvg 6y ago
There must have been others? Yog-Sothoth knows the gate. Yog-Sothoth is the gate. Yog-Sothoth is the key and guardian of the gate. Past, present, future, all are one in Yog-Sothoth. He knows where the Old Threads broke through of old, and where They shall break through again.