3 ms·

An ordinary regex obviously can't parse html, because html is not regular (given nested elements and the pumping lemma). But what you can easily do with a rege

An ordinary regex obviously can't parse html, because html is not regular (given nested elements and the pumping lemma). But what you can easily do with a regex is to tokenize html - extract anything that looks like a start tag, for example. The simply approach will obviously get some things wrong - a tag inside a comment is meaningless, for example - but for a lot of uses, this difference simply isn't important. If the job is to extract all links from a webpage, regexes will do just fine, and will probably be easier to write and understand than alternate approaches. (This is absolutely not a fair comparison of course; you could compare writing a regex engine to writing a html parser. But I digress.) If the job is to determine whether a given webpage is a member of the set that includes all valid html documents. then a regex is not sufficient. If the job is to extract a list of syntax tokens from a webpage, a regex is likely fine. If the job is to assign semantic meaning to every token in that list, a regex just won't work. Either way, the point is to know what you're doing. Much "parsing" of webpages is not parsing in the formal language sense, and who cares that it isn't because it doesn't need to be.

paulmd 11y ago
No, tasks like "extracting all links from a webpage" are absolutely trivial using an HTML parser. Run the following XPath query on an HTML object: //a[@class='specified_string']/@href Yes, you have to understand the syntax XPath to write such expressions, just like you have to know the language of regexes. Or at least be able to google them. The answer to "who cares" is "you", because you're the one who's going to catch hell when your regex failed to capture some hyperlink that utilized some feature of XML that exceeded your test-cases. The one-liner above is guaranteed to Just Work on all valid XML documents, so why even create such a monstrosity? Everyone knows that Regexes Cannot Parse HTML, and yet people still try it because they think they're smarter than Noam Chompsky. The real truth is that everything looks like a nail to these people, because all they have is a hammer.
- desas 11y ago
  Because valid documents are rarer than documents where regex parsing is good enough.
  - paulmd 11y ago
    Then you're not talking about parsing HTML/XML, are you? How could you possibly know which links or syntax tokens are actually going to be displayed on a page if you feed the browser's parser an invalid document? There are fault-tolerant HTML parsers like TagSoup that are specifically designed to handle dirty HTML and spit out a valid document object. If you have sources that are malformed badly enough that it's still not working, you can define custom SAX properties to handle them. But a task like that is certainly a best-case effort and the interpretation of such a library is no more or less valid than the interpretation of the browser's parser. It's not a valid document to start with and nothing can make it so. If you are only parsing values out of a single specific data template, you know it's not going to parse as HTML or XML, you know that it's never going to contain weird values, and you know it's never going to change - then go hog wild. But it's fundamentally a brittle approach that only holds as long as those assumptions do. I've made the mistake of believing some of those about my data and it's bitten me before. And I really question the implicit assertion that "most html parsing" would fall into that exceedingly narrow category. Especially after a couple years of feature creep. Just keep your logic general and normalize your data. Offer a failover to a fault-tolerant parser in your data layer with a logged warning. This is much more durable and doesn't silently generate invalid tokens or silently fail to capture valid tokens. Regexes simply cannot offer the capability to fail loudly. So once you are no longer actively babysitting your custom regex parser it could have started failing at any time - how would you even know?

paulmd 11y ago

No, tasks like "extracting all links from a webpage" are absolutely trivial using an HTML parser. Run the following XPath query on an HTML object: //a[@class='specified_string']/@href Yes, you have to understand the syntax XPath to write such expressions, just like you have to know the language of regexes. Or at least be able to google them. The answer to "who cares" is "you", because you're the one who's going to catch hell when your regex failed to capture some hyperlink that utilized some feature of XML that exceeded your test-cases. The one-liner above is guaranteed to Just Work on all valid XML documents, so why even create such a monstrosity? Everyone knows that Regexes Cannot Parse HTML, and yet people still try it because they think they're smarter than Noam Chompsky. The real truth is that everything looks like a nail to these people, because all they have is a hammer.

desas 11y ago
Because valid documents are rarer than documents where regex parsing is good enough.
- paulmd 11y ago
  Then you're not talking about parsing HTML/XML, are you? How could you possibly know which links or syntax tokens are actually going to be displayed on a page if you feed the browser's parser an invalid document? There are fault-tolerant HTML parsers like TagSoup that are specifically designed to handle dirty HTML and spit out a valid document object. If you have sources that are malformed badly enough that it's still not working, you can define custom SAX properties to handle them. But a task like that is certainly a best-case effort and the interpretation of such a library is no more or less valid than the interpretation of the browser's parser. It's not a valid document to start with and nothing can make it so. If you are only parsing values out of a single specific data template, you know it's not going to parse as HTML or XML, you know that it's never going to contain weird values, and you know it's never going to change - then go hog wild. But it's fundamentally a brittle approach that only holds as long as those assumptions do. I've made the mistake of believing some of those about my data and it's bitten me before. And I really question the implicit assertion that "most html parsing" would fall into that exceedingly narrow category. Especially after a couple years of feature creep. Just keep your logic general and normalize your data. Offer a failover to a fault-tolerant parser in your data layer with a logged warning. This is much more durable and doesn't silently generate invalid tokens or silently fail to capture valid tokens. Regexes simply cannot offer the capability to fail loudly. So once you are no longer actively babysitting your custom regex parser it could have started failing at any time - how would you even know?