6 ms·
It is not a question of "theoretical purity", it is a fact that it is simply not possible in the real world to parse a html document using only a regex. But thi
by olavk 11y ago
It is not a question of "theoretical purity", it is a fact that it is simply not possible in the real world to parse a html document using only a regex. But this is often misunderstood as saying they regex shouldn't be used in parsing. Regex'es are quite useful as a part of a parsing process, eg. to locate the tags and attributes and separate them from text. This is the scanning or tokenization step. But after this, you need to create a hierarchical data model corresponding the DOM, and you need to do that outside of a regex. A regex outputs only a string or flat list of strings (in all regex libraries I know of), it does not output a hierarchical data structure, so it follows that you need something more than a single regex to parse a hierarchical data structure like a programming language or a html document.
Only exception would be if you need to extract very specific data items which are not hierarchical in nature. Eg. you want to extract only the specified character encoding or something like that.
Your point about the "perfect HTML parser" is kind of missing the point, since regexes does not magically solve the problem of imperfect HTML either. A perfect HTML parser would be one which implement HTML spec fully, including the error recovery rules. These are quite complex, eg, if you reach a <p> then an open <h1> is implicitly closed, but an open <i> is not. Try implementing this logic in a single regex! ("Regexes does not care about perfect syntax" - what does that even mean? A regex match exactly what you tell it to match, just like a parser parses what you program it to parse.)
- joosters 11y agoI guess I must be doing something impossible then. I've been happily screen-scraping many different websites with regexes over the last ten years. The data I gather is critical to my work. And yet, regexes are fine. Your viewpoint is the exact problem I'm complaining about. You think what I am doing is impossible. How can you be so certain? That's the sign of someone who is too tied up in getting something perfect. In fact, while I've hit many problems in gathering data over those years, I can't think of any problem that has been because of regex limitations. And none of my regexes are overly complicated or rely on obscure extensions. Some sites get redesigned and I have to adjust my code, but those redesigns would break any kind of parsing as the pages got completely redesigned (and the URLs and site structure often change as part of this...) If you need to gather data from a website, your real problems will be in the networking and reliability side of things.
- sp332 11y agoYou're probably using some extension and not a "strict" regular expression engine. Regular expressions describe regular languages, which are level 3 in the Chomsky hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy and they formally, provably, do not have the expressive power to describe HTML. This has already been posted in this thread, but make sure to read Larry Wall's quote in the second answer: http://stackoverflow.com/questions/6751105/why-its-not-possible-to-use-regex-to-parse-html-xml-a-formal-explanation-in-la http://stackoverflow.com/questions/6751105/why-its-not-possi...
- koide 11y agoMost (all?) common languages use extended variety regex engines. It is really weird seeing smart people talking about this issue in the no regex works camp. I think it's probably because they haven't had to grab a few specific data points from websites ever.
- olavk 11y agoPossibly there is some confusion about the word "parsing". If it is just the question of scanning for some substring or pattern on a webpage, sure you can do that with a regex. But this is not what is usually considered parsing.
- koide 11y agoRight. And that confusion is probably fueled by the existence of smart (invalid html swallowing) parsers mainly used in scraping, like beautiful soup and nokogiri.
- inglor 11y agoIt's possible to a certain degree, but it's just way way harder then you'd do if you used a proper parser. For example, let's say I want to get all the recent question links off http://stackoverflow.com/questions/new http://stackoverflow.com/questions/new , With RegExp, I can write a clever thing that parses `<a href="(.?*)" class="question-link"` and _hope_ that the order never changes and the class never comes before the href and that they don't change that in a design or in other pages and that the href does not contain `class="question-link` inside it and to a degree that's valid. The thing is - the alternative is to write a query selector - which is another more suitable domain specific language for making selections - only on the DOM instead of text, I'd just write `$(".question-link").get().map(x => x.href)` to get the hrefs and I know it's __always perfectly safe__. Now that example is trivial, if I only want links where the questions are tagged with C#, I get a much harder problem with Regex, but with query selectors it's still mostly trivial. So, it's not that it's particularly hard to use regular expressions to solve it, it's just a lot harder than the alternative which is super simple and obviously correct.