8 ms·
Oh Yes You Can Use Regexes to Parse HTML
- draegtun 15y agoAnd related this previous HN discussion on different SO question RegEx match open tags except XHTML self-contained tags (http://news.ycombinator.com/item?id=1487695 http://news.ycombinator.com/item?id=1487695)
- masklinn 15y agoYou can use perl's string-matching facilities (which really are not regular expressions at all[0]) to parse HTML. [0] in fact it was a rather neat idea to rename those "patterns" (or something along those lines) in Perl 6. Unfortunately this name change has been rolled back. Shame.
- lloeki 15y agoThere was some form of consensus in a recent discussion not particularly pertaining to Perl that the term regex did not equate anymore to regular expression.
- gjm11 15y agoA slightly more accurate summary of Tom Christiansen's excellent answer there would be: "Oh yes you can use regexes to parse HTML, but you usually shouldn't, unless what you want to do is really, really simple." Actual quotations: "Even if my program is taken as illustrative of why you should not use regexes for parsing general HTML -- which is ok, because I kinda meant for it to be that"; "That was kinda my point, actually. I wanted to show how hard it is." (the latter in response to someone else who said "You can write a novel, like tchrist did, or you can use a DOM library and write one line of XPath").
- cafebabe 15y agoThere is a problem: The document must be valid. That's why people struggle with Regex instead. Or you have to use or build a fault tolerant parser.
- lloeki 15y agoThat said, his HTML chunker is, dare I say, gorgeous. If that is an example of what should not be done, I wish there was more of them like that around. Besides, lexing HTML in 234 lines grand total, most of them being whitespace, (169 SLOCs according to sloccount) is impressive. Writing even a basic non regex-based parser is bound to take quite some space. To me the real conclusion is not: "don't try to parse random HTML using regexes" but "don't try to write your own wide-purpose HTML parser". Or, as Tom put it in his SO answer: > The correct and honest answer is that they shouldn’t attempt [trying to parse arbitrary HTML] because it is too much of a bother to figure out from scratch
- jerf 15y ago"Besides, lexing HTML in 234 lines grand total, most of them being whitespace, (169 SLOCs according to sloccount) is impressive." I mean no disrespect at all to tchrist, but it isn't impressive at all; not because tchrist is wrong, but because lexing isn't hard. If you understand the problem, you can almost literally read the lexer right off the standard; indeed, that's part of the purpose of the standard. Look at it (taking HTML4 here as it's easier to see): http://www.w3.org/TR/html401/types.html#h-6.2 http://www.w3.org/TR/html401/types.html#h-6.2 You can literally read off the lexer expression for ID and NAME right from 'must begin with a letter ([A-Za-z]) and may be followed by any number of letters, digits ([0-9]), hyphens ("-"), underscores ("_"), colons (":"), and periods (".").' Generally, if you're having a hard time putting a lexer together for some language you're creating (bearing in mind this includes the broader definition of "language" beyond just "programming language", which includes things like JSON or text formats you may create ad hoc), that's a sign that you've got an overcomplicated language on your hand. (Hi, C++! I see you over there!)
- parenthesis 15y agoFor quick-and-dirty extraction of data from HTML documents, lynx -dump can be useful.
- rickmb 15y agoAnd for the next ten years, flawed attempts at imitating this will show up in production code all around the world... I'm not saying he shouldn't have (I've certainly learned something I didn't know), but let's face it, posting this on StackOverflow is like handing a loaded gun to a bunch of children and telling them not to pull the trigger.
- adobriyan 15y agoHow many problems would novice have if he decides to parse HTML with regexes in Perl?
- praptak 15y agoWhich one is this: a serious expression of concern or some kind of a lightbulb joke?
- telemachos 15y agoPretty sure it's a reference to this famous[1] joke: >> Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems. [1] http://regex.info/blog/2006-09-15/247 http://regex.info/blog/2006-09-15/247
- gloob 15y agoThe correct response to that saying is to point out that regular expressions are significantly simpler than general-purpose, Turing-complete programming languages.
- raganwald 15y agoI'm not saying he shouldn't have (I've certainly learned something I didn't know), but let's face it, posting this on StackOverflow is like handing a loaded gun to a bunch of children and telling them not to pull the trigger. True, but that's the case for posting anything non-trivial where Google can find it. And isomorphic to using non-trivial programming languages or patterns in industrial programs. At a certain point, you have to let go of the idea that if you hide anything requiring thought and judgment from people, they won't hurt themselves. Some people are going to foul things up no matter what you do or don't tell them, and some will excel even if you make it hard for them to find knowledge. The interesting group are those who are open to being influenced by what you have to share. I suggest that even if only a small proportion of SO readers walk away enlightened, this article is a win. The folks who use it to justify their own failed attempts to manage arbitrary HTML with regexen would have pooched the task in some other way even if they stuck to using a parser library.
- retube 15y agoOne of the best and academically proficient answers I've seen on SO. And if I understand correctly turns on its head the old refrain "You can't use regexes to parse HTML", of which I've always been a staunch proponent. Now I understood the reason _why_ you can't use regular expressions to parse HTML is that HTML is usually not regular. Is this true? Does this solution in perl work because of the extended capabilities of perl regexes?
- masklinn 15y ago> Does this solution in perl work because of the extended capabilities of perl regexes? Yes. It works because perl's "regexes" are not regular expressions (nb: almost no "regex engine" these days is limited to regular expressions). Actual regular expressions can express type-3 chomsky grammars, whereas HTML's grammar (and most programming languages's) is of type 2.
- _delirium 15y agoHTML isn't even actually context-free, though you can possibly parse it as context-free (type-2) with some kludges. The kludges are the same as the kludges for parsing it in a regular language, though! (I.e. "all" you need to do is add backreferences.) Recognizing paired start/end tags is equivalent to recognizing palindromes, and the palindrome language is only context-free if there's a finite, fixed alphabet. But the set of HTML tags a parser needs to match isn't finite. While there are a finite number of defined tags in any particular version of the HTML specification, a parser is still supposed to be able to match start/end tags it doesn't recognize, to distinguish the case of malformed syntax from the case of correctly formed syntax that uses unrecognized tags. An easy way to see it is to try to write out CFG productions for HTML. The intuitive way requires something like backreferences, which would make the language no longer context-free: html = '<' + tag + '>' + html + '</' + $0 + '>' But in a context-free language you don't have that, so you instead have to do something like: html = '<head>' + html + '</head>' | '<body>' + html + '</body>' | '<b>' = html + '</b>' | ... which of course only works if the number of tags is finite so that you can exhaustively enumerate them.
- reirob 15y agoWhen I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools iconv: necessary only when the page is NOT encoded in UTF-8 tidy: used to convert from HTML to XHTML which is XML. I call it as xmlstarlet: to extract data from the XML file using XPath. I find XPath a much better and much reliable tool for HTML data extraction.
- masklinn 15y ago> When I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools I just use lxml.html (handles 99.999% of the HTML out there, add beautifulsoup's UnicodeDammit[0] for wonky encodings) and then use lxml's built-in xpath support on top of that. Plus extra bonus, if the datamining paths are simple enough you can use CSS queries instead of XPath. [0] http://lxml.de/elementsoup.html#using-only-the-encoding-detection http://lxml.de/elementsoup.html#using-only-the-encoding-dete...
- reirob 15y agoThanks for pointing this out. I forgot to precise that I am using XPath on Unix/Cygwin command line or in Shell scripts. Here an example of extracting the Hacker News title;url;points;user data: curl -s http://news.ycombinator.com/news | tidy -quiet -asxml -numeric -utf8 -file /dev/null | xmlstarlet sel -N x=http://www.w3.org/1999/xhtml -t -m "//x:tr[x:td[1][@class='title']]" -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" -v "x:td[3][@class='title']/x:a/@href" -o ";" -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | xmlstarlet unesc
- sukuriant 15y agoCould you put some newlines in that? For some reason, my browser (IE9) is rendering that construct as a VERY long line and extending the page to match it. (no code-specific scroll-bar for me)
- 15y ago
- deleted 15y ago[deleted]
- d0m 15y agoErrr.. Did he just wrote an html parser (and then used it) to prove everyone that you can use regex to solve the use-html-parser-instead-of-regex problem?! It's like if I suggest someone to use Python instead of ASM to solve a simple problem, but then someone try to prove me wrong by writing a python interpreter in ASM and then USE it to solve the same problem! Also, that being said, I feel like the post is more a brag about "I'm the creator of a popular perl book and perl rocks your language here's why blahbahblah".
- JonnieCache 15y agoYeah, this is pretty much what seems to be going on. I don't think anyone has ever said that regexs can't be used in the parsing of html, just that they can't be used for the parsing of html. It's like someone saying "You can't use bricks to keep warm!" and countering that by saying "Observe! I have built a house using, among other things, bricks, and it shelters me from the weather and keeps me warm!" Deliberately missing the point in order to show off your housebuilding skills. Still, it was an informative and well written article. Article, rather than an answer to a question. Why do people write these thousands of words on stack overflow, when they could publish them on their blogs. Are SO points from confused corporate employees really worth more than the adulation of the blogosphere? Actually, who cares. I almost lost the will to live just writing that sentence...
- statictype 15y agoI'd rather see it in Stackoverflow. It has much more visibility there than if each person wrote it on their own blog (assuming they have one) which may not be seen by more than a handful of friends.
- iron_ball 15y agoLook at all the people on HN saying they want to hire based on SO rep and a Github account.
- Goladus 15y agoActually I'm more interested in the answer to this question: Was he able write an arbitrary HTML parser using regexes because HTML is a regular language or because Perl's string matching syntax, which can match some languages that are not regular, is called "regular expressions" anyway? I am pretty sure that, for example, an arbitrary number of balanced "<div></div>" pairs are not a regular expression however you can use modern PCRE to match it anyway.
- michael_dorfman 15y agoMeanwhile, the highest-rated answer in StackOverflow history says otherwise: http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 http://stackoverflow.com/questions/1732348/regex-match-open-...
- godDLL 15y agoOP doesn't actually do regex parsing of HTML, so all is well. It does regular 'ol parsing, and uses regex to chunk html into consumables.
- jgrahamc 15y agoI think the point here is don't use regexps for this. Some time ago I got into a spat with Eric Raymond about this very subject. Bottom line is that there are nice libraries for HTML parsing out there. Use one: http://blog.jgc.org/2009/11/parsing-html-in-python-with.html http://blog.jgc.org/2009/11/parsing-html-in-python-with.html
- benmmurphy 15y agoit doesn't handle script tags correctly for the example cited. anything between <script> and </script> shouldn't be interpreted as html. <html> <head> <script type='text/javascript'> var tag = '<input type="hidden" name="foo" value="bar"/>'; </script> </head> <body> body </body> </html> ./html_input_rx test.html input tag #1 at character 57: name => "foo" type => "hidden" value => "bar" but very cool and could probably be fixed to handle that case quite easily.
- sambeau 15y agoI'm just disappointed that he didn't use the flip-flop operator ..
- profquail 15y agoFor parsing HTML, I'd recommend using a purpose-built HTML-parsing library instead of bothering with regular expressions. (Though, like the author of that answer wrote, regex's can work just fine for parsing small snippets.) An interesting fact: you can parse HTML/XHTML (correctly) with some of the popular regex implementations. (Note the word implementations.)
- wazzupflow 15y agobest part is in the answer just below this one: "1. You can write a novel like tchrist did..."
- scrrr 15y agoIsn't the theory that for a regular grammar you can use regexp, for a context-free grammar you need sth. with a stack (= parser). HTML isn't regular, though, is it? So if there's not (even an implicit) stack in his example, this won't work for the general case.
- scott_s 15y agoI classify this as a parser using regular expressions rather than parsing using regular expressions. That is, his regular expressions don't parse the document. He wrote a parser, and uses regular expressions in that parser.
- pak 15y agoThis is correct. As soon as I saw his lex_html function, I could see that he was using regexes to tokenize and then basically drop into different contexts while consuming the document, thereby tracking multiple levels of state throughout. That's what a full-blown parser does and it goes beyond the scope of regular languages. By "parsing with regular expressions" most people mean applying one or two regexes to the entire document and using, for example, the group capture facility to extract information. That is and will always be a bad idea for HTML, because HTML is not a regular language. Applying regular expressions to tokenize a string for a parsing is actually a fairly standard pattern.
- justinhj 15y agoExactly. Gnu's Flex lets the user define regex's for each state of the parser... http://flex.sourceforge.net/manual/Patterns.html#Patterns http://flex.sourceforge.net/manual/Patterns.html#Patterns
- wisty 15y agoRight. Pyparsing is a nice parsing library, and it uses regular expression. I'd expect most parsing libraries do let you define stuff in regex. But they can't do recursion, right?
- jpr 15y agoNow you have two problems.
- ratsbane 15y agoYou still can't use A regular expression to parse HTML. Of course you can use a set of regular expressions and some other logic to parse HTML. There's shouldn't be anything surprising about this post.
- jdnier 15y agoIn the spirit of Tom Christiansen's lexer solution, here's a link to Robert Cameron's seemingly forgotten 1998 article, "REX: XML Shallow Parsing with Regular Expressions". http://www.cs.sfu.ca/~cameron/REX.html """ Abstract The syntax of XML is simple enough that it is possible to parse an XML document into a list of its markup and text items using a single regular expression. Such a shallow parse of an XML document can be very useful for the construction of a variety of lightweight XML processing tools. However, complex regular expressions can be difficult to construct and even more difficult to read. Using a form of literate programming for regular expressions, this paper documents a set of XML shallow parsing expressions that can be used a basis for simple, correct, efficient, robust and language-independent XML shallow parsing. Complete shallow parser implementations of less than 50 lines each in Perl, JavaScript and Lex/Flex are also given. The syntax of XML is simple enough that it is possible to parse an XML document into a list of its markup and text items using a single regular expression. Such a shallow parse of an XML document can be very useful for the construction of a variety of lightweight XML processing tools. However, complex regular expressions can be difficult to construct and even more difficult to read. Using a form of literate programming for regular expressions, this paper documents a set of XML shallow parsing expressions that can be used a basis for simple, correct, efficient, robust and language-independent XML shallow parsing. Complete shallow parser implementations of less than 50 lines each in Perl, JavaScript and Lex/Flex are also given. """
- jdnier 15y agoIf you enjoy reading about regular expressions, Cameron's paper is fascinating. His writing is concise, thorough, and very detailed. He's not simply showing you how to construct the REX regular expression but also an approach for constructing any complex regex. I've been using the REX regular expression on and off for 10 years to solve the sort of problem the initial poster on Stack Overflow asked about (how do I match this particular tag but not some other very similar tag?). I've found the regex he developed to be completely reliable.
- jdnier 15y agoREX is most useful is when you're focusing on lexical details of a document -- for example, when transforming one kind of text document (e.g., plain text, XML, SGML, HTML) into another, where the document may not be valid, well formed, or even parsable for most of the transformation. It lets you target islands of markup anywhere within a document without disturbing the rest of the document.
- linuxhansl 15y agoNo you can't. HTML is a Context Free Language (Type 2 in the Chomsky hierarchy) that is defined by a Context Free Grammar, and parsed by a stack machine. Regular expressions can describe regular (Type 3) languages. They do not have a stack. Note that there is a loop in the code, so it's not just regular expressions.