3 ms·

It is, but it doesn't answer the question. The asker doesn't ask how to parse HTML, he wants to match certain tag-like patterns, and not others. He doesn't defi

by bonaldi 17y ago

It is, but it doesn't answer the question. The asker doesn't ask how to parse HTML, he wants to match certain tag-like patterns, and not others. He doesn't define the document, or the context, and could plausibly be wanting to remove certain tags from a non-HTML text document or the like. There are plenty of potential areas involving HTML tags where you can manipulate the document with regexes, so long as you're not actually trying to make a proper parsing job of it. But everyone was so keen to show that they know you don't use regexes to parse HTML, they blew the actual question away. The asker did try to point out that he didn't want to parse the HTML, but still the points kept on rolling up for an answer that ... told him not to try and parse the HTML.

jordanb 17y ago
While I agree that using regexes for one-off jobs is not the kind of thing that would summon Cthulhu. I think it's still the wrong thing to do and you would still be better off using a parser. I think, to sum up this whole drama, we have one group saying "you should use a parser" and the other group saying "but sometimes regexes aren't totally horrible, and this is just a quick job. I don't want to take the effort to use the proper tool." The latter group sees themselves as being pretty reasonable. And perhaps they are. But the thing that exasperates the former group is that it is generally not quicker or easier to use regexps -- even for one-off jobs --- than using a nice parser like BS or lxml. Those tools give you a object-view of the document which, as JGC demonstrated, allows you to express what you want to do concisely and easily, even (especially) when you're picking a few tags out of the document. The reason why regex hacking on X/HTML documents remains popular isn't because it's 'easy' but because it doesn't require people knowing how to use the proper tools, nor does it require people to make the conceptual leap from the document as a long string of unstructured text to the document being a tree-structure of objects. It's a classic "The Wrong Way" choice made by people who don't have a mechanical or conceptual understanding of the right way. It's like this. Imagine you come upon someone banging in screws with a hammer. And laying next to them is a nice power drill with a Phillips bit. You say to them "why aren't you using the drill for that. It'd do a better job and be a hell of a lot easier too". And the guy responds "This is a quick job. These screws don't need to hold very much weight, or for very long. If this was a serious piece of construction I would totally use the drill, but for my needs this is just easier." Now, you might look at him with some incredulity. No way is his way "easier." It's much harder, on top of doing a worse job. The real reason why he's not using the drill is, quite obviously, he doesn't know how! So you say to him "seriously, try the drill" and he keeps on insisting that his approach is "totally sufficient" (which it may well be, that's not the point) and that using the drill would be "too much work." And then that's how we got here in this discussion.
- PHP-TROLL 17y ago
  Why is it better? I think HTML parsers have their place, but I don't understand why people think regular expressions are not a valid tool for getting data out of HTML. HTML parsers are only an appropriate solution if the HTML is valid, well structured and mostly semantic. If you are dealing with invalid, code generated via Frontpage 2000 by some bureaucrat with little HTML knowledge, regular expressions will almost always be a better option.
  - RyanMcGreal 17y ago
    > HTML parsers are only an appropriate solution if the HTML is valid, well structured and mostly semantic. Tell that to all the people using BeautifulSoup and hpricot.
- bonaldi 17y ago
  I agree with you, but I may not have made my point clearly enough: It's not about "should he use regexes?" it's the other classic mistake of not clarifying the problem domain first. Everybody is assuming he wants to parse this document, which isn't proven and is mostly unfounded. What if it's the text of a book, including all sorts of mentalism that will blow up any parser? Nowhere does it say that he's got valid (or invalid) HTML/XML here. It's actually as if he said "hey, can I borrow a screwdriver?" and everyone went "He wants a screwdriver? He must be building a box, and that means nails! Give him a hammer! A hammer! You can't put in nails with a screwdriver, Zalgo wouldn't do that!" So now he's off trying to put his screws in with a hammer, because in the race for points people wouldn't first ask him to clarify the problem domain and work out what he's actually trying to achieve.
  - jordanb 17y ago
    He narrowed his problem domain to the one suited to parsers when he said he had a structured document. He did that when he said "HTML." Regular expressions are a tool for extracting (semi)-structured patterns from unstructured text. For instance, it is unfortunately the case that we must treat English text as unstructured. It is not possible (at least not yet) to build a parser that will be able to correctly discern every syntactical structure in English text. So if you want to operate on that text (suppose, slicing it into sentences) with a computer, then doing something incredibly superficial --- say, looking for periods --- is the best that can be done. It may need special casing for abbrevs., etc, but it would still produce something, even when the parser would get confused because it couldn't find the predicate of half the sentences. Now, granted, HTML (or SGML) is not the same thing as XML. But the fact is that HTML is by definition structured. Even bad HTML is still structured.. and it's still machine-translatable because it has to be renderable by at least one browser (and therefore understandable by at least one quirks-mode SGML parser). Even the most technically incompetent designer in the world is aware that he has to load the page he's butchered into at least one browser and make sure it still renders. Trying to use a strict XML parser on HTML will likely end in tears, because it has a different definition of validity than the quirks-mode SGML parser the designer used to check his work. But Beautiful Soup is not a strict XML parser, it's an SGML tool. One that, in my experience, is a pretty good approximation of the parsers the browsers use.

by bonaldi 17y ago

jordanb 17y ago

While I agree that using regexes for one-off jobs is not the kind of thing that would summon Cthulhu. I think it's still the wrong thing to do and you would still be better off using a parser. I think, to sum up this whole drama, we have one group saying "you should use a parser" and the other group saying "but sometimes regexes aren't totally horrible, and this is just a quick job. I don't want to take the effort to use the proper tool." The latter group sees themselves as being pretty reasonable. And perhaps they are. But the thing that exasperates the former group is that it is generally not quicker or easier to use regexps -- even for one-off jobs --- than using a nice parser like BS or lxml. Those tools give you a object-view of the document which, as JGC demonstrated, allows you to express what you want to do concisely and easily, even (especially) when you're picking a few tags out of the document. The reason why regex hacking on X/HTML documents remains popular isn't because it's 'easy' but because it doesn't require people knowing how to use the proper tools, nor does it require people to make the conceptual leap from the document as a long string of unstructured text to the document being a tree-structure of objects. It's a classic "The Wrong Way" choice made by people who don't have a mechanical or conceptual understanding of the right way. It's like this. Imagine you come upon someone banging in screws with a hammer. And laying next to them is a nice power drill with a Phillips bit. You say to them "why aren't you using the drill for that. It'd do a better job and be a hell of a lot easier too". And the guy responds "This is a quick job. These screws don't need to hold very much weight, or for very long. If this was a serious piece of construction I would totally use the drill, but for my needs this is just easier." Now, you might look at him with some incredulity. No way is his way "easier." It's much harder, on top of doing a worse job. The real reason why he's not using the drill is, quite obviously, he doesn't know how! So you say to him "seriously, try the drill" and he keeps on insisting that his approach is "totally sufficient" (which it may well be, that's not the point) and that using the drill would be "too much work." And then that's how we got here in this discussion.

PHP-TROLL 17y ago
Why is it better? I think HTML parsers have their place, but I don't understand why people think regular expressions are not a valid tool for getting data out of HTML. HTML parsers are only an appropriate solution if the HTML is valid, well structured and mostly semantic. If you are dealing with invalid, code generated via Frontpage 2000 by some bureaucrat with little HTML knowledge, regular expressions will almost always be a better option.
- RyanMcGreal 17y ago
  > HTML parsers are only an appropriate solution if the HTML is valid, well structured and mostly semantic. Tell that to all the people using BeautifulSoup and hpricot.
bonaldi 17y ago
I agree with you, but I may not have made my point clearly enough: It's not about "should he use regexes?" it's the other classic mistake of not clarifying the problem domain first. Everybody is assuming he wants to parse this document, which isn't proven and is mostly unfounded. What if it's the text of a book, including all sorts of mentalism that will blow up any parser? Nowhere does it say that he's got valid (or invalid) HTML/XML here. It's actually as if he said "hey, can I borrow a screwdriver?" and everyone went "He wants a screwdriver? He must be building a box, and that means nails! Give him a hammer! A hammer! You can't put in nails with a screwdriver, Zalgo wouldn't do that!" So now he's off trying to put his screws in with a hammer, because in the race for points people wouldn't first ask him to clarify the problem domain and work out what he's actually trying to achieve.
- jordanb 17y ago
  He narrowed his problem domain to the one suited to parsers when he said he had a structured document. He did that when he said "HTML." Regular expressions are a tool for extracting (semi)-structured patterns from unstructured text. For instance, it is unfortunately the case that we must treat English text as unstructured. It is not possible (at least not yet) to build a parser that will be able to correctly discern every syntactical structure in English text. So if you want to operate on that text (suppose, slicing it into sentences) with a computer, then doing something incredibly superficial --- say, looking for periods --- is the best that can be done. It may need special casing for abbrevs., etc, but it would still produce something, even when the parser would get confused because it couldn't find the predicate of half the sentences. Now, granted, HTML (or SGML) is not the same thing as XML. But the fact is that HTML is by definition structured. Even bad HTML is still structured.. and it's still machine-translatable because it has to be renderable by at least one browser (and therefore understandable by at least one quirks-mode SGML parser). Even the most technically incompetent designer in the world is aware that he has to load the page he's butchered into at least one browser and make sure it still renders. Trying to use a strict XML parser on HTML will likely end in tears, because it has a different definition of validity than the quirks-mode SGML parser the designer used to check his work. But Beautiful Soup is not a strict XML parser, it's an SGML tool. One that, in my experience, is a pretty good approximation of the parsers the browsers use.