4 ms·

I agree with you, but I may not have made my point clearly enough: It's not about "should he use regexes?" it's the other classic mistake of not clarifying the

by bonaldi 17y ago

I agree with you, but I may not have made my point clearly enough: It's not about "should he use regexes?" it's the other classic mistake of not clarifying the problem domain first. Everybody is assuming he wants to parse this document, which isn't proven and is mostly unfounded. What if it's the text of a book, including all sorts of mentalism that will blow up any parser? Nowhere does it say that he's got valid (or invalid) HTML/XML here. It's actually as if he said "hey, can I borrow a screwdriver?" and everyone went "He wants a screwdriver? He must be building a box, and that means nails! Give him a hammer! A hammer! You can't put in nails with a screwdriver, Zalgo wouldn't do that!" So now he's off trying to put his screws in with a hammer, because in the race for points people wouldn't first ask him to clarify the problem domain and work out what he's actually trying to achieve.

jordanb 17y ago
He narrowed his problem domain to the one suited to parsers when he said he had a structured document. He did that when he said "HTML." Regular expressions are a tool for extracting (semi)-structured patterns from unstructured text. For instance, it is unfortunately the case that we must treat English text as unstructured. It is not possible (at least not yet) to build a parser that will be able to correctly discern every syntactical structure in English text. So if you want to operate on that text (suppose, slicing it into sentences) with a computer, then doing something incredibly superficial --- say, looking for periods --- is the best that can be done. It may need special casing for abbrevs., etc, but it would still produce something, even when the parser would get confused because it couldn't find the predicate of half the sentences. Now, granted, HTML (or SGML) is not the same thing as XML. But the fact is that HTML is by definition structured. Even bad HTML is still structured.. and it's still machine-translatable because it has to be renderable by at least one browser (and therefore understandable by at least one quirks-mode SGML parser). Even the most technically incompetent designer in the world is aware that he has to load the page he's butchered into at least one browser and make sure it still renders. Trying to use a strict XML parser on HTML will likely end in tears, because it has a different definition of validity than the quirks-mode SGML parser the designer used to check his work. But Beautiful Soup is not a strict XML parser, it's an SGML tool. One that, in my experience, is a pretty good approximation of the parsers the browsers use.
- bonaldi 17y ago
  He did that when he said "HTML." He didn't say that. There's not a single "HTML" anywhere in his question. (There's an "XHTML", but it's in the title as something he wants to exclude.) Everyone is making that assumption on his behalf, which is my point.
  - jordanb 17y ago
    You're right. :x It was a while since I looked at the original question. It would be interesting to hear what he's actually trying to do. Looking closely at the question, he wants all open tags, but not ones that are self-closing. My guess is that he is trying to find HTML elements that have content. That should, of course, be done with a parser. But you're right that he could be doing something completely tangential to the structure.
Tichy 17y ago
Even if he just wants to identify a single opening tag his task seems quite hopeless, though. Since the tag can have any number of attributes and apparently attributes can have all sorts of values (inluding "<" according to another SO comment). Maybe it is not impossible, but it is unlikely to be pretty.

by bonaldi 17y ago

jordanb 17y ago

He narrowed his problem domain to the one suited to parsers when he said he had a structured document. He did that when he said "HTML." Regular expressions are a tool for extracting (semi)-structured patterns from unstructured text. For instance, it is unfortunately the case that we must treat English text as unstructured. It is not possible (at least not yet) to build a parser that will be able to correctly discern every syntactical structure in English text. So if you want to operate on that text (suppose, slicing it into sentences) with a computer, then doing something incredibly superficial --- say, looking for periods --- is the best that can be done. It may need special casing for abbrevs., etc, but it would still produce something, even when the parser would get confused because it couldn't find the predicate of half the sentences. Now, granted, HTML (or SGML) is not the same thing as XML. But the fact is that HTML is by definition structured. Even bad HTML is still structured.. and it's still machine-translatable because it has to be renderable by at least one browser (and therefore understandable by at least one quirks-mode SGML parser). Even the most technically incompetent designer in the world is aware that he has to load the page he's butchered into at least one browser and make sure it still renders. Trying to use a strict XML parser on HTML will likely end in tears, because it has a different definition of validity than the quirks-mode SGML parser the designer used to check his work. But Beautiful Soup is not a strict XML parser, it's an SGML tool. One that, in my experience, is a pretty good approximation of the parsers the browsers use.

bonaldi 17y ago
He did that when he said "HTML." He didn't say that. There's not a single "HTML" anywhere in his question. (There's an "XHTML", but it's in the title as something he wants to exclude.) Everyone is making that assumption on his behalf, which is my point.
- jordanb 17y ago
  You're right. :x It was a while since I looked at the original question. It would be interesting to hear what he's actually trying to do. Looking closely at the question, he wants all open tags, but not ones that are self-closing. My guess is that he is trying to find HTML elements that have content. That should, of course, be done with a parser. But you're right that he could be doing something completely tangential to the structure.

Tichy 17y ago

Even if he just wants to identify a single opening tag his task seems quite hopeless, though. Since the tag can have any number of attributes and apparently attributes can have all sorts of values (inluding "<" according to another SO comment). Maybe it is not impossible, but it is unlikely to be pretty.