3 ms·

I guess I must be doing something impossible then. I've been happily screen-scraping many different websites with regexes over the last ten years. The data I ga

I guess I must be doing something impossible then. I've been happily screen-scraping many different websites with regexes over the last ten years. The data I gather is critical to my work. And yet, regexes are fine. Your viewpoint is the exact problem I'm complaining about. You think what I am doing is impossible. How can you be so certain? That's the sign of someone who is too tied up in getting something perfect. In fact, while I've hit many problems in gathering data over those years, I can't think of any problem that has been because of regex limitations. And none of my regexes are overly complicated or rely on obscure extensions. Some sites get redesigned and I have to adjust my code, but those redesigns would break any kind of parsing as the pages got completely redesigned (and the URLs and site structure often change as part of this...) If you need to gather data from a website, your real problems will be in the networking and reliability side of things.

sp332 11y ago
You're probably using some extension and not a "strict" regular expression engine. Regular expressions describe regular languages, which are level 3 in the Chomsky hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy and they formally, provably, do not have the expressive power to describe HTML. This has already been posted in this thread, but make sure to read Larry Wall's quote in the second answer: http://stackoverflow.com/questions/6751105/why-its-not-possible-to-use-regex-to-parse-html-xml-a-formal-explanation-in-la http://stackoverflow.com/questions/6751105/why-its-not-possi...
- koide 11y ago
  Most (all?) common languages use extended variety regex engines. It is really weird seeing smart people talking about this issue in the no regex works camp. I think it's probably because they haven't had to grab a few specific data points from websites ever.
  - olavk 11y ago
    Possibly there is some confusion about the word "parsing". If it is just the question of scanning for some substring or pattern on a webpage, sure you can do that with a regex. But this is not what is usually considered parsing.
    - koide 11y ago
      Right. And that confusion is probably fueled by the existence of smart (invalid html swallowing) parsers mainly used in scraping, like beautiful soup and nokogiri.
inglor 11y ago
It's possible to a certain degree, but it's just way way harder then you'd do if you used a proper parser. For example, let's say I want to get all the recent question links off http://stackoverflow.com/questions/new http://stackoverflow.com/questions/new , With RegExp, I can write a clever thing that parses `<a href="(.?*)" class="question-link"` and _hope_ that the order never changes and the class never comes before the href and that they don't change that in a design or in other pages and that the href does not contain `class="question-link` inside it and to a degree that's valid. The thing is - the alternative is to write a query selector - which is another more suitable domain specific language for making selections - only on the DOM instead of text, I'd just write `$(".question-link").get().map(x => x.href)` to get the hrefs and I know it's __always perfectly safe__. Now that example is trivial, if I only want links where the questions are tagged with C#, I get a much harder problem with Regex, but with query selectors it's still mostly trivial. So, it's not that it's particularly hard to use regular expressions to solve it, it's just a lot harder than the alternative which is super simple and obviously correct.
- joosters 11y ago
  Your method relies on the class names not being renamed (e.g. I see "question-hyperlink" on /questions, not "question-link"). I'd skip the class entirely and match on the URL, since I doubt stackoverflow want their URLs to break: /<a[^>]*href="(\/questions\/\d+[^"]+)"/i But... we can go back and forwards posting examples and find fault in any regex that I post or any selector that you post. It's missing the point. Both methods are at the mercy of web page redesigns. Both methods can be made more robust against certain changes, but cannot survive other changes. You are trying to say that regexes won't work. I am saying that both methods work.
- ifdefdebug 11y ago
  Look, you have essentially two options here: 1) add a full featured DOM parser to your program - do that if you have to understand the DOM; 2) write regex - do that if you don't care about the DOM, if you know how the server formats the data you need and if the server formats the data you need in a reproducible way. > (...) and that they don't change that in a design or in other pages (...) Well if they change the design of the pages then you will have to rewrite your regex accordingly in order to find the data you need. But if that happens, odds are high that your program, which uses a full featured DOM parser, will have to be rewritten as well in order to handle the modified output of the DOM parser...
olavk 11y ago
Maybe you are indeed doing the impossible, but I consider it more likely that you either (a) is using some parsing logic (like a stack, recursion or some such) intermingled with the pure regexes, or (b) is not extracting hierarchically structured data from the html.
- joosters 11y ago
  Could you please stop insisting that I'm doing something impossible? Do you think I am imagining my work? I assure you that no backtracking or recursion are involved. Stacks? Well, sometimes I extract bits of a page with one regex and process it more with further regexes in another function, so I guess you could argue that the language uses a stack for a function call, but still... please stop grasping. I'm very sure that you could invent some website that relied on some hideous deeply recursive complicated structure, and it would be painful to grab data from it. In my experience, those cases are extremely unlikely. If you let these extremely unlikely situations stop you from picking a simple and useful solution for all the other cases, you are indeed getting tied up in trying to be perfect.
  - olavk 11y ago
    I'm not suggesting that you code doesn't work, if that is what your thinking! It's just a terminology thing. What you describe ("sometimes I extract bits of a page with one regex and process it more with further regexes in another function") sounds like a recursive descent parser, which is a very common way to implement a simple and fast parser. I'm not suggesting you can't parse html with the help of regexes, I'm just stating that you cannot parse html only with a regex: You cannot apply a single regex to the html and get something similar to a DOM out. You need some kind of stack or recursive programming logic in order to extract a recursive or hierarchical data structure from a string.

sp332 11y ago

You're probably using some extension and not a "strict" regular expression engine. Regular expressions describe regular languages, which are level 3 in the Chomsky hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy https://en.wikipedia.org/wiki/Chomsky_hierarchy and they formally, provably, do not have the expressive power to describe HTML. This has already been posted in this thread, but make sure to read Larry Wall's quote in the second answer: http://stackoverflow.com/questions/6751105/why-its-not-possible-to-use-regex-to-parse-html-xml-a-formal-explanation-in-la http://stackoverflow.com/questions/6751105/why-its-not-possi...

koide 11y ago
Most (all?) common languages use extended variety regex engines. It is really weird seeing smart people talking about this issue in the no regex works camp. I think it's probably because they haven't had to grab a few specific data points from websites ever.
- olavk 11y ago
  Possibly there is some confusion about the word "parsing". If it is just the question of scanning for some substring or pattern on a webpage, sure you can do that with a regex. But this is not what is usually considered parsing.
  - koide 11y ago
    Right. And that confusion is probably fueled by the existence of smart (invalid html swallowing) parsers mainly used in scraping, like beautiful soup and nokogiri.

inglor 11y ago

It's possible to a certain degree, but it's just way way harder then you'd do if you used a proper parser. For example, let's say I want to get all the recent question links off http://stackoverflow.com/questions/new http://stackoverflow.com/questions/new , With RegExp, I can write a clever thing that parses `<a href="(.?*)" class="question-link"` and _hope_ that the order never changes and the class never comes before the href and that they don't change that in a design or in other pages and that the href does not contain `class="question-link` inside it and to a degree that's valid. The thing is - the alternative is to write a query selector - which is another more suitable domain specific language for making selections - only on the DOM instead of text, I'd just write `$(".question-link").get().map(x => x.href)` to get the hrefs and I know it's __always perfectly safe__. Now that example is trivial, if I only want links where the questions are tagged with C#, I get a much harder problem with Regex, but with query selectors it's still mostly trivial. So, it's not that it's particularly hard to use regular expressions to solve it, it's just a lot harder than the alternative which is super simple and obviously correct.

joosters 11y ago
Your method relies on the class names not being renamed (e.g. I see "question-hyperlink" on /questions, not "question-link"). I'd skip the class entirely and match on the URL, since I doubt stackoverflow want their URLs to break: /<a[^>]*href="(\/questions\/\d+[^"]+)"/i But... we can go back and forwards posting examples and find fault in any regex that I post or any selector that you post. It's missing the point. Both methods are at the mercy of web page redesigns. Both methods can be made more robust against certain changes, but cannot survive other changes. You are trying to say that regexes won't work. I am saying that both methods work.
ifdefdebug 11y ago
Look, you have essentially two options here: 1) add a full featured DOM parser to your program - do that if you have to understand the DOM; 2) write regex - do that if you don't care about the DOM, if you know how the server formats the data you need and if the server formats the data you need in a reproducible way. > (...) and that they don't change that in a design or in other pages (...) Well if they change the design of the pages then you will have to rewrite your regex accordingly in order to find the data you need. But if that happens, odds are high that your program, which uses a full featured DOM parser, will have to be rewritten as well in order to handle the modified output of the DOM parser...

olavk 11y ago

Maybe you are indeed doing the impossible, but I consider it more likely that you either (a) is using some parsing logic (like a stack, recursion or some such) intermingled with the pure regexes, or (b) is not extracting hierarchically structured data from the html.

joosters 11y ago
Could you please stop insisting that I'm doing something impossible? Do you think I am imagining my work? I assure you that no backtracking or recursion are involved. Stacks? Well, sometimes I extract bits of a page with one regex and process it more with further regexes in another function, so I guess you could argue that the language uses a stack for a function call, but still... please stop grasping. I'm very sure that you could invent some website that relied on some hideous deeply recursive complicated structure, and it would be painful to grab data from it. In my experience, those cases are extremely unlikely. If you let these extremely unlikely situations stop you from picking a simple and useful solution for all the other cases, you are indeed getting tied up in trying to be perfect.
- olavk 11y ago
  I'm not suggesting that you code doesn't work, if that is what your thinking! It's just a terminology thing. What you describe ("sometimes I extract bits of a page with one regex and process it more with further regexes in another function") sounds like a recursive descent parser, which is a very common way to implement a simple and fast parser. I'm not suggesting you can't parse html with the help of regexes, I'm just stating that you cannot parse html only with a regex: You cannot apply a single regex to the html and get something similar to a DOM out. You need some kind of stack or recursive programming logic in order to extract a recursive or hierarchical data structure from a string.