5 ms·
As someone who has created more hacks like you describe than I care to admit, you actually make the case of precisely why you don't parse HTML with regex. Beca
by phil21 8y ago
As someone who has created more hacks like you describe than I care to admit, you actually make the case of precisely why you don't parse HTML with regex.
Because that code works fine for 5 years until it blows up after something upstream changes. From a RoI calculation this might be perfectly fine if you're still around and remember how to fix it - it's a 5 minute fix after all! But if you've left, or if that system became old and crufty and fragile you may have just burned 3 days tracking it down.
If someone is asking this question, it is entirely appropriate to tell them "don't ever do that" - they obviously lack the experience and big picture thinking to know when it is appropriate to deviate due to business reasons. If you leave it at that I agree you're being a dick - whenever shooting someone's idea down you had damn well better have some appropriate alternative options.
- dave84 8y agoSerious question... why would any other approach be exempt from the problems you stated?
- phil21 8y agoNothing is exempt, some things are far less fragile than others though.
- ivanhoe 8y agoBut are also significantly slower and consume hundreds of times more memory, and will still break with the first little change in the DOM tree on the path that you rely on - which smart regexps can sometimes handle. For instance if you're looking just for a title and a price of a single product (common requirement for spiders), you can extract that with regexp without caring about the html page at all, just pick the <h1> and something that looks like a dollar value. I had spiders like that kept working after a full redesign of the site where even the platform was moved from WP to Magento. No DOM parser could handle that, plus they would be too slow in the first place. You need to pick the right tools for the job, everything else is BS...
- jerf 8y ago"Because that code works fine for 5 years until it blows up after something upstream changes." I believe the point here is exactly that nothing will change, because the regex in question isn't for a service, it is just for that specific file, right now, today. I've regexed specific HTML files myself too, because even though I am very comfortable with XPath and Beautiful Soup and tree representations in general, regexes are even easier on a static file like that.
- phil21 8y agoWe're not talking about one-liners used on a single file here (as far as I could tell) - if that's the case, use whatever quickest hack you can think of to get the job done. Use-once coding is entirely different - everyone loves some code golf once in a while, but it doesn't mean you commit that to git :) For re-usable tooling though, after some time you tend to avoid design patterns that you've personal witnessed break down repeatedly and cause issues.
- jerf 8y ago"We're not talking about one-liners used on a single file here (as far as I could tell)" Well, we are, because to quote coldtea, the person you directly replied to, "'Don't have a continuously running service parse arbitrary HTML with regular expressions' would be a bad thing. Parsing specific, given, HTML files, with known structure to the dev, with regular expressions (e.g. as part of a one-off scrapping script) is totally, absolutely, fine." In context the first sentence clearly means that running the service like that would be a bad thing (hooray English and it's deep ambiguities in double negatives). I still wouldn't necessarily be too upset about telling a junior dev to be suspicious of REs or avoid them (for instance, parsing HTML with REs typically requires using non-greedy matches, and there are some subtleties around that), but if you know what you're doing on a quick job it's fine. If you're a senior dev and you still haven't figured out what's likely to entrench itself and what really is a one-off job, well, that's your real problem. I haven't been surprised about what gets entrenched in quite a while.
- pwinnski 8y agoWhen I was young, I was asked to print some address labels, so I wrote a quick super-short BASIC program to parse a specific file, right now, today, and got those labels printed. Made all sorts of assumptions, but it didn't matter, because it was a static file, and nothing could ever change in it. Three or four years later, I was having lunch with the guy I'd done that for when he got a phone call. Turns out they were using that program of mine to print labels monthly now, and one of the completely-safe assumptions I'd made years earlier had bitten them. Fortunately, he knew how to resolve it easily, but I learned a very important lesson that day.
- watwut 8y agoThere are plenty of disposable code situations where the code wont break after 5 years, because it does not exist after two months. Also, failing regexp on likely generated html like that likely wont take 3 days to find. That is ridiculously high estimate even for messy codebase. You are trying to make estimates and decisions for project you know literally nothing about. That is about worst practice of them all. In case the question was asked by someone lacking the experience - well then it is absolutely ok to learn regexp by trying to parse info few downloaded files of toy scrapped site. Having newbie fight like this just to learn makes no sense.
- majewsky 8y ago> failing regexp on likely generated html like that likely wont take 3 days to find This is true, but only because you're implying that the reason (failing regexp on HTML input) has already been identified. Maybe the parsing job fails silently and then people down the line (who have no idea that the data was passed around as HTML at some point) wonder why no data is coming in anymore, or why they're seeing incomplete data. That could take weeks to trace back to the parsing problem.
- watwut 8y agoSilent failure seems to be root cause of delay in situation you described. Ast based parsing can be made to swallow errrors too.
- Franciscouzo 8y agoAnd parsing the HTML to an AST won't work either in 5 years, the format of the tree will probably have changed and you will be getting index errors.