4 ms·

I have a funny anecdote about CS. I am a CS dropout who has been working in startups for a few years. About once a year, I see another programmer making a comm

by apeace 10y ago

I have a funny anecdote about CS. I am a CS dropout who has been working in startups for a few years. About once a year, I see another programmer making a common mistake, and I draw on my CS knowledge to help them out. The mistake is parsing HTML with regular expressions. It is so tempting to write a good ole' regex to grab that attribute value off of that element. And it works on the 5-6 samples you write your unit tests with. If you have run into this before, you may know that zalgo[0] tends to appear in this situation. Parsing HTML with a regex will always fail eventually. Of course the reason is that HTML is not a regular language. It is a context-free grammar, and thus requires a parser to parse it. The funny part is, I failed CS Theory once, and was in the middle of taking it again when I decided to drop out. But this tidbit of knowledge has always stuck with me, and I've used it again and again to fix or prevent bugs in real software. Takeaway: CS knowledge does have real-world value to everyday programming. You just have to know what you're looking for. And of course, always use an HTML parser to parse HTML. (I'll add one more note to anticipate a common response. If you look at any HTML parser, you will see regular expressions in the code. These regex are used for chunking the HTML, and from that point the chunks are parsed.) [0] https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open... EDIT: Formatting

legulere 10y ago
I think the HTML/XML/JSON is not a regular language story is a bad one, because subsets of them are indeed regular. Most of the time the data you're trying to parse doesn't contain arbitrarily deep nesting and could actually be parsed with a regex. Further regexes of languages like perl can parse a superset of regular languages. The real problem with regexes is that they are hard to maintain and are extremely hard to get right to neither have false positives nor false negatives.
- apeace 10y ago
  That is a fair point. If you're sure you have a regular subset of HTML, a regex is a great way to extract data from it. In my experience (again about once a year), I have always seen this done with HTML that is coming in from the wild. After all, if your HTML is coming from a source you control, you most often have a means to provide the data in a format other than HTML.
- ultramancool 10y ago
  This is one I've never quite understood. Having written many scrapers I will say - Scraping is always hard to maintain. I've written regex and parser based scrapers many times, but it's still a need for constant updates as the page changes. HTML will change about as often (only slightly less in my experiences) than regexes will need to be changed and can take more time to test and develop on each update, especially if you don't have access to a very durable parser which can stand broken HTML, unlike most XML parsers. So if a regex does the job, use it IMO.
GFK_of_xmaspast 10y ago
This is a case where a little knowledge can be a dangerous thing, because "regexes" are often strictly stronger than just "regular expressions". (That said, the moral of this story is still correct, i.e. use a parser, but mostly because html-in-the-wild is uniformly awful and it's better to let someone else worry about that).
brassic 10y ago
I had to scrape the comments out of a blog. As you might expect, there was a div for comments, with comments nested within it. This made it easy to grab the comments using an HTML parser with xpath support. Unfortunately, the blog software had a bug and it was possible for markup to leak out of the comments. Sometimes a spurious </div> would close the comments div and the scraper would miss comments that came after it. However, the HTML did contain helpful HTML comments, something like "comments start here" and "comments end here". The reliable solution was to use a literal string search on these HTML comments to pull out the entire comments section, and then to use regexes to pull out the comments' content. The only unbreakable rule is that there are no unbreakable rules.

by apeace 10y ago

legulere 10y ago

I think the HTML/XML/JSON is not a regular language story is a bad one, because subsets of them are indeed regular. Most of the time the data you're trying to parse doesn't contain arbitrarily deep nesting and could actually be parsed with a regex. Further regexes of languages like perl can parse a superset of regular languages. The real problem with regexes is that they are hard to maintain and are extremely hard to get right to neither have false positives nor false negatives.

apeace 10y ago
That is a fair point. If you're sure you have a regular subset of HTML, a regex is a great way to extract data from it. In my experience (again about once a year), I have always seen this done with HTML that is coming in from the wild. After all, if your HTML is coming from a source you control, you most often have a means to provide the data in a format other than HTML.
ultramancool 10y ago
This is one I've never quite understood. Having written many scrapers I will say - Scraping is always hard to maintain. I've written regex and parser based scrapers many times, but it's still a need for constant updates as the page changes. HTML will change about as often (only slightly less in my experiences) than regexes will need to be changed and can take more time to test and develop on each update, especially if you don't have access to a very durable parser which can stand broken HTML, unlike most XML parsers. So if a regex does the job, use it IMO.

GFK_of_xmaspast 10y ago

This is a case where a little knowledge can be a dangerous thing, because "regexes" are often strictly stronger than just "regular expressions". (That said, the moral of this story is still correct, i.e. use a parser, but mostly because html-in-the-wild is uniformly awful and it's better to let someone else worry about that).

brassic 10y ago

I had to scrape the comments out of a blog. As you might expect, there was a div for comments, with comments nested within it. This made it easy to grab the comments using an HTML parser with xpath support. Unfortunately, the blog software had a bug and it was possible for markup to leak out of the comments. Sometimes a spurious </div> would close the comments div and the scraper would miss comments that came after it. However, the HTML did contain helpful HTML comments, something like "comments start here" and "comments end here". The reliable solution was to use a literal string search on these HTML comments to pull out the entire comments section, and then to use regexes to pull out the comments' content. The only unbreakable rule is that there are no unbreakable rules.