3 ms·
One of the most interesting ways to handle regex and general parsing IMO is by using derivitives. I'm not sure if this uses any of the ideas from that area, but
by slaymaker1907 8y ago
One of the most interesting ways to handle regex and general parsing IMO is by using derivitives. I'm not sure if this uses any of the ideas from that area, but with derivitive based regex parsing instead of constructing some specific program to parse some language, you parse by instead transforming the original language to be the language of the union of the old language as well as the character just seen.
In addition to being incredibly simple to implement compared to traditional regex and parser generators, I think it is also interesting in that while it is really simple and looks like you are interpreting a language, it is actually powerful enough to parse arbitrary CFGs in O(n^3) with some help from memoization and fixpoints. See http://matt.might.net/articles/parsing-with-derivatives/ http://matt.might.net/articles/parsing-with-derivatives/ for more details.
I think the interesting aspect in terms of performance with something like this is that the derivitive based parsing system is explicitly streaming since it only stores the remaining input string and a regex string. Also, I think you can get around some of the performance issues with constructing these strings by replacing certain parts of the regex with more traditional NFAs/DFAs. In particular, I think this method provides an easy way to use simple equality matching for constant strings, DFAs for Kleene star, NFAs for the rest of the typical NFA features like the or operator and negation, then finally using the derivitive for very complicated features like capturing groups and backreferences.
- fmap 8y agoDerivatives are simple to explain, but not entirely straightforward to implement efficiently. I personally prefer the Glushkov construction. The Glushkov construction (which Hyperscan uses among other things) is a similarly straightforward translation from a regular expression to an epsilon-free NFA with nice properties. There's a very neat paper explaining the construction in the form of a play: https://sebfisch.github.io/haskell-regexp/regexp-play.pdf https://sebfisch.github.io/haskell-regexp/regexp-play.pdf On the CFG side, I like to think of Early parsing as analogous to the Glushkov construction. At the very least they have similar properties.
- glangdale 8y agoI'm aware of derivatives. My concern with derivatives (which I think are awesome) is that the combinatorial blow-up that we associate with DFAs is always around the corner. I've never been a huge fan of putting all my eggs in the the 'I hope it won't explode' basket. It's been a major point of difference with RE2 all this time - IMO you need a high performance pass for things that don't determinize cleanly (even when you are doing lazy determinization). I would need to see a lot more detail on using derivatives for capturing groups and back-references. Sorry if I sound closed-minded, but derivatives look like they solve a lot of stuff I already know how to do really well.