4 ms·
Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state mach
by barrkel 8y ago
Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching.
Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to know exactly where you are in the regular language that defines the tokens to write good code. Writing a regex matcher in code like this is like writing code in assembly - mentally, you're mapping to a different set of concepts all the time.
- jstimpfle 8y agoYes. I don't think anyone is disagreeing here. If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how would you parse string literals (including escape sequences) with a regex?
- mcguire 8y ago"(.|\.)*"
- jstimpfle 8y ago"Evaluation:\tWrong!\x07\x07\x07\r\n" You want to convert the string literal to an internal buffer, interpreting the escape sequences. In the same way, you want to parse integers. You cannot really do that with a regular expression. RE is for matching, not for transforming.
- jcranberry 8y agoString literals are easy without escaped quotes. With escaped quotes its annoying and non regex is much cleaner.