Well, the practical answer that all languages I know of have separate lexical and grammatical structure. It's just easier to write if you separate them. Thoug

Well, the practical answer that all languages I know of have separate lexical and grammatical structure. It's just easier to write if you separate them. Though it's baroque, even bash has clear lexical structure. In my code, the lexer is the foundation and doing a lot of useful work. It's a map from the 13 modes to a list of regexes, which is a lot of logic that is kept out of the parser. https://github.com/oilshell/oil/blob/master/osh/lex.py https://github.com/oilshell/oil/blob/master/osh/lex.py The parser just calls lexer.Read(lex_mode), which hides the complexity of all these regexes. (The lexer might look weird, but I wrote it in a style to port to C, and in particular using re2c. This is an awesome tool which I've mentioned in my blog a couple times, but not written about yet. It combines regular expressions and turns them into a bunch of gotos in C. http://www.oilshell.org/cross-ref.html?tag=re2c#re2c http://www.oilshell.org/cross-ref.html?tag=re2c#re2c ) ----------- For a more theoretical answer: I've had this discussion before, and I would claim that lexing and parsing should be separate because they differ in both computational power and running time. Lexing is fast but not that powerful -- you are structuring the input stream with (at most) regular expressions. It can be done in linear time, without backtracking. On the other hand, parsing is slow but more powerful, e.g. CFGs or PEGs can recognize more than regular expressions, but they require backtracking/memoization/lookahead/etc. In other words, why not do the easy work with the fast algorithm, and then do the harder work with the more complicated, expensive algorithm -- on a shorter input? I came to the conclusion several years ago that PEGs only combined lexing and parsing for reasons of academic presentation and proof. I don't think it actually helps you in practice. Hope that answered the question! I think my shell lexer is somewhat interesting so I plan to blog about it. Honestly re2c was almost the thing that inspired me to write the shell -- it would have been intractable otherwise.