4 ms·

When you get into a situation where the approach you take causes people to start playing whack-a-mole to patch issues that arise from said approach, it's a sign

When you get into a situation where the approach you take causes people to start playing whack-a-mole to patch issues that arise from said approach, it's a sign that it may not be a good approach after all. Your IDE will do type checking as well. Yes, the lexer stage will parse all identifiers into 'ID', but to provide a contrived counter-example, what if I do want to allow 'pi' to be assigned based on something that can only be processed in a later stage (perhaps an annotation, or a result of compile time function evaluation, etc). It will push the logic of handling 'ID' to later stages anyway. Now you're handling IDs in at least 2 places - that's strictly worse than the status quo. Part of our jobs is to be able to reason about trade-offs when making decisions. Pointing out that the lexer has a limitation does not necessarily justify removing that limitation for an edge case because it might be due to that limitation that it can be kept simple for the general case. Adding complexity to the lexer without simplifying the later stages is a net negative in my opinion, which would make this trade-off not worth it. Is it an interesting approach? Sure. Is it worth it? Maybe in some very specific use cases, but it doesn't seem like a convincing argument to me to combine the lexing and parsing phases.

titzer 3y ago
> Your IDE will do type checking as well. Yes, the lexer stage will parse all identifiers into 'ID', but to provide a contrived counter-example, what if I do want to allow 'pi' to be assigned based on something that can only be processed in a later stage (perhaps an annotation, or a result of compile time function evaluation, etc). It will push the logic of handling 'ID' to later stages anyway. Now you're handling IDs in at least 2 places - that's strictly worse than the status quo. Typechecking and access control are clearly semantic analysis considerations, which are better handled in a separate pass on an explicit AST. In particular, they may require a symbol table (i.e. an environment of names) that is not possible to compute during the parsing phase.

titzer 3y ago

> Your IDE will do type checking as well. Yes, the lexer stage will parse all identifiers into 'ID', but to provide a contrived counter-example, what if I do want to allow 'pi' to be assigned based on something that can only be processed in a later stage (perhaps an annotation, or a result of compile time function evaluation, etc). It will push the logic of handling 'ID' to later stages anyway. Now you're handling IDs in at least 2 places - that's strictly worse than the status quo. Typechecking and access control are clearly semantic analysis considerations, which are better handled in a separate pass on an explicit AST. In particular, they may require a symbol table (i.e. an environment of names) that is not possible to compute during the parsing phase.