4 ms·
Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. Th
by maxbrunsfeld 8y ago
Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. There's also a `token()` function that you can use to specify that an arbitrary rule should be handled by the lexer as a single token.
In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.
- mncharity 8y ago> grammar author does fully control the division between the parser and lexer Nifty. So explicit GLR forks at rule level (`conflicts:`), non-forking token "conflict" resolution[1], no token regex backtracking pressure across tokens, and token resolution at a single code position can(?) differ across conflicting rules? I'm uncertain on that last bit. > In most cases, you don't have to think about it Well yes, but, some of us crave much more syntactically flexible languages. :) [1] https://tree-sitter.github.io/tree-sitter/creating-parsers#conflicting-tokens https://tree-sitter.github.io/tree-sitter/creating-parsers#c...