4 ms·
The concept of a "token tree" as a separate stage between ordinary tokenization and full parsing seems interesting on its own: was it based on another system or
by panic 9y ago
The concept of a "token tree" as a separate stage between ordinary tokenization and full parsing seems interesting on its own: was it based on another system or invented for Rust? For example, you could imagine implementing incremental parsing by writing your parser as a transformation on a token tree.
- nine_k 9y agoIsn't it how Lisp macros work?
- db48x 9y agoYep. Although in some lisps you can write reader macros that control tokenization to add new syntax.
- panic 9y agoYep! I have also been pointed at this PhD thesis: https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf which defines a macro system on a uniform Algol-like syntax with intermediate "reading" and "enforestation" phases and corresponding syntax representations (the "read" representation being pretty much the same as the token tree representation here).