3 ms·
Excellent summary. A couple other reasons for a separate tokenizer: 1. Sometimes all you need is a tokenizer - such as for highlighting in a code editor 2. D
by WalterBright 4y ago
Excellent summary. A couple other reasons for a separate tokenizer:
1. Sometimes all you need is a tokenizer - such as for highlighting in a code editor
2. D has a construct called a token string - where a string literal consists of tokens
3. A separate tokenizer means the lexer and parser can run in separate threads