3 ms·
Yes, but that's a parsing-time distinction. The lexer does not need to care. And even though the token's syntactic meaning "depends on the context", the grammar
by Sharlin 7y ago
Yes, but that's a parsing-time distinction. The lexer does not need to care. And even though the token's syntactic meaning "depends on the context", the grammar itself can be (and usually is) perfectly context-free!
- Doxin 7y agoThat depends a lot on your parser. Personally if I write a parser I don't generally bother with having a separate tokenizing/lexing stage. Tokenizing happens as-needed and inline.