2 ms·
IMHO one of the benefits to having a separate tokenization step is in error handling. It can be easier to generate good error messages if you've already separat
by jasonhansel 5y ago
IMHO one of the benefits to having a separate tokenization step is in error handling. It can be easier to generate good error messages if you've already separated the input into tokens (e.g. "unexpected identifier" instead of "unexpected letter 'i'.")
There can also be performance advantages, at least in some cases.
- brundolf 5y agoNormally my error messages take the form "expected identifier" or "expected {" or whatever (shown with the line and column numbers). I feel like that's more informative anyway