3 ms·

Wrote a C-subset compiler for a compilers course back in university. In general it's useful for a compiler to not just hit the first error it can find in the s

Wrote a C-subset compiler for a compilers course back in university. In general it's useful for a compiler to not just hit the first error it can find in the source code and immediately error out. Instead, if you keep parsing after encountering an error, you can often encounter more errors, so that you can give the user a list of errors they need to fix, not just the first one. In that sense, a compiler's job isn't over the instant it encounters a syntax error, so the extra tokenization would not be useless.

alcover 4y ago
I'm struggling with this idea. You could end up tokenizing the inside of an unopened string literal for (contrived) ex : String s = hello world"; [-> typename:'String' id's' op'=' id'hello' id'world' ...] And it would cascade if then foo("blah"); (...) [-> strlit';\nfoo(' id'blah' ...] A parser-first approach will stop with unknown id 'hello' and avoid the cascade.
- swatcoder 4y ago
  In a mature product, you would apply heuristics to help you provide the most useful feedback to the user while optimizing the time it takes to do so. Users love consolidated, efficient feedback without having to peel back errors one at a time. It’s a distinguishing feature when you can find a way to do it that suits your input, and specifically so because it can be a hard problem to solve well!
  - 1letterunixname 4y ago
    Precise diagnostics usually follow one of 2 approaches: 1. Hand-written, recursive descent 2. Error matching expanded grammar beyond minimal gramar
- 1letterunixname 4y ago
  Strings typically have a token state context. Antlr allows pushing and popping a context state. Not a CFG and won't work with simple tools or book approaches.
- PretzelPirate 4y ago
  Check out this section of Crafting Interpreters on Error Recovery: https://craftinginterpreters.com/parsing-expressions.html#panic-mode-error-recovery https://craftinginterpreters.com/parsing-expressions.html#pa...

alcover 4y ago

I'm struggling with this idea. You could end up tokenizing the inside of an unopened string literal for (contrived) ex : String s = hello world"; [-> typename:'String' id's' op'=' id'hello' id'world' ...] And it would cascade if then foo("blah"); (...) [-> strlit';\nfoo(' id'blah' ...] A parser-first approach will stop with unknown id 'hello' and avoid the cascade.

swatcoder 4y ago
In a mature product, you would apply heuristics to help you provide the most useful feedback to the user while optimizing the time it takes to do so. Users love consolidated, efficient feedback without having to peel back errors one at a time. It’s a distinguishing feature when you can find a way to do it that suits your input, and specifically so because it can be a hard problem to solve well!
- 1letterunixname 4y ago
  Precise diagnostics usually follow one of 2 approaches: 1. Hand-written, recursive descent 2. Error matching expanded grammar beyond minimal gramar
1letterunixname 4y ago
Strings typically have a token state context. Antlr allows pushing and popping a context state. Not a CFG and won't work with simple tools or book approaches.
PretzelPirate 4y ago
Check out this section of Crafting Interpreters on Error Recovery: https://craftinginterpreters.com/parsing-expressions.html#panic-mode-error-recovery https://craftinginterpreters.com/parsing-expressions.html#pa...