3 ms·

Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST (and sometimes even do analyses) while y

Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST (and sometimes even do analyses) while you're reading the input than to allocate and store a giant list of tokens and then parse that. Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document, usually starting from the next valid statement or declaration. This lets them report more syntax errors and moreover, is very important for good IDE support (if you've ever used an IDE/language combo where you only get completion after you've written the code, you know why). Although if the syntax error messes up tokenization (e.g. missing quote or end comment) it usually screws up the rest of the document anyways.

eatonphil 4y ago
> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST And some don't even generate an AST. :) Just read in and emit or interpret. https://briancallahan.net/blog/20210814.html https://briancallahan.net/blog/20210814.html
alcover 4y ago
> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory Rather legit reasons.. The one I'm writing does this. It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering. > Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document This seems ardous for you'd maybe have to keep parallel explanations of what you read ?
- pavlov 4y ago
  "> It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering." But if you're reading a book and can't understand a sentence, you'll probably glance at the rest of the page to see if there are clues in the context. Maybe it's a printing error and a paragraph has been repeated. A human who sees the entire page will detect that problem immediately and simply skip over the repeated paragraph. A computer that gives up at the first sight of incongruency has no idea that recovery was so easy. The analogy here might be relevant to error messages. A compiler that "sees the whole page" can potentially offer more useful suggestions to the programmer about how to fix a problem.

eatonphil 4y ago

> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST And some don't even generate an AST. :) Just read in and emit or interpret. https://briancallahan.net/blog/20210814.html https://briancallahan.net/blog/20210814.html

alcover 4y ago

> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory Rather legit reasons.. The one I'm writing does this. It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering. > Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document This seems ardous for you'd maybe have to keep parallel explanations of what you read ?

pavlov 4y ago
"> It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering." But if you're reading a book and can't understand a sentence, you'll probably glance at the rest of the page to see if there are clues in the context. Maybe it's a printing error and a paragraph has been repeated. A human who sees the entire page will detect that problem immediately and simply skip over the repeated paragraph. A computer that gives up at the first sight of incongruency has no idea that recovery was so easy. The analogy here might be relevant to error messages. A compiler that "sees the whole page" can potentially offer more useful suggestions to the programmer about how to fix a problem.