3 ms·

So, one of the key things that I see is that the constructed abstract syntax tree must have the raw tokens within it so all other layers have full awareness. I

So, one of the key things that I see is that the constructed abstract syntax tree must have the raw tokens within it so all other layers have full awareness. I feel the criticism is valid because most parsers in production are done via hand without tooling or fancy techniques.

Quekid5 6y ago
> most parsers in production are done via hand without tooling or fancy techniques. What? Do you have any data you'd like to share? I'm given to understand that e.g. the C++ compilers usually have a hand-coded, but AFAIUI that's mostly due to the complexity of actually parsing it (and fitting that into anything other than just raw code).
- mathgladiator 6y ago
  only anecdotal and observations over the years https://www.drdobbs.com/architecture-and-design/so-you-want-to-write-your-own-language/240165488 https://www.drdobbs.com/architecture-and-design/so-you-want-... A common theme is that the parser generator does not provide you the tools to write the high quality error messages. Having used ANTLR and other tools, I believe it now that I'm trying to ship a real language.
  - Quekid5 6y ago
    You're talking about parser generators... the article is about parser combinators. Fully agreed that parser generators are often very limited and often require hacks to do anything non-trivial. (IME MegaParsec basically solves most of the issues.)
wwright 6y ago
Python 3.9 is actually switching to a PEG :)
ookdatnog 6y ago
> So, one of the key things that I see is that the constructed abstract syntax tree must have the raw tokens within it so all other layers have full awareness. I don't see why you say that, parser combinators are just a structured way to compose recursive descent parsers, you can do everything with them that you can with any other hand-rolled parser. You don't even need to construct an AST at all, a mathematical expression "parser combinator" could just return a single number, the result of evaluating the expression.

Quekid5 6y ago

> most parsers in production are done via hand without tooling or fancy techniques. What? Do you have any data you'd like to share? I'm given to understand that e.g. the C++ compilers usually have a hand-coded, but AFAIUI that's mostly due to the complexity of actually parsing it (and fitting that into anything other than just raw code).

mathgladiator 6y ago
only anecdotal and observations over the years https://www.drdobbs.com/architecture-and-design/so-you-want-to-write-your-own-language/240165488 https://www.drdobbs.com/architecture-and-design/so-you-want-... A common theme is that the parser generator does not provide you the tools to write the high quality error messages. Having used ANTLR and other tools, I believe it now that I'm trying to ship a real language.
- Quekid5 6y ago
  You're talking about parser generators... the article is about parser combinators. Fully agreed that parser generators are often very limited and often require hacks to do anything non-trivial. (IME MegaParsec basically solves most of the issues.)

wwright 6y ago

Python 3.9 is actually switching to a PEG :)

ookdatnog 6y ago

> So, one of the key things that I see is that the constructed abstract syntax tree must have the raw tokens within it so all other layers have full awareness. I don't see why you say that, parser combinators are just a structured way to compose recursive descent parsers, you can do everything with them that you can with any other hand-rolled parser. You don't even need to construct an AST at all, a mathematical expression "parser combinator" could just return a single number, the result of evaluating the expression.