3 ms·

Actually do you mind expanding on the quality of yacc, bison being awful. I dont mean that as passive "oh yeah?" line to "trap" you. Language front-ends (an

Actually do you mind expanding on the quality of yacc, bison being awful. I dont mean that as passive "oh yeah?" line to "trap" you. Language front-ends (and their tooling) is something I am pretty passionate about and having developed parser generators in the past (and working on something now) I just wanted to explore specifics that make yacc, bison unweildy for others. Just to start off - I still find their debugging or their (lack of) ability to reason a huge barrier to wannabe language developers. But I am still unable to put this in concrete terms. My last few months has been immersed in reading a ton of papers and I still see the papers use bison as the staging ground for most new ideas (menhir these days seem to be gaining popularity) so curious what is keeping yacc/bison in their throne position?

PaulHoule 5y ago
For one thing the callback-based interface in yacc is unergonomic. In a language like C# or Python you usually want to get an object tree for the AST and you shouldn't have to write any code to get that. It should come "for free" with the grammar. Another missing capability that people don't know they are missing is the ability to "unparse" the AST back to text. This is valuable when you want to write a code transformer: for instance in query languages like SQL there is a lot of value in doing a transformation on the query and then writing it back to SQL. You should be able to transform the AST and not have to write a line of code to transform it. Another problem is composability of grammars. If you want to support SQLLite, Microsoft SQL Server and Oracle SQL dialects you should have a base SQL grammar and extended grammars that inherit from the base. All of these capabilities are so beyond the pale of what parser generator "insiders" are used to that it's a very hard conversation to communicate that these capabilities are both possible and useful.
- metadata 5y ago
  Your thinking very closely reflects mine. I actually did build a C# parser generator producing strongly typed AST from base ANSI grammar and inherited Oracle, SQL Server, MySQL, and Postgres grammars. We are using it in our database migration products and will soon use it for Resharper style intelligence in a very lightweight and fast database manager. It never occurred to me to publish the parser generator, though. It's our secret sauce.
- flashgordon 5y ago
  Nice. Starting from the easy(ish) one - composability is slowly becoming a solved problem these days or atleast something that can be tacked on via templating (not an ideal or native solution to these tools I agree). The AST availability is defintely a big one that I cannot align on a decent contract on. At some point in time not all AST transformations are reversible yeah? Heck even a conversion from Parse Tree to AST is often not bidirectional. A lot of AST directed IDEs (even MPS from Jetbrains) end up being very cumbersome when typing-at-scale. I wonder what kind of semantics can be foundational enough that AST generation is simplified for atleast 90% of the cases? Again my interest in this comes from building incremental editors for certain domains so I am definitely looking to learn more.

PaulHoule 5y ago

For one thing the callback-based interface in yacc is unergonomic. In a language like C# or Python you usually want to get an object tree for the AST and you shouldn't have to write any code to get that. It should come "for free" with the grammar. Another missing capability that people don't know they are missing is the ability to "unparse" the AST back to text. This is valuable when you want to write a code transformer: for instance in query languages like SQL there is a lot of value in doing a transformation on the query and then writing it back to SQL. You should be able to transform the AST and not have to write a line of code to transform it. Another problem is composability of grammars. If you want to support SQLLite, Microsoft SQL Server and Oracle SQL dialects you should have a base SQL grammar and extended grammars that inherit from the base. All of these capabilities are so beyond the pale of what parser generator "insiders" are used to that it's a very hard conversation to communicate that these capabilities are both possible and useful.

metadata 5y ago
Your thinking very closely reflects mine. I actually did build a C# parser generator producing strongly typed AST from base ANSI grammar and inherited Oracle, SQL Server, MySQL, and Postgres grammars. We are using it in our database migration products and will soon use it for Resharper style intelligence in a very lightweight and fast database manager. It never occurred to me to publish the parser generator, though. It's our secret sauce.
flashgordon 5y ago
Nice. Starting from the easy(ish) one - composability is slowly becoming a solved problem these days or atleast something that can be tacked on via templating (not an ideal or native solution to these tools I agree). The AST availability is defintely a big one that I cannot align on a decent contract on. At some point in time not all AST transformations are reversible yeah? Heck even a conversion from Parse Tree to AST is often not bidirectional. A lot of AST directed IDEs (even MPS from Jetbrains) end up being very cumbersome when typing-at-scale. I wonder what kind of semantics can be foundational enough that AST generation is simplified for atleast 90% of the cases? Again my interest in this comes from building incremental editors for certain domains so I am definitely looking to learn more.