6 ms·

It really wasn't how compilers were constructed in '96 either. I was very into compilers in the late 80's and even then we didn't handwrite lexers or parsers.

by hasbot 7y ago

It really wasn't how compilers were constructed in '96 either. I was very into compilers in the late 80's and even then we didn't handwrite lexers or parsers. I was using bison and flex around '89.

onceUponADime 7y ago
Still usefull little tools to parse in configuration dsls etc.
- chrisseaton 7y ago
  Yes that's true - configuration files, network protocols, etc.
patrec 7y ago
Basically no serious programming language uses bison or flex, and everything that does (awk is YACC based) has terrible error messages. People totally do handwrite lexers and parsers, if you use some language in earnest the chances it has a handwritten parser or lexer or both are far greater than that it has been developed with bison/flex.
- hasbot 7y ago
  Hmm, I wrote a VHDL (a superset of ADA used for hardware design) front end using bison and flex. That's a serious language. Regarding error messages, if one implements the grammar as specified by the language spec, yes, confusing and useless error messages will result. That's because the language spec grammar includes restrictions based on semantics. To get good error messages, one has to remove/relax the spec grammar and then use semantic checking. Parsing is an art; not a science.
  - chrisseaton 7y ago
    Right - no 'serious language' is hyperbole, and another counter example is Ruby. But... if I was asked to implement a language tomorrow I wouldn't even consider using Bison and Flex. Why? Because they're context-free tools, and almost every industrial language is context-sensitive. Square pegs and round holes. These tools are a great example of over-complicating and over-theorising programming languages. We invented a theory that doesn't even apply to the problem we have, and then created crazy tools to solve the wrong problem, and gloss over the fact that they don't apply with terrible hacks and side-effects. We made a mistake with that phase in programming language history...
    - judofyr 7y ago
      Interesting. I've always thought that there's no real reason why e.g. Ruby has to be context-sensitive. Yes, I am aware of the fact that local variables are resolved statically, but do they have to be resolved during parsing? Couldn't they just be parsed as a method_call_or_var_lookup, and then later resolved? The main "feature" that Ruby's context-sensitive parser has is the ability to parse differently depending on what precedes it is a local variable or not, and I'm not sure if that's a good feature or not. Are there other parts of Ruby's grammars that are also context-sensitive? I'd say that Bison's biggest issue is that it's limited to LALR; if it actually supported full context-free grammars it would be far more useful.
      - chrisseaton 7y ago
        > Yes, I am aware of the fact that local variables are resolved statically That's the one that comes to my mind yes. I'd bet that there are others, but couldn't list them to you now. > do they have to be resolved during parsing? I guess not in practice? But why are we doing this to ourselves, right? These parsing tools work great! (As long as you only do half do the job you were trying to do and then sort of hack it back together afterward.)
        judofyr 7y ago
        > But why are we doing this to ourselves, right? These parsing tools work great! (As long as you only do half do the job you were trying to do and then sort of hack it back together afterward.) I agree with you in practice: The current parser generators are not as good as I'd like them to be. They are too restrictive and too locked into a "let's parse this subset of grammars mathematically optimally" instead of "let's make it easy for programmers to declare grammars". (Disclaimer: I started working on a completely new parser compiler https://www.sanity.io/blog/why-we-wrote-yet-another-parser-compiler https://www.sanity.io/blog/why-we-wrote-yet-another-parser-c... which is capable of parsing full context-free grammars. I personally believe this is part of the puzzle of making them more usable.) However, since we're talking about Ruby's syntax: I don't think Ruby's parser would be any nicer if it was handwritten. The problem here is that they discovered the power of "lexer tweaks parser state" and let it drive the design of the language. This has caused the language itself to be complicated (both for humans and for computers!) Unfortunately I don't have time for it, but I'd love to dive into the changes required in Ruby's syntax to make it context-free (e.g. pushing the local variable lookup into the bytecode generator).
  - MaxBarraclough 7y ago
    I'm reminded of another comment from some time ago [0] that recommended 'permissive' parsers for a different reason: you can build a list of syntax errors for the user, which isn't possible if you explode immediately, and you may be able to generate better syntax errors than your parser would have. [0] https://news.ycombinator.com/item?id=17072670 https://news.ycombinator.com/item?id=17072670
- judofyr 7y ago
  > Basically no serious programming language uses bison or flex, https://en.wikipedia.org/wiki/GNU_Bison#Use https://en.wikipedia.org/wiki/GNU_Bison#Use: Ruby, PHP, GCC until 2004, Go, Bash, PostgreSQL, MySQL, Perl. Can we stop with this "duh, real programmers write their own handwritten parsers"? Parser generators are awesome! They force you to write a declarative grammar which a human can understand. If you're designing a language this is very useful. With a handwritten parser you're very likely to encode some unintended behaviour in edge cases; using a parser generator early in this process will very quickly reveal the ambiguities. That doesn't mean that parser generators are always the best. There are many use cases where a handwritten parser is better. But the picture is way more nuanced than "no serious programming language uses bison or flex".
  - chrisseaton 7y ago
    > They force you to write a declarative grammar which a human can understand. The problem is many languages' grammars are not declarative. They're full of completely uncontrolled side effects. So why are we trying to use a declarative tool for them? It's the wrong tool for the job.
    - hasbot 7y ago
      How is that relevant to this thread in which I am saying that the parsing/lexing as described in the link is very outdated? Maybe you should start a separate discussion.
      - chrisseaton 7y ago
        > Maybe you should start a separate discussion. I started this one!
    - judofyr 7y ago
      > It's the wrong tool for the job. Yes! We should have both handwritten parsers and parser generators in our toolkit and then depending on the language we want to parse we should choose the best tool for the job. My point about a declarative grammar was more that if the language permits it (e.g. is simple enough for Bison), it's so valuable to have a declarative grammar which serves as both specification and parser. It will be way less bug-prone than any handwritten parser. (From my experience handwritten parsers tends to have both weird crashes and unexpected handling of edge cases. But maybe I just suck at programming?) I've seen people using Bison on one complicated languages, given up, and then they bring the negative bias to other languages they want to parse — even when the other languages are very simple.
seanmcdirmid 7y ago
Many mass-distributed compilers have hand-crafted lexers and parsers since (a) they aren’t hard to write and (b) hand crafting gives you complete control over error recovery.

by hasbot 7y ago

It really wasn't how compilers were constructed in '96 either. I was very into compilers in the late 80's and even then we didn't handwrite lexers or parsers. I was using bison and flex around '89.

onceUponADime 7y ago

Still usefull little tools to parse in configuration dsls etc.

chrisseaton 7y ago
Yes that's true - configuration files, network protocols, etc.

patrec 7y ago

Basically no serious programming language uses bison or flex, and everything that does (awk is YACC based) has terrible error messages. People totally do handwrite lexers and parsers, if you use some language in earnest the chances it has a handwritten parser or lexer or both are far greater than that it has been developed with bison/flex.

hasbot 7y ago
Hmm, I wrote a VHDL (a superset of ADA used for hardware design) front end using bison and flex. That's a serious language. Regarding error messages, if one implements the grammar as specified by the language spec, yes, confusing and useless error messages will result. That's because the language spec grammar includes restrictions based on semantics. To get good error messages, one has to remove/relax the spec grammar and then use semantic checking. Parsing is an art; not a science.
- chrisseaton 7y ago
  Right - no 'serious language' is hyperbole, and another counter example is Ruby. But... if I was asked to implement a language tomorrow I wouldn't even consider using Bison and Flex. Why? Because they're context-free tools, and almost every industrial language is context-sensitive. Square pegs and round holes. These tools are a great example of over-complicating and over-theorising programming languages. We invented a theory that doesn't even apply to the problem we have, and then created crazy tools to solve the wrong problem, and gloss over the fact that they don't apply with terrible hacks and side-effects. We made a mistake with that phase in programming language history...
  - judofyr 7y ago
    Interesting. I've always thought that there's no real reason why e.g. Ruby has to be context-sensitive. Yes, I am aware of the fact that local variables are resolved statically, but do they have to be resolved during parsing? Couldn't they just be parsed as a method_call_or_var_lookup, and then later resolved? The main "feature" that Ruby's context-sensitive parser has is the ability to parse differently depending on what precedes it is a local variable or not, and I'm not sure if that's a good feature or not. Are there other parts of Ruby's grammars that are also context-sensitive? I'd say that Bison's biggest issue is that it's limited to LALR; if it actually supported full context-free grammars it would be far more useful.
    - chrisseaton 7y ago
      > Yes, I am aware of the fact that local variables are resolved statically That's the one that comes to my mind yes. I'd bet that there are others, but couldn't list them to you now. > do they have to be resolved during parsing? I guess not in practice? But why are we doing this to ourselves, right? These parsing tools work great! (As long as you only do half do the job you were trying to do and then sort of hack it back together afterward.)
      - judofyr 7y ago
        > But why are we doing this to ourselves, right? These parsing tools work great! (As long as you only do half do the job you were trying to do and then sort of hack it back together afterward.) I agree with you in practice: The current parser generators are not as good as I'd like them to be. They are too restrictive and too locked into a "let's parse this subset of grammars mathematically optimally" instead of "let's make it easy for programmers to declare grammars". (Disclaimer: I started working on a completely new parser compiler https://www.sanity.io/blog/why-we-wrote-yet-another-parser-compiler https://www.sanity.io/blog/why-we-wrote-yet-another-parser-c... which is capable of parsing full context-free grammars. I personally believe this is part of the puzzle of making them more usable.) However, since we're talking about Ruby's syntax: I don't think Ruby's parser would be any nicer if it was handwritten. The problem here is that they discovered the power of "lexer tweaks parser state" and let it drive the design of the language. This has caused the language itself to be complicated (both for humans and for computers!) Unfortunately I don't have time for it, but I'd love to dive into the changes required in Ruby's syntax to make it context-free (e.g. pushing the local variable lookup into the bytecode generator).
- MaxBarraclough 7y ago
  I'm reminded of another comment from some time ago [0] that recommended 'permissive' parsers for a different reason: you can build a list of syntax errors for the user, which isn't possible if you explode immediately, and you may be able to generate better syntax errors than your parser would have. [0] https://news.ycombinator.com/item?id=17072670 https://news.ycombinator.com/item?id=17072670
judofyr 7y ago
> Basically no serious programming language uses bison or flex, https://en.wikipedia.org/wiki/GNU_Bison#Use https://en.wikipedia.org/wiki/GNU_Bison#Use: Ruby, PHP, GCC until 2004, Go, Bash, PostgreSQL, MySQL, Perl. Can we stop with this "duh, real programmers write their own handwritten parsers"? Parser generators are awesome! They force you to write a declarative grammar which a human can understand. If you're designing a language this is very useful. With a handwritten parser you're very likely to encode some unintended behaviour in edge cases; using a parser generator early in this process will very quickly reveal the ambiguities. That doesn't mean that parser generators are always the best. There are many use cases where a handwritten parser is better. But the picture is way more nuanced than "no serious programming language uses bison or flex".
- chrisseaton 7y ago
  > They force you to write a declarative grammar which a human can understand. The problem is many languages' grammars are not declarative. They're full of completely uncontrolled side effects. So why are we trying to use a declarative tool for them? It's the wrong tool for the job.
  - hasbot 7y ago
    How is that relevant to this thread in which I am saying that the parsing/lexing as described in the link is very outdated? Maybe you should start a separate discussion.
    - chrisseaton 7y ago
      > Maybe you should start a separate discussion. I started this one!
  - judofyr 7y ago
    > It's the wrong tool for the job. Yes! We should have both handwritten parsers and parser generators in our toolkit and then depending on the language we want to parse we should choose the best tool for the job. My point about a declarative grammar was more that if the language permits it (e.g. is simple enough for Bison), it's so valuable to have a declarative grammar which serves as both specification and parser. It will be way less bug-prone than any handwritten parser. (From my experience handwritten parsers tends to have both weird crashes and unexpected handling of edge cases. But maybe I just suck at programming?) I've seen people using Bison on one complicated languages, given up, and then they bring the negative bias to other languages they want to parse — even when the other languages are very simple.

seanmcdirmid 7y ago

Many mass-distributed compilers have hand-crafted lexers and parsers since (a) they aren’t hard to write and (b) hand crafting gives you complete control over error recovery.