6 ms·

ANTLR 4.6 Released

123 points by carbocation 10y ago

carbocation 10y ago
(Mostly notable for my purposes because it includes golang as a code generation target.)
CalChris 10y ago
Nice to see C++ and Go in 4.6 mainline. I've been stuck on ANTLR 3 in order to use C++ and it's great to see the new target finally mainlined. Go is good and I'd like to see a Rust target sometime soon.
- avecfromage 10y ago
  Same here. Antlr is great, finally the wait for reintroduction of C++ support is over. Never nice to see features removed in newer releases of software you want to use.
- glangdale 10y ago
  Agreed! I hope full (not half-hearted and late) C++ support becomes the 'new normal' for ANTLR.
- nonsince 10y ago
  At the risk of sounding inflammatory, what would be the point of using a parser generator over a powerful applicative-style combinator library like https://github.com/Marwes/combine https://github.com/Marwes/combine ? I've used both in the past and personally massively prefer parser combinators, but I'd be interested to know what the benefits are of parser generators.
  - z681781 10y ago
    I cannot answer that directly, since I've never used parser combinators. The general problem is that there's so much software out there to learn that most people are happy when they know yacc-style and antlr parser generators. Also, most people are regularly disappointed by software that over-promises, which costs them a lot of time. So they stick to what works for them. In my case, I find Bison/Menhir such great tools that I have zero inclination to even look at other things. EDIT: A corollary to this is that people who are very productive (Thompson, Torvalds, Bellard,...) don't constantly change their software stacks.
  - Matthias247 10y ago
    I think one advantage of things like ANTLR is that the grammer is described in an abstract fashion and that you can then generate parsers in multiple target languages with it. With a parser combinator library you are only targeting a single language. I also think that for project outsiders and beginners it's easier to look at and understand a grammar instead of a parser in combinator style. I personally have worked with parser generators (Irony for .NET) as well as parser combinator (fparsec) and enjoyed both a lot.
rombix 10y ago
Nice to see Swift added to the list of targets. Any idea about the performance of generated parsers in Swift vs other targets?
chubot 10y ago
I'd be interested see pointers to ANTLR-based languages people are developing. I found it to be a powerful tool, but also limiting. As far as I know, ANTLR v4 is even more of a "framework" than v3, in that it dictates your program structure and always outputs a full parse tree. It appears to have lost the ability to verify the value of k for LL(k) grammars, due to a more powerful underlying algorithm. I think that simplifies it for most use cases but makes others hard or impossible.
- chrisseaton 10y ago
  I work on implementing languages and I often need new parsers. Either for full languages or little embedded languages or expression syntaxes of various kinds. I often think 'right, this time I'm going to write a really nice little Antlr grammar and do it properly'. But every single time I regret it because there's always some complexity that means things don't fit into the Antlr model, and making things work the way Antlr works means more complexity than just writing a simple lexer and parser by hand. My most recent example is that Ruby strings can't always be safely converted into Java strings, but Antlr wants to parse Java strings only and not a byte[]. So I had to copy my Ruby byte[] into a Java string a character at a time and just use the character as numbers pretending that they had no encoding. That's just one example. It's always some other new problem that means it's more ceremony to use Antlr than the tool saves me later one. Despite lexing and parsing being an entire sub-field of computer science, to be honest I'm not sure that in practice lexing and parsing are really problems that are so complex in the first place that a tool is ever needed.
  - setq 10y ago
    Agree here. I recently implemented a small query language DSL uaing Antlr and I regret not writing my own lexer and parser. It's difficult to bend a framework like Antlr into a useable state if you hit an edge case or require unusual look ahead semantics. Next time, I'll start with a simple recursive descent parser.
  - verdex_blarg 10y ago
    This is exactly the sort of thing I was looking for. I keep on thinking that my life would be easier if I just adopted something like ANTLR, but every time I look into it it looks like it's more work than it's worth. Good to hear that this has been a similar experience to people who have actually gone and actually used ANTLR.
clumsysmurf 10y ago
Just curious if anyone has used ANTLR on Android lately; I recall it was easy to run out of stack because the runtime relies on very deep recursion. Also I think there were some dependencies on AWT or Swing for debug console. https://github.com/antlr/antlr4/issues/744 https://github.com/antlr/antlr4/issues/744
butner 10y ago
Ya ANTLR (+TParr).
manishsharan 10y ago
Compiler design was one subject that I struggled with when I was an undergrad. I feel this has held me back from creating DSLs. I am looking for a gentle introduction to ANTLR that can take me from newbie to mastery. Any books anyone can recommend would be helpful.
- fusiongyro 10y ago
  The book "Language Implementation Patterns"[1] covers ANTLR 3, and the author (who is one of the implementors of ANTLR) walks you through successively more interesting DSLs until you're basically building a programming language, but does it in small steps. I didn't struggle with compilers as an undegrad at all, and I found this book too slow for me, so I'm optimistic that you will be pleased by the pacing. :) If you want to see a way older approach, I did really enjoy Jack Crenshaw's "Let's Build a Compiler"[2]—but it's so vintage it may be totally hipster by now. :) [1] https://www.amazon.com/Language-Implementation-Patterns-Domain-Specific-Programming/dp/193435645X/ref=sr_1_2?ie=UTF8&qid=1481865280&sr=8-2&keywords=antlr https://www.amazon.com/Language-Implementation-Patterns-Doma... [2] http://compilers.iecc.com/crenshaw/ http://compilers.iecc.com/crenshaw/
- breck 10y ago
  The Antlr-4 book is terrific: https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-re...
  - kriro 10y ago
    Indeed. As someone with no real language background it allowed me to get to a state where I can create my own DSLs. An alternative would be to use something like Xtext (which also allows you to create langauge tools, not just the language itself). I played around with it and it's very nice (granted you'll kind of need to embrace Eclipse) but ultimately decided that ANTLR was "enough".
- heysunshine123 10y ago
  Look into "Pratt Parsers". It's reasonably easy to implement one based on some blog posts etc.
NicolaiS 10y ago
More detailed changelog: https://github.com/antlr/antlr4/releases/tag/4.6 https://github.com/antlr/antlr4/releases/tag/4.6

123 points by carbocation 10y ago

(Mostly notable for my purposes because it includes golang as a code generation target.)

Nice to see C++ and Go in 4.6 mainline. I've been stuck on ANTLR 3 in order to use C++ and it's great to see the new target finally mainlined. Go is good and I'd like to see a Rust target sometime soon.

avecfromage 10y ago
Same here. Antlr is great, finally the wait for reintroduction of C++ support is over. Never nice to see features removed in newer releases of software you want to use.
glangdale 10y ago
Agreed! I hope full (not half-hearted and late) C++ support becomes the 'new normal' for ANTLR.
nonsince 10y ago
At the risk of sounding inflammatory, what would be the point of using a parser generator over a powerful applicative-style combinator library like https://github.com/Marwes/combine https://github.com/Marwes/combine ? I've used both in the past and personally massively prefer parser combinators, but I'd be interested to know what the benefits are of parser generators.
- z681781 10y ago
  I cannot answer that directly, since I've never used parser combinators. The general problem is that there's so much software out there to learn that most people are happy when they know yacc-style and antlr parser generators. Also, most people are regularly disappointed by software that over-promises, which costs them a lot of time. So they stick to what works for them. In my case, I find Bison/Menhir such great tools that I have zero inclination to even look at other things. EDIT: A corollary to this is that people who are very productive (Thompson, Torvalds, Bellard,...) don't constantly change their software stacks.
- Matthias247 10y ago
  I think one advantage of things like ANTLR is that the grammer is described in an abstract fashion and that you can then generate parsers in multiple target languages with it. With a parser combinator library you are only targeting a single language. I also think that for project outsiders and beginners it's easier to look at and understand a grammar instead of a parser in combinator style. I personally have worked with parser generators (Irony for .NET) as well as parser combinator (fparsec) and enjoyed both a lot.

rombix 10y ago

Nice to see Swift added to the list of targets. Any idea about the performance of generated parsers in Swift vs other targets?

chubot 10y ago

I'd be interested see pointers to ANTLR-based languages people are developing. I found it to be a powerful tool, but also limiting. As far as I know, ANTLR v4 is even more of a "framework" than v3, in that it dictates your program structure and always outputs a full parse tree. It appears to have lost the ability to verify the value of k for LL(k) grammars, due to a more powerful underlying algorithm. I think that simplifies it for most use cases but makes others hard or impossible.

chrisseaton 10y ago
I work on implementing languages and I often need new parsers. Either for full languages or little embedded languages or expression syntaxes of various kinds. I often think 'right, this time I'm going to write a really nice little Antlr grammar and do it properly'. But every single time I regret it because there's always some complexity that means things don't fit into the Antlr model, and making things work the way Antlr works means more complexity than just writing a simple lexer and parser by hand. My most recent example is that Ruby strings can't always be safely converted into Java strings, but Antlr wants to parse Java strings only and not a byte[]. So I had to copy my Ruby byte[] into a Java string a character at a time and just use the character as numbers pretending that they had no encoding. That's just one example. It's always some other new problem that means it's more ceremony to use Antlr than the tool saves me later one. Despite lexing and parsing being an entire sub-field of computer science, to be honest I'm not sure that in practice lexing and parsing are really problems that are so complex in the first place that a tool is ever needed.
- setq 10y ago
  Agree here. I recently implemented a small query language DSL uaing Antlr and I regret not writing my own lexer and parser. It's difficult to bend a framework like Antlr into a useable state if you hit an edge case or require unusual look ahead semantics. Next time, I'll start with a simple recursive descent parser.
- verdex_blarg 10y ago
  This is exactly the sort of thing I was looking for. I keep on thinking that my life would be easier if I just adopted something like ANTLR, but every time I look into it it looks like it's more work than it's worth. Good to hear that this has been a similar experience to people who have actually gone and actually used ANTLR.

clumsysmurf 10y ago

Just curious if anyone has used ANTLR on Android lately; I recall it was easy to run out of stack because the runtime relies on very deep recursion. Also I think there were some dependencies on AWT or Swing for debug console. https://github.com/antlr/antlr4/issues/744 https://github.com/antlr/antlr4/issues/744

butner 10y ago

Ya ANTLR (+TParr).

manishsharan 10y ago

Compiler design was one subject that I struggled with when I was an undergrad. I feel this has held me back from creating DSLs. I am looking for a gentle introduction to ANTLR that can take me from newbie to mastery. Any books anyone can recommend would be helpful.

fusiongyro 10y ago
The book "Language Implementation Patterns"[1] covers ANTLR 3, and the author (who is one of the implementors of ANTLR) walks you through successively more interesting DSLs until you're basically building a programming language, but does it in small steps. I didn't struggle with compilers as an undegrad at all, and I found this book too slow for me, so I'm optimistic that you will be pleased by the pacing. :) If you want to see a way older approach, I did really enjoy Jack Crenshaw's "Let's Build a Compiler"[2]—but it's so vintage it may be totally hipster by now. :) [1] https://www.amazon.com/Language-Implementation-Patterns-Domain-Specific-Programming/dp/193435645X/ref=sr_1_2?ie=UTF8&qid=1481865280&sr=8-2&keywords=antlr https://www.amazon.com/Language-Implementation-Patterns-Doma... [2] http://compilers.iecc.com/crenshaw/ http://compilers.iecc.com/crenshaw/
breck 10y ago
The Antlr-4 book is terrific: https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-re...
- kriro 10y ago
  Indeed. As someone with no real language background it allowed me to get to a state where I can create my own DSLs. An alternative would be to use something like Xtext (which also allows you to create langauge tools, not just the language itself). I played around with it and it's very nice (granted you'll kind of need to embrace Eclipse) but ultimately decided that ANTLR was "enough".
heysunshine123 10y ago
Look into "Pratt Parsers". It's reasonably easy to implement one based on some blog posts etc.

NicolaiS 10y ago

More detailed changelog: https://github.com/antlr/antlr4/releases/tag/4.6 https://github.com/antlr/antlr4/releases/tag/4.6