7 ms·
Show HN: Pest – Fast parser generator written in Rust
- OtterCoder 9y agoI remember playing with PEGjs back in the day, making a toy implementation of JS in JS. It had terrible Unicode support though. To build a token out of valid Unicode non-whitespace characters took pages and pages of range definitions. Is that any better here?
- runevault 9y agoRust by default uses Unicode for strings. I would be surprised if this had problems with them.
- OtterCoder 9y agoThe difficulty wasn't in the engine. JS itself is just as good with accepting Unicode characters as Rust, both in code and strings. The problem was with the PEG implementation. Without some sort of shorthand for Unicode families, defining programming languages that are valid in multiple real-world languages becomes a serious burden.
- yellowapple 9y agoThis is one of the things Perl 6 does really well,).
- hobofan 9y agoShouldn't the title better by "Pest - Fast, modern parser _in_ Rust"? The current title made me think that this would be a Rust AST parser, similar to syntex.
- grimgrin 9y agoYes this is no doubt the current implication. I was expecting something like racer.
- reificator 9y agoExactly what I came into this thread expecting. Was curious why when it looks like there's already great IDE support for Rust with existing tools.
- dikaiosune 9y agoThere's actually a parser for rust called syn, and it's very useful for procedural macros, as the current API exposes a stream of tokens, not a full AST.
- steveklabnik 9y agoTo elaborate a bit on that, exposing an AST directly means that you're tied to that version of the AST; this was considered really hard for compatibility over versions. Token streams are much easier to define an interface for, and more flexible: you're not just stuck with whatever AST representation we'd have given you. For example: https://doc.rust-lang.org/book/first-edition/procedural-macros.html https://doc.rust-lang.org/book/first-edition/procedural-macr...
- dragostis 9y agoI agree. I guess I was trying hard to suggest the fact that it can be used _with_ Rust and not the fact that it is written _in_ Rust.
- z1mm32m4n 9y agoThe confusion here is because Pest is not, in fact, a parser. It's a parser generator: you can use it to generate parsers (maybe even a parser for Rust), but it is not in itself a parser. A better title would be "Pest - Fast, modern parser generator for Rust". One cool thing is that rather than the parser generation being a standalone tool, it's written using Rust's macro system. Normal parser generators use some sort of command line tool to generate say C code which can then be used like any other library. Here, there's no need for generated code, because the Rust macro system generates it on the fly.
- dang 9y agoOK we've attempted to make a more accurate title above.
- DC-3 9y agoWhat type of parser does this produce?
- jmcomets 9y ago"Parsing expression grammars (PEGs) are an alternative to context free grammars for formally specifying syntax" [1] Basically, you have tools for context free grammars such as GNU Bison or Yacc. This is an alternative, but with PEG. For more in-depth info, there's the original PEG paper. [2] [1]: http://bford.info/packrat http://bford.info/packrat [2]: http://bford.info/pub/lang/peg http://bford.info/pub/lang/peg
- yellowapple 9y agoThat's the grammar, not the parser resulting from the grammar. The README doesn't say, but most PEGs in the wild (AFAICT) compile down to / are implemented by packrat parsers.
- yellowapple 9y agoWith PEGs (like Pest), it's probably going to be a packrat parser (which would also make sense given the project's name).
- DC-3 9y agoThank you for the actual answer. Funny how some people are so excited to give an RTFM answer that they forget to read the question. E: turns out this is wrong anyway.
- dragostis 9y agoFor now, it produces a recursive descent parser. Packrat parsing is an open-question since I'm afraid that adding a memoization layer all throughout the parser will lead to a consistent general slowdown.
- DC-3 9y agoInteresting. I'll try and find some time to read the source - am interested to see how this is implemented.
- weberc2 9y agoThis is a neat project! I might take a look at this if/when I get started on my dream language. :p One constructive criticism, which may be only an unpopular opinion, but I find creative operator overloading unnecessarily hard to read. And I don't think my opinion is entirely subjective since, by definition, you have to learn new semantics whereas a good function name would make the meaning obvious. Maybe this syntax would be familiar to people who are already familiar with formal grammar notations?
- dragostis 9y agoI agree with your point. I tried my best not to sway too far off of some of syntax I've seen in other PEG projects, while also keeping it compatible with Rust's standard macros. While pest does have this constraint any longer with the new beta, changing the grammar too much would have been a nuisance for anyone who had grammars written in the older versions.
- weberc2 9y agoUnderstood. Thanks for taking the time to explain!
- jorams 9y agoI'm a bit confused by the benchmark. I looked through the benchmark code and, without really understanding what I'm looking at, the result of the nom parser looks much closer to the "pest (custom AST)" benchmark than the "pest" benchmark. The description above the image, however, primarily compares the "pest" result. Am I misunderstanding something or is the comparison not fair?
- dragostis 9y agoI've added a clarification. The point of the benchmark is not to compete with other projects, it's merely there to put the parsing speed (and not necessarily the processing that comes after it) in a representative window of performance.
- turboladen 9y agoBeen using this for a couple weeks now and dig it. The DSL feels familiar and the speed is yummy. I'd been using LALRPOP for some months; I dig it too, but just couldn't get the speed I was after from it.