4 ms·
This is a great series! I noticed that the author was using https://github.com/hydro-project/rust-sitter https://github.com/hydro-project/rust-sitter as a pars
by ekidd 3y ago
This is a great series!
I noticed that the author was using https://github.com/hydro-project/rust-sitter https://github.com/hydro-project/rust-sitter as a parser. Which is based on https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/. I've been hearing about Tree-sitter a lot recently, so I dug into it.
Tree-sitter is a tool for generating fast, incremental parsers. In particular, the algorithm is suited towards writing "language servers" for IDEs, which re-parse code incrementally as the user works. These kinds of incremental parsers have historically been a huge problem. It looks like Tree-sitter is an enormous practical advance in this area.
And discovering that there's a way to use Tree-sitter from Rust is fantastic. From the post:
#[rust_sitter::language]
pub enum EvalExpr {
Number(
#[rust_sitter::leaf(
pattern = r"(\d+|0x[0-9a-fA-F]+)",
transform = parse_int
)]
u64
),
Symbol(
#[rust_sitter::leaf(
pattern = r"(([a-zA-Z0-9_@#.]+!)?[a-zA-Z0-9_@#.]+)",
transform = parse_sym
)]
String
),
// ...
Getting easy access to fast, incremental parsing is a huge win. And Tree-Sitter has support for being used from a huge list of languages, not just Rust.
- junon 3y agoTree sitter also has a bunch of deficiencies that don't make it ideal for a number of usecases, or sort of act bizarrely in some edge cases. Just evaluate tools like this cautiously, of course. But I like what it's done for the ecosystem as a whole!
- a1o 3y agoI still don't know how to deal with forward declarations when using tree-sitter. :/
- jamra 3y agoBecause tree-sitter lexes as it parses, you may have to use an external scanner in order to deal with this kind of stuff. Where are you stuck trying to deal with forward declarations?
- a1o 3y agoIt's a simple parser that was originally made to be used through Atom that I would like to repurpose elsewhere https://github.com/edmundito/tree-sitter-ags-script/issues/1 https://github.com/edmundito/tree-sitter-ags-script/issues/1 If this could be solved, we could port this AGS Script parser to the AGS Editor. Today, the parser Adventure Game Studio uses for the needs like auto-complete and it's very simple refactor like things uses a custom handmade parser built in C#. I think if we could leverage tree-sitter we could speed things up and repurpose it to build things like a LSP for AGS Script.
- jamra 3y agoIt looks like a bug in the grammar. I’ll bookmark this and see if I can make time for it later. Probably won’t be able to. I recently build a grammar from scratch so I’m okay at tree sitter
- a1o 3y agoOh, but if you do find time I would be externally grateful! :) The Tree Sitter generated parser is amazing for being super fast and also for being able to tolerate partially written code. This working would mean a lot for the AGS community.
- jamra 3y agoThe part that makes tree-sitter useful for this kind of thing is the error recovery. It's hard to do error recovery correctly. Tree-sitter gives you the ability to continue parsing your code which makes it useful for authoring tools.
- timmisiak 3y agoAbsolutely, rust sitter is fantastic. I haven't used any other parsers in Rust so I don't have much of a comparison point, but it's probably hard to get much more clear and concise, which I think really helps.