3 ms·
I've just built a parser in Rust that seems to function in exactly the way you say. It reads the text into a single buffer, assembles the AST nodes in an arena
by rstuart4133 1y ago
I've just built a parser in Rust that seems to function in exactly the way you say. It reads the text into a single buffer, assembles the AST nodes in an arena (indexes are NonZeroU8's to save space), and all token nodes refer to the original rather duplicating the text. The things I'm parsing are smallish (much less than 64k, 256 tokens), but their is around 100M of them so was worth the effort to build it in a way that used 0 allocations (the arena is re-used).
It was a bit of effort, but it can't be that difficult given it's my first non-toy Rust program and it wasn't that hard to get going. I'd written maybe 50 training exercises in Rust before that, over the years. Yes, the same thing in C would have been faster and easier to write (I've written many 100's of thousands of lines of C over the years), but I'm not sure it would of worked the day after it compiled.
I also had to build a 2nd parser that looked at maybe 100 lines. It was clone() all the way down for that one because I could afford the cost. I think that was easier to write in Rust than C, mostly because of the very good standard library Rust comes with.