3 ms·

I used to feel similarly until I spent a bunch of time with nom [1]. There's not really any difference between tokens and grammar rules, in my experience. They

I used to feel similarly until I spent a bunch of time with nom [1]. There's not really any difference between tokens and grammar rules, in my experience. They're just terminal and non-terminal symbols yeah? Doing these things in one place doesn't get in the way of writing a nice LL recursive-descent parser. I'm sure it will depend on what you're doing but for me at least splitting into lex and parse has a much higher bar now. I've done it both ways many times and at least for my applications adding a separate lexer has historically been the wrong abstraction in retrospect. [1] https://blog.logrocket.com/parsing-in-rust-with-nom/ https://blog.logrocket.com/parsing-in-rust-with-nom/

Eliah_Lakhin 4y ago
Well, my personal experience was in the opposite direction actually. I used to use combinators-based approach without Lex/Syn separation (aka PEGs) for a long time. But then I came up to understanding that the separation approach is actually better in performance. And also that working and debugging of the Token sequences while writing parser manually is just more handy (at least for me). But this is my personal experience of course. I do believe too that it all depends on the goal, and parsers micro-optimizations is not that much critical in many cases, and that combinators approach actually works quite well too. As of Nom, I can say that it works quite well. But I think that the it's performance gains stem from the fact that Rust is a systems-based PL, and it optimizes function combinations just better than, let say, JavaScript or Python. In my incremental parsers library Lady Deirdre I utilize Lex/Syn separation, and the LL(1) recursive-descend parsing, and it shows much better performance than in Tree-Sitter at least on relatively big files [1]. [1] https://github.com/Eliah-Lakhin/lady-deirdre/tree/master/work/crates/examples#results https://github.com/Eliah-Lakhin/lady-deirdre/tree/master/wor...

Eliah_Lakhin 4y ago

Well, my personal experience was in the opposite direction actually. I used to use combinators-based approach without Lex/Syn separation (aka PEGs) for a long time. But then I came up to understanding that the separation approach is actually better in performance. And also that working and debugging of the Token sequences while writing parser manually is just more handy (at least for me). But this is my personal experience of course. I do believe too that it all depends on the goal, and parsers micro-optimizations is not that much critical in many cases, and that combinators approach actually works quite well too. As of Nom, I can say that it works quite well. But I think that the it's performance gains stem from the fact that Rust is a systems-based PL, and it optimizes function combinations just better than, let say, JavaScript or Python. In my incremental parsers library Lady Deirdre I utilize Lex/Syn separation, and the LL(1) recursive-descend parsing, and it shows much better performance than in Tree-Sitter at least on relatively big files [1]. [1] https://github.com/Eliah-Lakhin/lady-deirdre/tree/master/work/crates/examples#results https://github.com/Eliah-Lakhin/lady-deirdre/tree/master/wor...