6 ms·

Lezer: A parsing system for CodeMirror, inspired by Tree-sitter

166 points by goranmoomin 3y ago

lewisjoe 3y ago
I'm always amazed by Marijn's brilliance (Author of Lezer, CodeMirror and the awesome ProseMirror toolkit) The level of depth he dives into to perfect his projects is insane. For example, lezer is a parser generator( which by itself is not a trivial feat with novel ideas like incremental computations applied to parsing) to power his mainstream project which is CodeMirror. Like this Prosemirror too has some insane levels of engineering underneath married with thoughtful architectural decisions. Apparently, a big fan of his works.
- AlexErrant 3y ago
  He's incredibly responsive too. He just answered my question about an hour after it was asked... on a Sunday evening. Slightly off topic; is anyone aware of any Lezer grammar for regex? I've not been able to find any in the FOSS world. I suspect Regex101 has one, bit it's sadly closed source.
  - lewisjoe 3y ago
    Yeah, this happened to us too a couple of times. Once, we asked for a clarification and he took it as a sensible requirement, made the changes and took it to master in just over an evening. The next day we just had to update our package to the latest version and marvel at his response time.
  - ananthakumaran 3y ago
    Have you checked the js grammar? It has a regex grammar, though not sure if that's what you are looking for
    - AlexErrant 3y ago
      Indeed ;) https://discuss.codemirror.net/t/help-with-regex-expression-parsing/8030/2 https://discuss.codemirror.net/t/help-with-regex-expression-...
- simonw 3y ago
  He's also the author of a really great JavaScript book: https://eloquentjavascript.net/ https://eloquentjavascript.net/ - which he's been intermittently updating since 2007!
- RyanHamilton 3y ago
  Massively agree. Code mirror 5 was excellent. Code mirror 6 was a big enough improvement to justify the upgrade. I've used it as part of 2 large projects and it's handled every expanding use case I've needed it to. It supports themes, sql,js and 2 weeks ago I used its diff functionality. Really great library I can fully recommend.
  - umvi 3y ago
    Code mirror 6 is super hard to use though unless you are a front end expert. Code mirror 5 was basically plug and play. Code mirror 6 is a box of Legos and you are responsible to build and bundle your own editor from scratch.
    - spankalee 3y ago
      I have a project that's intended to make Code Mirror 6 as ready to use as any HTML tag: https://github.com/justinfagnani/codemirror-elements https://github.com/justinfagnani/codemirror-elements You can put a basic code editor on your page like: <cm-editor></cm-editor> and drop in themes and modes like: <cm-editor> <cm-lang-javascript typescript></cm-lang-javascript> <cm-theme-one-dark></cm-theme-one-dark> </cm-editor> This works in plain HTML or any framework.
      - Turing_Machine 3y ago
        Looking at this now. Thanks! How about a dist folder with a plain js file (or a js file and a css file) that you can load directly? I am sick unto death of Node and all its associates.
        kermire 3y ago
        Not as extensive as the previous poster. I wrote a small custom component wrapper a while back: https://github.com/flawiddsouza/code-mirror-custom-element https://github.com/flawiddsouza/code-mirror-custom-element.
        3y ago
bpev 3y ago
I jusy used this recently for making Traindown syntax highlighting! It was pretty intuitive, even though I haven't done this kind of thing before! https://github.com/inro-digital/lang-traindown https://github.com/inro-digital/lang-traindown https://traindown.com/ https://traindown.com/ I did notice that it seems (rightfully) very focused on the exact usecase of syntax highlighting. Do people also use this kind of system for quickly building ways to parse data from arbitrary text syntaxes?
- deleted 3y ago
  [deleted]
timenova 3y ago
(2019)
jitl 3y ago
On one hand, it’s a nice framework. I customized the Typescript one a bunch for a lil side project and enjoyed myself. On the other hand, it would be great if CodeMirror could just work with Tree-sitter or similar. There’s a lot of ecosystem around other parsing systems, and needing to figure out Lezer stuff is a big friction for adopting CodeMirror 6 for me. There are not a lot of language packages listed: https://codemirror.net/docs/community/ https://codemirror.net/docs/community/ There’s a kind of importer thingy here but it doesn’t work well for complex grammars: https://github.com/lezer-parser/import-tree-sitter https://github.com/lezer-parser/import-tree-sitter
- AlexErrant 3y ago
  > Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would. I would be curious if there's been an effort to get tree-sitter working on the web.
  - gushogg-blake 3y ago
    Tree-sitter does run on the web. I got it working for my editor, but it did involve several days' worth of effort and getting into the weeds with emscripten. Details here - https://gushogg-blake.com/p/emscripten-web-modules/ https://gushogg-blake.com/p/emscripten-web-modules/.
    - lukan 3y ago
      Can you explain the motivation why you did not use lezer? I just discovered lezer and am tinkering with integrating it into my projects, but maybe tree-sitter would be a better fit? (Speed could be a determining factor, but working with wasm can also make things slower, if you often have to get data in and out)
      - k8svet 3y ago
        Lezer is 5 years old. It's quite clear which won out between Tree-Sitter and Lezer, and it's not Lezer.
timhh 3y ago
I attempted to use this but was disheartened but the fact that it doesn't statically type node names. Tree Sitter doesn't either but it has much more of an excuse given that it targets C. https://github.com/lezer-parser/lezer/issues/8 https://github.com/lezer-parser/lezer/issues/8 The dev seems mildly hostile to outside involvement too, so I moved on. These days I use Chumsky which is Rust rather than Typescript, but also way more awesome, if you can deal with the often incomprehensible compilation errors at least! https://github.com/zesterer/chumsky https://github.com/zesterer/chumsky
- stevemk14ebr 3y ago
  None of those responses were at all hostile
  - lukan 3y ago
    He probably means this: "It's rarely a good idea to jump in and start a big refactor of someone else's library without first discussing the direction you're going in." But I agree, not hostile, just unwilling to do something, he does not want to do. Even if the other one already put work in it.
    - timhh 3y ago
      Yes it's unsolicited "advice" that shows an uncooperative attitude IMO. You can say "I don't want to do that" in a nice way without being patronising. If you look at the other closed MRs you'll see a similar attitude. E.g. here (I'd forgotten about this actually): https://github.com/lezer-parser/generator/pull/6#issuecomment-892007596 https://github.com/lezer-parser/generator/pull/6#issuecommen... Here https://github.com/lezer-parser/lr/pull/64#issuecomment-1802218165 https://github.com/lezer-parser/lr/pull/64#issuecomment-1802... It's nothing major but just emanates "difficult to work with" vibes so I didn't want to spend my time working with a project like that (looks like not many other people do either).
      - lukan 3y ago
jonrosner 3y ago
I've been using both codemirror and lezer in Yaade (https://github.com/EsperoTech/yaade https://github.com/EsperoTech/yaade). Thanks to lezer I was able to write a JSON extension language that supports Yaade environment variables. Pretty cool project and very nicely documented! I love building OSS on top of OSS.
josh11b 3y ago
(2019)
amluto 3y ago
Silly question. The OP says: if (true) something() // Comment otherStatement() > ... the comment should not be part of the if statement's node. Why not? A comment seems like it isn’t functionally of the syntax tree, but, to the extent that one treats it like it is, it does seem like it’s in the middle of an if statement.
- lioeters 3y ago
  The first line is a complete statement that finishes at the end of the line - unless there's an else clause in the following line (not counting comments or empty lines). Expressed another way: if (true) { something() } // Comment otherStatement() If this last line started with "else", then the comment would be within the if statement.
habitue 3y ago
The discussion of the error recovery strategy is pretty interesting. It seems very similar to me to monte carlo tree search, which kind of suggests using a value network to estimate which branches to prune

166 points by goranmoomin 3y ago

I'm always amazed by Marijn's brilliance (Author of Lezer, CodeMirror and the awesome ProseMirror toolkit) The level of depth he dives into to perfect his projects is insane. For example, lezer is a parser generator( which by itself is not a trivial feat with novel ideas like incremental computations applied to parsing) to power his mainstream project which is CodeMirror. Like this Prosemirror too has some insane levels of engineering underneath married with thoughtful architectural decisions. Apparently, a big fan of his works.

AlexErrant 3y ago
He's incredibly responsive too. He just answered my question about an hour after it was asked... on a Sunday evening. Slightly off topic; is anyone aware of any Lezer grammar for regex? I've not been able to find any in the FOSS world. I suspect Regex101 has one, bit it's sadly closed source.
- lewisjoe 3y ago
  Yeah, this happened to us too a couple of times. Once, we asked for a clarification and he took it as a sensible requirement, made the changes and took it to master in just over an evening. The next day we just had to update our package to the latest version and marvel at his response time.
- ananthakumaran 3y ago
  Have you checked the js grammar? It has a regex grammar, though not sure if that's what you are looking for
  - AlexErrant 3y ago
    Indeed ;) https://discuss.codemirror.net/t/help-with-regex-expression-parsing/8030/2 https://discuss.codemirror.net/t/help-with-regex-expression-...
simonw 3y ago
He's also the author of a really great JavaScript book: https://eloquentjavascript.net/ https://eloquentjavascript.net/ - which he's been intermittently updating since 2007!
RyanHamilton 3y ago
Massively agree. Code mirror 5 was excellent. Code mirror 6 was a big enough improvement to justify the upgrade. I've used it as part of 2 large projects and it's handled every expanding use case I've needed it to. It supports themes, sql,js and 2 weeks ago I used its diff functionality. Really great library I can fully recommend.
- umvi 3y ago
  Code mirror 6 is super hard to use though unless you are a front end expert. Code mirror 5 was basically plug and play. Code mirror 6 is a box of Legos and you are responsible to build and bundle your own editor from scratch.
  - spankalee 3y ago
    I have a project that's intended to make Code Mirror 6 as ready to use as any HTML tag: https://github.com/justinfagnani/codemirror-elements https://github.com/justinfagnani/codemirror-elements You can put a basic code editor on your page like: <cm-editor></cm-editor> and drop in themes and modes like: <cm-editor> <cm-lang-javascript typescript></cm-lang-javascript> <cm-theme-one-dark></cm-theme-one-dark> </cm-editor> This works in plain HTML or any framework.
    - Turing_Machine 3y ago
      Looking at this now. Thanks! How about a dist folder with a plain js file (or a js file and a css file) that you can load directly? I am sick unto death of Node and all its associates.
      - kermire 3y ago
        Not as extensive as the previous poster. I wrote a small custom component wrapper a while back: https://github.com/flawiddsouza/code-mirror-custom-element https://github.com/flawiddsouza/code-mirror-custom-element.
      - 3y ago

bpev 3y ago

I jusy used this recently for making Traindown syntax highlighting! It was pretty intuitive, even though I haven't done this kind of thing before! https://github.com/inro-digital/lang-traindown https://github.com/inro-digital/lang-traindown https://traindown.com/ https://traindown.com/ I did notice that it seems (rightfully) very focused on the exact usecase of syntax highlighting. Do people also use this kind of system for quickly building ways to parse data from arbitrary text syntaxes?

deleted 3y ago
[deleted]

timenova 3y ago

(2019)

jitl 3y ago

On one hand, it’s a nice framework. I customized the Typescript one a bunch for a lil side project and enjoyed myself. On the other hand, it would be great if CodeMirror could just work with Tree-sitter or similar. There’s a lot of ecosystem around other parsing systems, and needing to figure out Lezer stuff is a big friction for adopting CodeMirror 6 for me. There are not a lot of language packages listed: https://codemirror.net/docs/community/ https://codemirror.net/docs/community/ There’s a kind of importer thingy here but it doesn’t work well for complex grammars: https://github.com/lezer-parser/import-tree-sitter https://github.com/lezer-parser/import-tree-sitter

AlexErrant 3y ago
> Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would. I would be curious if there's been an effort to get tree-sitter working on the web.
- gushogg-blake 3y ago
  Tree-sitter does run on the web. I got it working for my editor, but it did involve several days' worth of effort and getting into the weeds with emscripten. Details here - https://gushogg-blake.com/p/emscripten-web-modules/ https://gushogg-blake.com/p/emscripten-web-modules/.
  - lukan 3y ago
    Can you explain the motivation why you did not use lezer? I just discovered lezer and am tinkering with integrating it into my projects, but maybe tree-sitter would be a better fit? (Speed could be a determining factor, but working with wasm can also make things slower, if you often have to get data in and out)
    - k8svet 3y ago
      Lezer is 5 years old. It's quite clear which won out between Tree-Sitter and Lezer, and it's not Lezer.

timhh 3y ago

I attempted to use this but was disheartened but the fact that it doesn't statically type node names. Tree Sitter doesn't either but it has much more of an excuse given that it targets C. https://github.com/lezer-parser/lezer/issues/8 https://github.com/lezer-parser/lezer/issues/8 The dev seems mildly hostile to outside involvement too, so I moved on. These days I use Chumsky which is Rust rather than Typescript, but also way more awesome, if you can deal with the often incomprehensible compilation errors at least! https://github.com/zesterer/chumsky https://github.com/zesterer/chumsky

stevemk14ebr 3y ago
None of those responses were at all hostile
- lukan 3y ago
  He probably means this: "It's rarely a good idea to jump in and start a big refactor of someone else's library without first discussing the direction you're going in." But I agree, not hostile, just unwilling to do something, he does not want to do. Even if the other one already put work in it.
  - timhh 3y ago
    Yes it's unsolicited "advice" that shows an uncooperative attitude IMO. You can say "I don't want to do that" in a nice way without being patronising. If you look at the other closed MRs you'll see a similar attitude. E.g. here (I'd forgotten about this actually): https://github.com/lezer-parser/generator/pull/6#issuecomment-892007596 https://github.com/lezer-parser/generator/pull/6#issuecommen... Here https://github.com/lezer-parser/lr/pull/64#issuecomment-1802218165 https://github.com/lezer-parser/lr/pull/64#issuecomment-1802... It's nothing major but just emanates "difficult to work with" vibes so I didn't want to spend my time working with a project like that (looks like not many other people do either).
    - lukan 3y ago

jonrosner 3y ago

I've been using both codemirror and lezer in Yaade (https://github.com/EsperoTech/yaade https://github.com/EsperoTech/yaade). Thanks to lezer I was able to write a JSON extension language that supports Yaade environment variables. Pretty cool project and very nicely documented! I love building OSS on top of OSS.

josh11b 3y ago

(2019)

amluto 3y ago

Silly question. The OP says: if (true) something() // Comment otherStatement() > ... the comment should not be part of the if statement's node. Why not? A comment seems like it isn’t functionally of the syntax tree, but, to the extent that one treats it like it is, it does seem like it’s in the middle of an if statement.

lioeters 3y ago
The first line is a complete statement that finishes at the end of the line - unless there's an else clause in the following line (not counting comments or empty lines). Expressed another way: if (true) { something() } // Comment otherStatement() If this last line started with "else", then the comment would be within the if statement.

habitue 3y ago

The discussion of the error recovery strategy is pretty interesting. It seems very similar to me to monte carlo tree search, which kind of suggests using a value network to estimate which branches to prune