4 ms·

> Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very h

> Unfortunately, tree-sitter is written in C, which is still awkward to run in the browser (and CodeMirrror targets non-WASM browsers). It also generates very hefty grammar files because it makes the size/speed trade-off in a different way than a web system would. I would be curious if there's been an effort to get tree-sitter working on the web.

gushogg-blake 3y ago
Tree-sitter does run on the web. I got it working for my editor, but it did involve several days' worth of effort and getting into the weeds with emscripten. Details here - https://gushogg-blake.com/p/emscripten-web-modules/ https://gushogg-blake.com/p/emscripten-web-modules/.
- lukan 3y ago
  Can you explain the motivation why you did not use lezer? I just discovered lezer and am tinkering with integrating it into my projects, but maybe tree-sitter would be a better fit? (Speed could be a determining factor, but working with wasm can also make things slower, if you often have to get data in and out)
  - k8svet 3y ago
    Lezer is 5 years old. It's quite clear which won out between Tree-Sitter and Lezer, and it's not Lezer.
    - lukan 3y ago
      Sorry, but it is not clear to me. Lezer obviously works in the real world, as codemirror is quite widespread (integrated in chrome dev tools for example). And since I use codemirror anyway, it might make sense to also use lezer, unless there are downsides I am not aware of.
      - jitl 3y ago
        I think Lezer is great if there’s a parser for your language, or you only need to support a few languages and you have time to build grammars yourself. When you need to build a general tool that supports a ton of languages, that’s when Lezer doesn’t fit as well since there just aren’t a ton of existing grammars out there.
        lukan 3y ago
        I see. Well I am fine with just js and ts for now, but long term I need more language support. So I probably should choose tree sitter.
      - deleted 3y ago
        [deleted]
  - gushogg-blake 3y ago
    I didn't know about Lezer. I think I probably would have used it if I'd known - or at least tried it before Tree-sitter, as WASM obviously brings in extra complexity. Having used Lezer and CodeMirror a bit on a different project, they both seem like they are probably high quality and well thought out projects from a speed and reliability perspective, but I found the architecture and docs to be confusing and unergonomic. The main issue I have with the docs/architecture (for CM at least) is that they use a concept called "facets" without really explaining what it means, and to be honest it felt like a level of abstraction/indirection that my brain couldn't handle. The other issue with the docs is that they don't seem contextualised enough, somehow. They would list the functions/methods of an object but not really explain how it fit into the system as a whole. Maybe this is to do with not understanding facets, or CM's extension architecture, enough, or something, but it was definitely a recurring theme of my experience. I also found the CM API slightly confusing in that (almost) everything you do is via a function that takes some state object as input, as opposed to via calling methods on that object, but that's more of a style issue once you realise that you have to `import {doSomething} from "codemirror"` and `doSomething(state)` as opposed to `state.doSomething()`.
    - lukan 3y ago
      Thx. I will just try both I think. The other comment mentioned treesitter now provides native bindings, so it should be easy to try it out.
jitl 3y ago
I learned from a google search that these days upstream tree-sitter provides WebAssembly bindings. Source: https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web https://github.com/tree-sitter/tree-sitter/tree/master/lib/b... NPM: https://www.npmjs.com/package/web-tree-sitter https://www.npmjs.com/package/web-tree-sitter Download from the latest Github release: js file (https://github.com/tree-sitter/tree-sitter/releases/download/v0.22.2/tree-sitter.js https://github.com/tree-sitter/tree-sitter/releases/download...) and wasm file (https://github.com/tree-sitter/tree-sitter/releases/download/v0.22.2/tree-sitter.wasm https://github.com/tree-sitter/tree-sitter/releases/download...) It's great to see upstream maintaining the bindings. I maintain a Typescript/Empscripten/WebAssembtly binding for quickjs and it's more involved than I would like. Besides providing a WASM build, Emscripten can build a lot of C code to straight JS, which you can use in non-WebAssembly browsers, or in runtimes like Cloudflare or Vercel Edge that support WebAssembly but it's a pain in the behind. Unfortunately the tree-sitter upstream doesn't output such a build but given they already maintain the bindings it wouldn't be a lot of effort to pass the option to emcc (set -s WASM=0 -s SINGLE_FILE=1 instead of -s WASM=1)
conartist6 3y ago
I will probably be able to run tree-sitter grammars in plain JS (no WASM) soonish (within the next six months, say) on the BABLR VM

gushogg-blake 3y ago

Tree-sitter does run on the web. I got it working for my editor, but it did involve several days' worth of effort and getting into the weeds with emscripten. Details here - https://gushogg-blake.com/p/emscripten-web-modules/ https://gushogg-blake.com/p/emscripten-web-modules/.

lukan 3y ago
Can you explain the motivation why you did not use lezer? I just discovered lezer and am tinkering with integrating it into my projects, but maybe tree-sitter would be a better fit? (Speed could be a determining factor, but working with wasm can also make things slower, if you often have to get data in and out)
- k8svet 3y ago
  Lezer is 5 years old. It's quite clear which won out between Tree-Sitter and Lezer, and it's not Lezer.
  - lukan 3y ago
    Sorry, but it is not clear to me. Lezer obviously works in the real world, as codemirror is quite widespread (integrated in chrome dev tools for example). And since I use codemirror anyway, it might make sense to also use lezer, unless there are downsides I am not aware of.
    - jitl 3y ago
      I think Lezer is great if there’s a parser for your language, or you only need to support a few languages and you have time to build grammars yourself. When you need to build a general tool that supports a ton of languages, that’s when Lezer doesn’t fit as well since there just aren’t a ton of existing grammars out there.
      - lukan 3y ago
        I see. Well I am fine with just js and ts for now, but long term I need more language support. So I probably should choose tree sitter.
    - deleted 3y ago
      [deleted]
- gushogg-blake 3y ago
  I didn't know about Lezer. I think I probably would have used it if I'd known - or at least tried it before Tree-sitter, as WASM obviously brings in extra complexity. Having used Lezer and CodeMirror a bit on a different project, they both seem like they are probably high quality and well thought out projects from a speed and reliability perspective, but I found the architecture and docs to be confusing and unergonomic. The main issue I have with the docs/architecture (for CM at least) is that they use a concept called "facets" without really explaining what it means, and to be honest it felt like a level of abstraction/indirection that my brain couldn't handle. The other issue with the docs is that they don't seem contextualised enough, somehow. They would list the functions/methods of an object but not really explain how it fit into the system as a whole. Maybe this is to do with not understanding facets, or CM's extension architecture, enough, or something, but it was definitely a recurring theme of my experience. I also found the CM API slightly confusing in that (almost) everything you do is via a function that takes some state object as input, as opposed to via calling methods on that object, but that's more of a style issue once you realise that you have to `import {doSomething} from "codemirror"` and `doSomething(state)` as opposed to `state.doSomething()`.
  - lukan 3y ago
    Thx. I will just try both I think. The other comment mentioned treesitter now provides native bindings, so it should be easy to try it out.

jitl 3y ago

I learned from a google search that these days upstream tree-sitter provides WebAssembly bindings. Source: https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web https://github.com/tree-sitter/tree-sitter/tree/master/lib/b... NPM: https://www.npmjs.com/package/web-tree-sitter https://www.npmjs.com/package/web-tree-sitter Download from the latest Github release: js file (https://github.com/tree-sitter/tree-sitter/releases/download/v0.22.2/tree-sitter.js https://github.com/tree-sitter/tree-sitter/releases/download...) and wasm file (https://github.com/tree-sitter/tree-sitter/releases/download/v0.22.2/tree-sitter.wasm https://github.com/tree-sitter/tree-sitter/releases/download...) It's great to see upstream maintaining the bindings. I maintain a Typescript/Empscripten/WebAssembtly binding for quickjs and it's more involved than I would like. Besides providing a WASM build, Emscripten can build a lot of C code to straight JS, which you can use in non-WebAssembly browsers, or in runtimes like Cloudflare or Vercel Edge that support WebAssembly but it's a pain in the behind. Unfortunately the tree-sitter upstream doesn't output such a build but given they already maintain the bindings it wouldn't be a lot of effort to pass the option to emcc (set -s WASM=0 -s SINGLE_FILE=1 instead of -s WASM=1)

conartist6 3y ago

I will probably be able to run tree-sitter grammars in plain JS (no WASM) soonish (within the next six months, say) on the BABLR VM