6 ms·

Tree-sitter – a new parsing system for programming tools [video]

82 points by matt_d 8y ago

habitue 8y ago
Ok, now I just need to get this working in emacs.
dwenzek 8y ago
Pretty cool! I found impressive the way Max refactor a bunch of code using extend selection (16 mn 45).
transreal 8y ago
The project website is here: http://tree-sitter.github.io/tree-sitter/ http://tree-sitter.github.io/tree-sitter/, and the Github repo is here: https://github.com/tree-sitter/tree-sitter https://github.com/tree-sitter/tree-sitter.
lioeters 8y ago
Fascinating work! This talk is a part of an archive not accessible from the site's menus: https://thestrangeloop.com/2018/sessions.html https://thestrangeloop.com/2018/sessions.html
specialp 8y ago
The Strange Loop videos are in the process of being uploaded still. There are a lot that deserve a look. I was there at this talk as well. This is one of the most diverse and high level conferences out there. https://www.youtube.com/channel/UC_QIfHvN9auy2CoOdSfMWDw/videos https://www.youtube.com/channel/UC_QIfHvN9auy2CoOdSfMWDw/vid...
maxbrunsfeld 8y ago
Hey, author of Tree-sitter here. Thanks for sharing this! I'd be happy to answer any questions people have about the project.
- mncharity 8y ago
  Is the JSON representation of a grammar intended as a public interface? I noticed [1] doesn't have a version field. "No version: means version:1" is ok, but I then wondered. [1] https://github.com/tree-sitter/tree-sitter-json/blob/master/src/grammar.json https://github.com/tree-sitter/tree-sitter-json/blob/master/...
  - maxbrunsfeld 8y ago
    Good question. At one time, I had the thought that folks who disliked JavaScript might want to write their own grammar DSL in another language, which could generate compatible JSON to be consumed by the core Tree-sitter compiler library. Now, I think this flexibility is pretty unnecessary. I've also thought that some applications might want to inspect the grammar JSON at runtime in order to do some kind of meta-programming. I haven't really thought that through though. In either case, versioning would probably be a good idea. The actual parser is already versioned so that we can error out if you try to use a parser with an incompatible version of the runtime. We could just add that same version number into the grammar.
    - mncharity 8y ago
      > versioning One possible approach might be "part of spec" but "optional" and "not actively it use"? So present in schema and tests, to somewhat reduce the chance of getting locked into not having it by hypothetical others failing to support it. And easily available, so absence doesn't discourage any hypothetical interest. But without spending much time on it, absent non-hypothetical need. Maybe. > that same version number Hmm. Parser-runtime versioning would seem only lightly coupled with grammar-parser versioning? Unless there's a grammar-runtime coupling to express? EDIT: Ok, I was thinking of "grammar" in too general a sense, decoupled from choice of parser tech, rather than the usual "bison grammar tightly coupled to bison parser". Failure to RTFM. Grammar's `conflict:`, `inline:`, `word:`, `externals:` are all somewhat runtime coupled. Though perhaps looser than parser-runtime? It can be nice to flexibly move work between a compiler and its runtime, without affecting the external api.
    - 8y ago
DannyBee 8y ago
It's interesting that this appears based on Wagner and Graham's papers/work. A bunch of that was eventually open sourced, but for years it was very state of the art for incremental parsing, but imho languishing, because all the implementations were closed[1] [1] ensemble, pan, etc.
Avi-D-coder 8y ago
This is great. Has any one implemented utilized Tree-sitter in another editor? How does the performance of atom compare to vscode these days?
crb002 8y ago
TIL Tim Wagner did a thesis on IDEs before the serverless thing. The man has a passion for developer user experience.
mncharity 8y ago
Here's the js grammar[1]. Despite the worrying talk of tokenization, it looks scannerless. Yay. One big grammar. JSX is blended in. And ecmascript versions aren't segregated. Which I didn't expect... but the motivation wasn't writing picky compilers. [1] https://github.com/tree-sitter/tree-sitter-javascript/blob/master/grammar.js https://github.com/tree-sitter/tree-sitter-javascript/blob/m...
- maxbrunsfeld 8y ago
  It's not actually implemented as a scannerless parser, but the grammar API mostly abstracts away the parser/lexer distinction. We have to handle JSX and all language versions combined because for our use case, users need to be able to open up `.js` files and have them Just Work. It's a similar story with Python and Ruby - we have to handle the union of all language versions.
  - mncharity 8y ago
    > open up `.js` files and have them Just Work Nod. A single big grammar just wasn't what I'd thoughtlessly expected. When I did something similar, there was a gaggle of small grammars slicing up javascript aspects and history, then variously composed into assorted language grammar variants. But part of the motivation for that was sharing grammar fragments across languages, and "parse exactly language version X" in support of parser validation and compiler use. So a bit different use cases.
  - mncharity 8y ago
    > It's not actually implemented as a scannerless parser, but the grammar API mostly abstracts away the parser/lexer distinction. Any aspects of the 'not' part of 'mostly' which one should bear in mind?
    - maxbrunsfeld 8y ago
      Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. There's also a `token()` function that you can use to specify that an arbitrary rule should be handled by the lexer as a single token. In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.
deleted 8y ago
[deleted]

82 points by matt_d 8y ago

habitue 8y ago

Ok, now I just need to get this working in emacs.

dwenzek 8y ago

Pretty cool! I found impressive the way Max refactor a bunch of code using extend selection (16 mn 45).

transreal 8y ago

The project website is here: http://tree-sitter.github.io/tree-sitter/ http://tree-sitter.github.io/tree-sitter/, and the Github repo is here: https://github.com/tree-sitter/tree-sitter https://github.com/tree-sitter/tree-sitter.

lioeters 8y ago

Fascinating work! This talk is a part of an archive not accessible from the site's menus: https://thestrangeloop.com/2018/sessions.html https://thestrangeloop.com/2018/sessions.html

specialp 8y ago

The Strange Loop videos are in the process of being uploaded still. There are a lot that deserve a look. I was there at this talk as well. This is one of the most diverse and high level conferences out there. https://www.youtube.com/channel/UC_QIfHvN9auy2CoOdSfMWDw/videos https://www.youtube.com/channel/UC_QIfHvN9auy2CoOdSfMWDw/vid...

maxbrunsfeld 8y ago

Hey, author of Tree-sitter here. Thanks for sharing this! I'd be happy to answer any questions people have about the project.

mncharity 8y ago
Is the JSON representation of a grammar intended as a public interface? I noticed [1] doesn't have a version field. "No version: means version:1" is ok, but I then wondered. [1] https://github.com/tree-sitter/tree-sitter-json/blob/master/src/grammar.json https://github.com/tree-sitter/tree-sitter-json/blob/master/...
- maxbrunsfeld 8y ago
  Good question. At one time, I had the thought that folks who disliked JavaScript might want to write their own grammar DSL in another language, which could generate compatible JSON to be consumed by the core Tree-sitter compiler library. Now, I think this flexibility is pretty unnecessary. I've also thought that some applications might want to inspect the grammar JSON at runtime in order to do some kind of meta-programming. I haven't really thought that through though. In either case, versioning would probably be a good idea. The actual parser is already versioned so that we can error out if you try to use a parser with an incompatible version of the runtime. We could just add that same version number into the grammar.
  - mncharity 8y ago
    > versioning One possible approach might be "part of spec" but "optional" and "not actively it use"? So present in schema and tests, to somewhat reduce the chance of getting locked into not having it by hypothetical others failing to support it. And easily available, so absence doesn't discourage any hypothetical interest. But without spending much time on it, absent non-hypothetical need. Maybe. > that same version number Hmm. Parser-runtime versioning would seem only lightly coupled with grammar-parser versioning? Unless there's a grammar-runtime coupling to express? EDIT: Ok, I was thinking of "grammar" in too general a sense, decoupled from choice of parser tech, rather than the usual "bison grammar tightly coupled to bison parser". Failure to RTFM. Grammar's `conflict:`, `inline:`, `word:`, `externals:` are all somewhat runtime coupled. Though perhaps looser than parser-runtime? It can be nice to flexibly move work between a compiler and its runtime, without affecting the external api.
  - 8y ago

DannyBee 8y ago

It's interesting that this appears based on Wagner and Graham's papers/work. A bunch of that was eventually open sourced, but for years it was very state of the art for incremental parsing, but imho languishing, because all the implementations were closed[1] [1] ensemble, pan, etc.

Avi-D-coder 8y ago

This is great. Has any one implemented utilized Tree-sitter in another editor? How does the performance of atom compare to vscode these days?

crb002 8y ago

TIL Tim Wagner did a thesis on IDEs before the serverless thing. The man has a passion for developer user experience.

mncharity 8y ago

Here's the js grammar[1]. Despite the worrying talk of tokenization, it looks scannerless. Yay. One big grammar. JSX is blended in. And ecmascript versions aren't segregated. Which I didn't expect... but the motivation wasn't writing picky compilers. [1] https://github.com/tree-sitter/tree-sitter-javascript/blob/master/grammar.js https://github.com/tree-sitter/tree-sitter-javascript/blob/m...

maxbrunsfeld 8y ago
It's not actually implemented as a scannerless parser, but the grammar API mostly abstracts away the parser/lexer distinction. We have to handle JSX and all language versions combined because for our use case, users need to be able to open up `.js` files and have them Just Work. It's a similar story with Python and Ruby - we have to handle the union of all language versions.
- mncharity 8y ago
  > open up `.js` files and have them Just Work Nod. A single big grammar just wasn't what I'd thoughtlessly expected. When I did something similar, there was a gaggle of small grammars slicing up javascript aspects and history, then variously composed into assorted language grammar variants. But part of the motivation for that was sharing grammar fragments across languages, and "parse exactly language version X" in support of parser validation and compiler use. So a bit different use cases.
- mncharity 8y ago
  > It's not actually implemented as a scannerless parser, but the grammar API mostly abstracts away the parser/lexer distinction. Any aspects of the 'not' part of 'mostly' which one should bear in mind?
  - maxbrunsfeld 8y ago
    Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. There's also a `token()` function that you can use to specify that an arbitrary rule should be handled by the lexer as a single token. In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.

deleted 8y ago

[deleted]