2 ms·

This is a great article and I think tree-sitter's design choices are creating a budding ecosystem of new program analysis tooling. This article discusses the sp

by ievans 5y ago

This is a great article and I think tree-sitter's design choices are creating a budding ecosystem of new program analysis tooling. This article discusses the speed advantages, but I think the fact that tree-sitter is dependency-free (which the author's previous article did mention) is worth highlighting again. For context, some of my teammates maintain the OCaml tree-sitter bindings and often contribute to grammars as part of our work on Semgrep (Semgrep uses tree-sitter for searching code and parsing queries that are code snippets themselves into AST matchers). Often when writing a linter, you need to bring along the runtime of the language you're targeting. E.g., in python if you're writing a parser using the builtin `ast` module, you need to match the language version & features. So you can't parse Python 3 code with Pylint running on Python 2.7, for instance. This ends up being more obnoxious than you'd think at first, especially if you're targeting multiple languages. Before tree-sitter, using a language's built-in AST tooling was often the best approach because it is guaranteed to keep up with the latest syntax. IMO the genius of tree-sitter is that it's made it way easier than with traditional grammars to keep the language parsers updated. Highly recommend Max Brunsfield's strange loop talk if you want to learn more about the design choices behind tree-sitter: https://www.thestrangeloop.com/2018/tree-sitter---a-new-parsing-system-for-programming-tools.html https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars...

dcreager 5y ago
Many folks have often (correctly!) pointed out that each language's existing parsing packages will be more guaranteed to keep up with language changes. Our hope is that if we can get a critical mass of useful tools that all use tree-sitter under the covers, there will _also_ be incentives to ensure that the tree-sitter parser for each language stays up-to-date. And because of how tree-sitter grammars typically live in dedicated repos, that work might be done by external volunteers, and not by the core language developers. Another benefit to our approach is that it should be much easier to adapt OP's linter to work with other languages, since in a sense it's “parameterized” by the language grammar and queries. If you use Python's `ast` module, you can't easily adapt that code to work on Go programs, for instance.

by ievans 5y ago

dcreager 5y ago

Many folks have often (correctly!) pointed out that each language's existing parsing packages will be more guaranteed to keep up with language changes. Our hope is that if we can get a critical mass of useful tools that all use tree-sitter under the covers, there will _also_ be incentives to ensure that the tree-sitter parser for each language stays up-to-date. And because of how tree-sitter grammars typically live in dedicated repos, that work might be done by external volunteers, and not by the core language developers. Another benefit to our approach is that it should be much easier to adapt OP's linter to work with other languages, since in a sense it's “parameterized” by the language grammar and queries. If you use Python's `ast` module, you can't easily adapt that code to work on Go programs, for instance.