3 ms·

For those unfamiliar with it, tree-sitter (https://emacs-tree-sitter.github.io/ https://emacs-tree-sitter.github.io/) aims to be a foundational package that und

For those unfamiliar with it, tree-sitter (https://emacs-tree-sitter.github.io/ https://emacs-tree-sitter.github.io/) aims to be a foundational package that understands code structurally (think abstract syntax trees). This was done earlier via regex's, which has its limitations. This talk: https://www.thestrangeloop.com/2018/tree-sitter---a-new-parsing-system-for-programming-tools.html https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars... by the author is quite instructive as well.

robenkleene 4y ago
One thing I'll add, because I think it's an interesting insight about the priorities of code parsing for text editors: Tree-sitter is specifically designed to be very effective at parsing code that's in an invalid state. E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out.
- thaumasiotes 4y ago
  > E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out. That doesn't really bother me. The line of code you're currently typing is generally so invalid that there's not much point trying to color it. But what does bother me is the code below where you're typing changing its color as the IDE tries to make your partial line fit.
  - d-us-vb 4y ago
    The point is trying to make sure that the parser doesn't choke on the entire file just because one line is incorrect.
- srcreigh 4y ago
  That and very fast incremental parsing as you type. Slow parsing is why emacs used regexps for syntax highlighting instead of parsers in the first place.
- armchairhacker 4y ago
  Unfortunately this is a really hard problem, and I’ve used projects with tree-sitter and it actually chokes up on invalid syntax lines. Granted, maybe the projects weren’t using tree-sitter correctly. But regex parsing is surprisingly practical, so despite being really amazing at its goals, tree-sitter may not have a definitive advantage
josteink 4y ago
This HN post though is about a (new) core tree-sitter implementation in Emacs itself, which is not the same as the third party package[1] you linked. To give credit where credit is due though, it was obviously inspired by this work and what it allowed in community-maintained packages. The new implementation has been authored by Yuan Fu in close collaboration with the core Emacs maintainers and the rest of the community. It has been an ongoing effort for many, many months. This is great news, and means that also core Emacs language-binding provided as part of Emacs itself will now be able to make use of tree-sitter based parsers as well, something which wouldn't have been happening if they would have to depend on a third-party package to get those bindings. I've been somewhat involved in the process, although not a major player, but needless to say I'm very excited about these news and can't wait to see what sort of improvements this enables across the line once people start using it. [1] https://github.com/emacs-tree-sitter/elisp-tree-sitter https://github.com/emacs-tree-sitter/elisp-tree-sitter
- phtrivier 4y ago
  So, we still have to wait for each major-mode mainteners to update their code in order to benefits from those change ? In this case, how big should the change be for a "typical" mode ? Is it going to happen for C/python/typescript/etc.. anytime soon ?
  - josteink 4y ago
    If you follow the Emacs-devel mailing list, you will see many of the built in modes adds support for tree-sitter to various degrees lots of languages are already on the list (C, Python, Javascript, Bash, JSON and CSS). It also includes some new language-modes which has never been part of Emacs before (like typescript). I'd love to see C# on the list, but that might depend on me having the time to land production-grade major-mode, so that might end up happening later rather than sooner. Anyway, from what I understand what has been merged so far should all be available as part of Emacs 29 once released.
tmalsburg2 4y ago
Is there a chance that this is going to make the parsing of large org mode files faster?
- AlanYx 4y ago
  It's not related to tree-sitter, but recent work on using text properties instead of overlays for folded regions in org has improved performance opening org files with folded regions from O(n^2) to O(nlogn). See https://blog.tecosaur.com/tmio/2022-05-31-folding.html https://blog.tecosaur.com/tmio/2022-05-31-folding.html It's a big improvement in practice.
  - tmalsburg2 4y ago
    Sounds promising, thanks. The most annoyingly slow thing is the agenda buffer which takes ages to build. (Have to admit that the files from which it is sourcing entries are humongous after 15 years of managing my life with org mode.)

robenkleene 4y ago

One thing I'll add, because I think it's an interesting insight about the priorities of code parsing for text editors: Tree-sitter is specifically designed to be very effective at parsing code that's in an invalid state. E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out.

thaumasiotes 4y ago
> E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out. That doesn't really bother me. The line of code you're currently typing is generally so invalid that there's not much point trying to color it. But what does bother me is the code below where you're typing changing its color as the IDE tries to make your partial line fit.
- d-us-vb 4y ago
  The point is trying to make sure that the parser doesn't choke on the entire file just because one line is incorrect.
srcreigh 4y ago
That and very fast incremental parsing as you type. Slow parsing is why emacs used regexps for syntax highlighting instead of parsers in the first place.
armchairhacker 4y ago
Unfortunately this is a really hard problem, and I’ve used projects with tree-sitter and it actually chokes up on invalid syntax lines. Granted, maybe the projects weren’t using tree-sitter correctly. But regex parsing is surprisingly practical, so despite being really amazing at its goals, tree-sitter may not have a definitive advantage

josteink 4y ago

This HN post though is about a (new) core tree-sitter implementation in Emacs itself, which is not the same as the third party package[1] you linked. To give credit where credit is due though, it was obviously inspired by this work and what it allowed in community-maintained packages. The new implementation has been authored by Yuan Fu in close collaboration with the core Emacs maintainers and the rest of the community. It has been an ongoing effort for many, many months. This is great news, and means that also core Emacs language-binding provided as part of Emacs itself will now be able to make use of tree-sitter based parsers as well, something which wouldn't have been happening if they would have to depend on a third-party package to get those bindings. I've been somewhat involved in the process, although not a major player, but needless to say I'm very excited about these news and can't wait to see what sort of improvements this enables across the line once people start using it. [1] https://github.com/emacs-tree-sitter/elisp-tree-sitter https://github.com/emacs-tree-sitter/elisp-tree-sitter

phtrivier 4y ago
So, we still have to wait for each major-mode mainteners to update their code in order to benefits from those change ? In this case, how big should the change be for a "typical" mode ? Is it going to happen for C/python/typescript/etc.. anytime soon ?
- josteink 4y ago
  If you follow the Emacs-devel mailing list, you will see many of the built in modes adds support for tree-sitter to various degrees lots of languages are already on the list (C, Python, Javascript, Bash, JSON and CSS). It also includes some new language-modes which has never been part of Emacs before (like typescript). I'd love to see C# on the list, but that might depend on me having the time to land production-grade major-mode, so that might end up happening later rather than sooner. Anyway, from what I understand what has been merged so far should all be available as part of Emacs 29 once released.

tmalsburg2 4y ago

Is there a chance that this is going to make the parsing of large org mode files faster?

AlanYx 4y ago
It's not related to tree-sitter, but recent work on using text properties instead of overlays for folded regions in org has improved performance opening org files with folded regions from O(n^2) to O(nlogn). See https://blog.tecosaur.com/tmio/2022-05-31-folding.html https://blog.tecosaur.com/tmio/2022-05-31-folding.html It's a big improvement in practice.
- tmalsburg2 4y ago
  Sounds promising, thanks. The most annoyingly slow thing is the agenda buffer which takes ages to build. (Have to admit that the files from which it is sourcing entries are humongous after 15 years of managing my life with org mode.)