8 ms·

Emacs: Feature/tree-sitter merged into master

305 points by signa11 4y ago

signa11 4y ago
For those unfamiliar with it, tree-sitter (https://emacs-tree-sitter.github.io/ https://emacs-tree-sitter.github.io/) aims to be a foundational package that understands code structurally (think abstract syntax trees). This was done earlier via regex's, which has its limitations. This talk: https://www.thestrangeloop.com/2018/tree-sitter---a-new-parsing-system-for-programming-tools.html https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars... by the author is quite instructive as well.
- robenkleene 4y ago
  One thing I'll add, because I think it's an interesting insight about the priorities of code parsing for text editors: Tree-sitter is specifically designed to be very effective at parsing code that's in an invalid state. E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out.
  - thaumasiotes 4y ago
    > E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out. That doesn't really bother me. The line of code you're currently typing is generally so invalid that there's not much point trying to color it. But what does bother me is the code below where you're typing changing its color as the IDE tries to make your partial line fit.
    - d-us-vb 4y ago
      The point is trying to make sure that the parser doesn't choke on the entire file just because one line is incorrect.
  - srcreigh 4y ago
    That and very fast incremental parsing as you type. Slow parsing is why emacs used regexps for syntax highlighting instead of parsers in the first place.
  - armchairhacker 4y ago
    Unfortunately this is a really hard problem, and I’ve used projects with tree-sitter and it actually chokes up on invalid syntax lines. Granted, maybe the projects weren’t using tree-sitter correctly. But regex parsing is surprisingly practical, so despite being really amazing at its goals, tree-sitter may not have a definitive advantage
- josteink 4y ago
  This HN post though is about a (new) core tree-sitter implementation in Emacs itself, which is not the same as the third party package[1] you linked. To give credit where credit is due though, it was obviously inspired by this work and what it allowed in community-maintained packages. The new implementation has been authored by Yuan Fu in close collaboration with the core Emacs maintainers and the rest of the community. It has been an ongoing effort for many, many months. This is great news, and means that also core Emacs language-binding provided as part of Emacs itself will now be able to make use of tree-sitter based parsers as well, something which wouldn't have been happening if they would have to depend on a third-party package to get those bindings. I've been somewhat involved in the process, although not a major player, but needless to say I'm very excited about these news and can't wait to see what sort of improvements this enables across the line once people start using it. [1] https://github.com/emacs-tree-sitter/elisp-tree-sitter https://github.com/emacs-tree-sitter/elisp-tree-sitter
  - phtrivier 4y ago
    So, we still have to wait for each major-mode mainteners to update their code in order to benefits from those change ? In this case, how big should the change be for a "typical" mode ? Is it going to happen for C/python/typescript/etc.. anytime soon ?
    - josteink 4y ago
      If you follow the Emacs-devel mailing list, you will see many of the built in modes adds support for tree-sitter to various degrees lots of languages are already on the list (C, Python, Javascript, Bash, JSON and CSS). It also includes some new language-modes which has never been part of Emacs before (like typescript). I'd love to see C# on the list, but that might depend on me having the time to land production-grade major-mode, so that might end up happening later rather than sooner. Anyway, from what I understand what has been merged so far should all be available as part of Emacs 29 once released.
- tmalsburg2 4y ago
  Is there a chance that this is going to make the parsing of large org mode files faster?
  - AlanYx 4y ago
    It's not related to tree-sitter, but recent work on using text properties instead of overlays for folded regions in org has improved performance opening org files with folded regions from O(n^2) to O(nlogn). See https://blog.tecosaur.com/tmio/2022-05-31-folding.html https://blog.tecosaur.com/tmio/2022-05-31-folding.html It's a big improvement in practice.
    - tmalsburg2 4y ago
      Sounds promising, thanks. The most annoyingly slow thing is the agenda buffer which takes ages to build. (Have to admit that the files from which it is sourcing entries are humongous after 15 years of managing my life with org mode.)
hnarayanan 4y ago
To save others from needing to research what this is: tree-sitter is an Emacs binding for Tree-sitter, an incremental parsing system. It aims to be the foundation for a new breed of Emacs packages that understand code structurally.
deafpolygon 4y ago
But have they made Emacs multithreaded yet?
- terminal_d 4y ago
  Have you seriously tried using emacs yet? I always find this sort of text editor comparisons pointless. And anyway, emacs is hardly a "text editor". It's a text manipulation engine, likely the only one.
  - deafpolygon 4y ago
    I have. Most long running commands of any kind blocks until it is done.
    - terminal_d 4y ago
      What sort of long-running commands are you using? There's an `async` call for system functions in elisp.
mickeyp 4y ago
If you're wondering what Tree-Sitter is and why Emacs would want it, I wrote about it a while ago: https://www.masteringemacs.org/article/tree-sitter-complications-of-parsing-languages https://www.masteringemacs.org/article/tree-sitter-complicat...
precompute 4y ago
Awesome! Pretty soon vanilla emacs will be more than enough, won't need melpa packages.
mcqueenjordan 4y ago
I have a huge belief in tree-sitter. I think it's going to continue to grow and become an important tool, especially in security/code tooling contexts.
- norir 4y ago
  The main innovation of tree-sitter, even more than incremental parsing, as I see it is that it provides a uniform api for traversing a parse tree, which makes it relatively straightforward to onboard a new language to a tool with tree-sitter support. The problem though is that the tree-sitter grammar is nearly always going to be an approximation to the actual language grammar, unless the language compiler/interpreter uses tree-sitter for parsing. To me, this is problematic for tooling because it is always possible for a tree-sitter based tool to be flat out wrong relative to the actual language. For syntax highlighting, this is generally not a huge deal (and tree-sitter does generally work well, though there are exceptions), but I'd be more cautious with security tools based on tree-sitter. If all languages changed their reference parsers to tree-sitter, this would be moot, but that seems unlikely. Language parsers are often optimized beyond what is possible in a general purpose parser generator like tree-sitter and/or have ambiguities that cannot be resolved with the tree-sitter dsl. What feels perhaps likely in the future is that a standard parse tree api emerges, analogous to lsp, and then language parsers could emit trees traversable by this api. Maybe it's just the tree-sitter c api with an alternate front end? Hard to say, but I suspect either something better than (but likely at least partially inspired by) tree-sitter will emerge or we will get stuck in a local minimum with tooling based on slightly incorrect language parsers.
  - debugnik 4y ago
    > unless the language compiler/interpreter uses tree-sitter for parsing Doubtful, last time I tried tree-sitter would parse invalid inputs without even tagging any errors in the parse tree. For example, it would silently accept extra tokens, or keywords in the place of identifiers. Replacing the built-in lexer and then validating the parse tree for correctness would be close to writing the grammar twice. And accepting partially correct inputs within the compiler toolchain isn't too hard, so I don't really see the advantage of agreeing on tree-sitter and not just on a parse tree representation that editors can then query, as you then suggested. If the big deal is having it execute client-side or being sandboxed, I feel that's orthogonal to parsing algorithms.
antipaul 4y ago
What's the "explain it like I'm 5 years old" (ELI5) for tree-sitter? Why should I, an emacs user but not lisp hacker, care about it?
- chriswarbo 4y ago
  tree-sitter creates parsers, e.g. for programming languages, config formats, etc. Emacs modes can use those parsers on buffer contents, e.g. for syntax colouring/highlighting, finding matching delimiters (e.g. moving the cursor over an `if`, and having all the corresponding clauses (e.g. else/elif/fi) highlighted), for contextual editing (e.g. escaping " when inside a string), etc. This can be remarkably tricky to get right; e.g. consider languages which can splice expressions inside strings (which can themselves contain strings, containing spliced expressions, etc.) Using tree-sitter should make this easier and more robust (i.e. less time spent implementing parsers; more time spent implementing features!). I think it would also allow grammars to be re-used across different tools, which should improve support for obscure/niche languages.
  - 2pEXgD0fZ5cF 4y ago
    Does this mean that every emacs language package would automatically make use of this once it is built in. Or will this rather enable the possibility to write/rewrite programming language modes so they make use of tree-sitter because they can assume it is available in the default emacs install from then on?
    - omnicognate 4y ago
      It needs to be explicitly used. As far as I'm aware it doesn't slot in behind an existing API and magically make things better.
      - 2pEXgD0fZ5cF 4y ago
        Got it. Are there any beginner guides yet on how to write an emacs (language) package while making use of it?
Ian_Macharia 4y ago
I use tree-sitter in neovim and the syntax highlighting is on par with VSCode
mickeyp 4y ago
If you're wondering what Tree-Sitter is and why Emacs would want it, I wrote about it a while ago: https://www.masteringemacs.org/article/tree-sitter-complications-of-parsing-languages https://www.masteringemacs.org/article/tree-sitter-complicat...
- sph 4y ago
  There is not an Emacs topic I'd like to know more about haven't already covered on your website. Thanks, your articles and your book are the best guides into the world of Emacs.
  - mickeyp 4y ago
    Thank you :) I'm glad you like my site and my book!
- afry1 4y ago
  That point you make about syntax highlighting being slow while using eglot/LSP-mode is a great one. I've been a bit underwhelmed with eglot, and I think that must be the reason: it feels like I'm programming in a bowl of oatmeal with every keystroke. Do you have any tips or guides for using treesitter for syntax highlighting/structural editing and eglot/LSP-mode for everything else?
  - omnicognate 4y ago
    AFAIK eglot/lsp-mode don't do syntax highlighting. The article's just explaining why that is (i.e. because it would be too slow). If you don't have tree-sitter your syntax highlighting will be done by the regex based font-lock-mode. I don't think eglot/lsp-mode make that slower, and I believe tree-sitter should speed it up (and make it more correct) without affecting them. I haven't tried it yet, though.
    - TeMPOraL 4y ago
      It must be a matter of configuration. At work, I use buffer re-fontification as an indicator that clangd correctly processed the C++ source file I just opened. That's with an Emacs built from source ~half a year ago + LSP mode.
erganemic 4y ago
I'm really impressed with the strides Emacs has made recently: native compilation, project.el, eglot, and now tree-sitter? As a user who hadn't kept up with development news until recently, I'd always mentally sorted Emacs into the same taxonomy as stuff like `find`: old, powerful, with a clunky interface and a stodgy resistance to updating how it does things (though not without reason). I'm increasingly feeling like that's an unfair classification on my part--I'm genuinely super excited to see where Emacs is in 5 years.
- sph 4y ago
  Yes, it feels there is a lot of momentum going on recently. Both neovim and Emacs are being improved at breakneck pace, and it is quite incredible for such an old piece of software with, dare I say, a quirky contribution model. The maintainers are working really hard on keeping it current and competitive.
- bloopernova 4y ago
  I'm really hoping that Emacs becomes multithreaded somehow. Or at least improves some operations so that they're non-blocking. I've been using Emacs primarily for org-mode/roam/babel for a few years now. I'm very glad for its existence, I really think I've become a more effective DevOps person because of it.
  - wyuenho 4y ago
    I'll be entirely satisfied with a process/event queue/loop that we can submit tasks to like Javascript's. There is already a command loop in Emacs, we just can't use it for anything other than input events and commands. Once we have an good event loop, we can build a state machine like Redux on it, then we can start rebuilding the display machinery, then we can start deleting all those hooks that constantly interfere with each other...
    - ywei3410 4y ago
      It already has a process/event queue like JS since Emacs 26. What did you mean in particular?
jamborine 4y ago
I'm so impressed by this
arc-in-space 4y ago
Interesting. Current syntax highlighting in emacs is mostly fine, except for how it occasionally blows up - an unterminated quote, in some languages, can run out and match the entire tail as a string, potentially freezing on a large file. Paredit(♡) avoids this by not even letting you do that unless you ask very nicely. I wonder if tree-sitter helps there.
- clircle 4y ago
  Maybe you and i are using different major modes, but i see the warts of regex high lighting in eMacs. It’s quite bad in auctex
davidkunz 4y ago
Congratulations, Emacs! I hope it will be a similar success story as in Neovim. If more systems use it, the question "should my programming language provide a Tree-Sitter parser" becomes a no brainer.
aidenn0 4y ago
Incremental parsing of incorrect code is one of those things that is literally impossible in the general case, but tree-sitter has found a lot of good ways to do it that are not just possible for a large fraction of reality, but also performant. It's hard to understate how impressive a piece of engineering this is.
- Jenk 4y ago
  I think you mean "hard to overstate" :)
  - aidenn0 4y ago
    Indeed. Too late to edit now though.
Wonnk13 4y ago
So if I have a fairly unremarkable setup with LSP to give me completions, what do I get by fooling around with tree-sitter. It seems like this is more geared toward building an AST, so I'm not sure how it would present itself to the end user currently?
- omnicognate 4y ago
  Faster and more correct syntax highlighting is the main benefit atm, as I understand it. In general it's for functionality that needs to understand syntax but doesn't need a full compilation-level understanding of the code, that can benefit from much faster responses than an LSP server can provide and that people may want working out of the box for many languages without having to install and configure language servers, generate compilation DBs, etc.
  - hvis 4y ago
    Also: indentation, beginning-of-defun/end-of-defun navigation, and probably Imenu later. All the basics of major modes that up until now have been implemented using bespoke algorithms, rewritten ad-hoc for each language.
    - srcreigh 4y ago
      For easy thread reading. Imenu is a way to jump around in the file M-x imenu gives menu of different types of locations (ex functions), you select functions, then you can select the function to jump to.
- josteink 4y ago
  I think at this point it’s a new building primitive mostly aimed at major-mode authors. That said, tree-sitter should make it possible to create paredit-like implementations for languages not LISP and other stuff like that, which IMO could turn out to be really neat. As a change, this is quite significant, but not directly aimed at end-users.
  - 4y ago
jpe90 4y ago
Has anyone been able to replicate vscode/coc.nvim’s instant lsp autocompletion on each keystroke with emacs when eglot/company are enabled? I tried configuring company to lower the idle delay to 0 and set the minimum prefix to 0, but the gui locks up on every keystroke

305 points by signa11 4y ago

For those unfamiliar with it, tree-sitter (https://emacs-tree-sitter.github.io/ https://emacs-tree-sitter.github.io/) aims to be a foundational package that understands code structurally (think abstract syntax trees). This was done earlier via regex's, which has its limitations. This talk: https://www.thestrangeloop.com/2018/tree-sitter---a-new-parsing-system-for-programming-tools.html https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars... by the author is quite instructive as well.

robenkleene 4y ago
One thing I'll add, because I think it's an interesting insight about the priorities of code parsing for text editors: Tree-sitter is specifically designed to be very effective at parsing code that's in an invalid state. E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out.
- thaumasiotes 4y ago
  > E.g., think about adding a new line to a program, the new line you're adding is typically invalid for the majority of time until you've finished typing it out. That doesn't really bother me. The line of code you're currently typing is generally so invalid that there's not much point trying to color it. But what does bother me is the code below where you're typing changing its color as the IDE tries to make your partial line fit.
  - d-us-vb 4y ago
    The point is trying to make sure that the parser doesn't choke on the entire file just because one line is incorrect.
- srcreigh 4y ago
  That and very fast incremental parsing as you type. Slow parsing is why emacs used regexps for syntax highlighting instead of parsers in the first place.
- armchairhacker 4y ago
  Unfortunately this is a really hard problem, and I’ve used projects with tree-sitter and it actually chokes up on invalid syntax lines. Granted, maybe the projects weren’t using tree-sitter correctly. But regex parsing is surprisingly practical, so despite being really amazing at its goals, tree-sitter may not have a definitive advantage
josteink 4y ago
This HN post though is about a (new) core tree-sitter implementation in Emacs itself, which is not the same as the third party package[1] you linked. To give credit where credit is due though, it was obviously inspired by this work and what it allowed in community-maintained packages. The new implementation has been authored by Yuan Fu in close collaboration with the core Emacs maintainers and the rest of the community. It has been an ongoing effort for many, many months. This is great news, and means that also core Emacs language-binding provided as part of Emacs itself will now be able to make use of tree-sitter based parsers as well, something which wouldn't have been happening if they would have to depend on a third-party package to get those bindings. I've been somewhat involved in the process, although not a major player, but needless to say I'm very excited about these news and can't wait to see what sort of improvements this enables across the line once people start using it. [1] https://github.com/emacs-tree-sitter/elisp-tree-sitter https://github.com/emacs-tree-sitter/elisp-tree-sitter
- phtrivier 4y ago
  So, we still have to wait for each major-mode mainteners to update their code in order to benefits from those change ? In this case, how big should the change be for a "typical" mode ? Is it going to happen for C/python/typescript/etc.. anytime soon ?
  - josteink 4y ago
    If you follow the Emacs-devel mailing list, you will see many of the built in modes adds support for tree-sitter to various degrees lots of languages are already on the list (C, Python, Javascript, Bash, JSON and CSS). It also includes some new language-modes which has never been part of Emacs before (like typescript). I'd love to see C# on the list, but that might depend on me having the time to land production-grade major-mode, so that might end up happening later rather than sooner. Anyway, from what I understand what has been merged so far should all be available as part of Emacs 29 once released.
tmalsburg2 4y ago
Is there a chance that this is going to make the parsing of large org mode files faster?
- AlanYx 4y ago
  It's not related to tree-sitter, but recent work on using text properties instead of overlays for folded regions in org has improved performance opening org files with folded regions from O(n^2) to O(nlogn). See https://blog.tecosaur.com/tmio/2022-05-31-folding.html https://blog.tecosaur.com/tmio/2022-05-31-folding.html It's a big improvement in practice.
  - tmalsburg2 4y ago
    Sounds promising, thanks. The most annoyingly slow thing is the agenda buffer which takes ages to build. (Have to admit that the files from which it is sourcing entries are humongous after 15 years of managing my life with org mode.)

hnarayanan 4y ago

To save others from needing to research what this is: tree-sitter is an Emacs binding for Tree-sitter, an incremental parsing system. It aims to be the foundation for a new breed of Emacs packages that understand code structurally.

deafpolygon 4y ago

But have they made Emacs multithreaded yet?

terminal_d 4y ago
Have you seriously tried using emacs yet? I always find this sort of text editor comparisons pointless. And anyway, emacs is hardly a "text editor". It's a text manipulation engine, likely the only one.
- deafpolygon 4y ago
  I have. Most long running commands of any kind blocks until it is done.
  - terminal_d 4y ago
    What sort of long-running commands are you using? There's an `async` call for system functions in elisp.

mickeyp 4y ago

If you're wondering what Tree-Sitter is and why Emacs would want it, I wrote about it a while ago: https://www.masteringemacs.org/article/tree-sitter-complications-of-parsing-languages https://www.masteringemacs.org/article/tree-sitter-complicat...

precompute 4y ago

Awesome! Pretty soon vanilla emacs will be more than enough, won't need melpa packages.

mcqueenjordan 4y ago

I have a huge belief in tree-sitter. I think it's going to continue to grow and become an important tool, especially in security/code tooling contexts.

norir 4y ago
The main innovation of tree-sitter, even more than incremental parsing, as I see it is that it provides a uniform api for traversing a parse tree, which makes it relatively straightforward to onboard a new language to a tool with tree-sitter support. The problem though is that the tree-sitter grammar is nearly always going to be an approximation to the actual language grammar, unless the language compiler/interpreter uses tree-sitter for parsing. To me, this is problematic for tooling because it is always possible for a tree-sitter based tool to be flat out wrong relative to the actual language. For syntax highlighting, this is generally not a huge deal (and tree-sitter does generally work well, though there are exceptions), but I'd be more cautious with security tools based on tree-sitter. If all languages changed their reference parsers to tree-sitter, this would be moot, but that seems unlikely. Language parsers are often optimized beyond what is possible in a general purpose parser generator like tree-sitter and/or have ambiguities that cannot be resolved with the tree-sitter dsl. What feels perhaps likely in the future is that a standard parse tree api emerges, analogous to lsp, and then language parsers could emit trees traversable by this api. Maybe it's just the tree-sitter c api with an alternate front end? Hard to say, but I suspect either something better than (but likely at least partially inspired by) tree-sitter will emerge or we will get stuck in a local minimum with tooling based on slightly incorrect language parsers.
- debugnik 4y ago
  > unless the language compiler/interpreter uses tree-sitter for parsing Doubtful, last time I tried tree-sitter would parse invalid inputs without even tagging any errors in the parse tree. For example, it would silently accept extra tokens, or keywords in the place of identifiers. Replacing the built-in lexer and then validating the parse tree for correctness would be close to writing the grammar twice. And accepting partially correct inputs within the compiler toolchain isn't too hard, so I don't really see the advantage of agreeing on tree-sitter and not just on a parse tree representation that editors can then query, as you then suggested. If the big deal is having it execute client-side or being sandboxed, I feel that's orthogonal to parsing algorithms.

antipaul 4y ago

What's the "explain it like I'm 5 years old" (ELI5) for tree-sitter? Why should I, an emacs user but not lisp hacker, care about it?

chriswarbo 4y ago
tree-sitter creates parsers, e.g. for programming languages, config formats, etc. Emacs modes can use those parsers on buffer contents, e.g. for syntax colouring/highlighting, finding matching delimiters (e.g. moving the cursor over an `if`, and having all the corresponding clauses (e.g. else/elif/fi) highlighted), for contextual editing (e.g. escaping " when inside a string), etc. This can be remarkably tricky to get right; e.g. consider languages which can splice expressions inside strings (which can themselves contain strings, containing spliced expressions, etc.) Using tree-sitter should make this easier and more robust (i.e. less time spent implementing parsers; more time spent implementing features!). I think it would also allow grammars to be re-used across different tools, which should improve support for obscure/niche languages.
- 2pEXgD0fZ5cF 4y ago
  Does this mean that every emacs language package would automatically make use of this once it is built in. Or will this rather enable the possibility to write/rewrite programming language modes so they make use of tree-sitter because they can assume it is available in the default emacs install from then on?
  - omnicognate 4y ago
    It needs to be explicitly used. As far as I'm aware it doesn't slot in behind an existing API and magically make things better.
    - 2pEXgD0fZ5cF 4y ago
      Got it. Are there any beginner guides yet on how to write an emacs (language) package while making use of it?

Ian_Macharia 4y ago

I use tree-sitter in neovim and the syntax highlighting is on par with VSCode

mickeyp 4y ago

sph 4y ago
There is not an Emacs topic I'd like to know more about haven't already covered on your website. Thanks, your articles and your book are the best guides into the world of Emacs.
- mickeyp 4y ago
  Thank you :) I'm glad you like my site and my book!
afry1 4y ago
That point you make about syntax highlighting being slow while using eglot/LSP-mode is a great one. I've been a bit underwhelmed with eglot, and I think that must be the reason: it feels like I'm programming in a bowl of oatmeal with every keystroke. Do you have any tips or guides for using treesitter for syntax highlighting/structural editing and eglot/LSP-mode for everything else?
- omnicognate 4y ago
  AFAIK eglot/lsp-mode don't do syntax highlighting. The article's just explaining why that is (i.e. because it would be too slow). If you don't have tree-sitter your syntax highlighting will be done by the regex based font-lock-mode. I don't think eglot/lsp-mode make that slower, and I believe tree-sitter should speed it up (and make it more correct) without affecting them. I haven't tried it yet, though.
  - TeMPOraL 4y ago
    It must be a matter of configuration. At work, I use buffer re-fontification as an indicator that clangd correctly processed the C++ source file I just opened. That's with an Emacs built from source ~half a year ago + LSP mode.

erganemic 4y ago

I'm really impressed with the strides Emacs has made recently: native compilation, project.el, eglot, and now tree-sitter? As a user who hadn't kept up with development news until recently, I'd always mentally sorted Emacs into the same taxonomy as stuff like `find`: old, powerful, with a clunky interface and a stodgy resistance to updating how it does things (though not without reason). I'm increasingly feeling like that's an unfair classification on my part--I'm genuinely super excited to see where Emacs is in 5 years.

sph 4y ago
Yes, it feels there is a lot of momentum going on recently. Both neovim and Emacs are being improved at breakneck pace, and it is quite incredible for such an old piece of software with, dare I say, a quirky contribution model. The maintainers are working really hard on keeping it current and competitive.
bloopernova 4y ago
I'm really hoping that Emacs becomes multithreaded somehow. Or at least improves some operations so that they're non-blocking. I've been using Emacs primarily for org-mode/roam/babel for a few years now. I'm very glad for its existence, I really think I've become a more effective DevOps person because of it.
- wyuenho 4y ago
  I'll be entirely satisfied with a process/event queue/loop that we can submit tasks to like Javascript's. There is already a command loop in Emacs, we just can't use it for anything other than input events and commands. Once we have an good event loop, we can build a state machine like Redux on it, then we can start rebuilding the display machinery, then we can start deleting all those hooks that constantly interfere with each other...
  - ywei3410 4y ago
    It already has a process/event queue like JS since Emacs 26. What did you mean in particular?

jamborine 4y ago

I'm so impressed by this

arc-in-space 4y ago

Interesting. Current syntax highlighting in emacs is mostly fine, except for how it occasionally blows up - an unterminated quote, in some languages, can run out and match the entire tail as a string, potentially freezing on a large file. Paredit(♡) avoids this by not even letting you do that unless you ask very nicely. I wonder if tree-sitter helps there.

clircle 4y ago
Maybe you and i are using different major modes, but i see the warts of regex high lighting in eMacs. It’s quite bad in auctex

davidkunz 4y ago

Congratulations, Emacs! I hope it will be a similar success story as in Neovim. If more systems use it, the question "should my programming language provide a Tree-Sitter parser" becomes a no brainer.

aidenn0 4y ago

Incremental parsing of incorrect code is one of those things that is literally impossible in the general case, but tree-sitter has found a lot of good ways to do it that are not just possible for a large fraction of reality, but also performant. It's hard to understate how impressive a piece of engineering this is.

Jenk 4y ago
I think you mean "hard to overstate" :)
- aidenn0 4y ago
  Indeed. Too late to edit now though.

Wonnk13 4y ago

So if I have a fairly unremarkable setup with LSP to give me completions, what do I get by fooling around with tree-sitter. It seems like this is more geared toward building an AST, so I'm not sure how it would present itself to the end user currently?

omnicognate 4y ago
Faster and more correct syntax highlighting is the main benefit atm, as I understand it. In general it's for functionality that needs to understand syntax but doesn't need a full compilation-level understanding of the code, that can benefit from much faster responses than an LSP server can provide and that people may want working out of the box for many languages without having to install and configure language servers, generate compilation DBs, etc.
- hvis 4y ago
  Also: indentation, beginning-of-defun/end-of-defun navigation, and probably Imenu later. All the basics of major modes that up until now have been implemented using bespoke algorithms, rewritten ad-hoc for each language.
  - srcreigh 4y ago
    For easy thread reading. Imenu is a way to jump around in the file M-x imenu gives menu of different types of locations (ex functions), you select functions, then you can select the function to jump to.
josteink 4y ago
I think at this point it’s a new building primitive mostly aimed at major-mode authors. That said, tree-sitter should make it possible to create paredit-like implementations for languages not LISP and other stuff like that, which IMO could turn out to be really neat. As a change, this is quite significant, but not directly aimed at end-users.
- 4y ago

jpe90 4y ago

Has anyone been able to replicate vscode/coc.nvim’s instant lsp autocompletion on each keystroke with emacs when eglot/company are enabled? I tried configuring company to lower the idle delay to 0 and set the minimum prefix to 0, but the gui locks up on every keystroke