3 ms·

While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release (0.5). Tree-sitter - A New Parsing System

While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release (0.5). Tree-sitter - A New Parsing System For Programming Tools "Developer tools that support multiple programming languages generally have very limited, regex-based code-analysis capabilities. Tree-sitter is a new parsing system that aims to change this paradigm. It provides a uniform C API for parsing an ever-growing set of languages. It features high-performance incremental parsing and robust error recovery, which allow it to be used to parse code in real-time in a text editor. We're in the process of integrating Tree-sitter into both GitHub.com and the Atom text editor, which will allow us to analyze code accurately and efficiently, paving the way for better syntax highlighting, code navigation, and refactoring." -- Here's an interesting talk about it: https://www.thestrangeloop.com/2018/tree-sitter---a-new-parsing-system-for-programming-tools.html https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars... ¹ http://tree-sitter.github.io/tree-sitter/ http://tree-sitter.github.io/tree-sitter/ ² https://github.com/neovim/neovim/pull/10124 https://github.com/neovim/neovim/pull/10124

rjzzleep 7y ago
Oh! This looks great. Regex parser can have nasty side effects. But for C++ it still seems like something libclang based is still better right? Edit: I see there is a discussion on asynchronous highlighting work, what's the status on that? https://github.com/jeaye/color_coded https://github.com/jeaye/color_coded
coldtea 7y ago
>While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release Well, one is an existing syntax highlighter one can install and use now, the other is something new, we haven't seen, that might or might not come with the next release. Does anybody use it atm on any editor?
- jchw 7y ago
  Tree sitter? It can be used with an extension in VS Code: https://marketplace.visualstudio.com/items?itemName=georgewfraser.vscode-tree-sitter https://marketplace.visualstudio.com/items?itemName=georgewf... It works just fine, although it is not a massive improvement by any means, from the dev experience we have today. I do not know if any editors are shipping with it yet.
  - Liquid_Fire 7y ago
    Presumably also Atom and GitHub's online editor, for which Tree-sitter was written.
- minxomat 7y ago
  Tree sitter is used in Atom and is replacing Textmate grammars. In fact, tree sitter recently got s-expression query support, which will replace the old CSS selector syntax. So atom is already in the second generation of tree sitter support. GitHub semantic (go to definition, call graph) is powered by tree sitter wrapped in an Haskell analyzer. The next version of Codemirror is using Lezer, a JS port of tree sitter. I use tree sitter in a static analysis tool that runs on our internal repos (about 4000).
ashkankiani 7y ago
I mean, to be fair, this only took me a few hours to write, and most of it was spent on the README/docs/demo, so it's not a huge deal. I do wish I had done it sooner, considering how long the Lua integration has been in Neovim. Even the trie implementation was rather easy since I used a naive algorithm (but it was fun to flex my ffi). I think treesitter would be overkill for this, however, considering the size and ffi overhead involved and any speed improvement would be very minimal while increasing the complexity. Considering that we are only told what "lines" changed by neovim right now, it would be very difficult to improve on what I have right now. I had seen mention of treesitter in the code, and I had no idea what it was, so thank you very much for enlightening me. I have a decent amount of experience writing pratt parsers and DFAs by hand, which will probably still be faster, but the incremental algorithm and other advanced features wouldn't come for free. I had been looking for more advanced reading material on parsers (I was even thinking of emailing Donald Knuth to see if I could preview chapter 7's section on parsing), and also for parsers in some of those languages, so this is very nicely timed. I'm definitely going to be reading their papers and watching the talk.

rjzzleep 7y ago

Oh! This looks great. Regex parser can have nasty side effects. But for C++ it still seems like something libclang based is still better right? Edit: I see there is a discussion on asynchronous highlighting work, what's the status on that? https://github.com/jeaye/color_coded https://github.com/jeaye/color_coded

coldtea 7y ago

>While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release Well, one is an existing syntax highlighter one can install and use now, the other is something new, we haven't seen, that might or might not come with the next release. Does anybody use it atm on any editor?

jchw 7y ago
Tree sitter? It can be used with an extension in VS Code: https://marketplace.visualstudio.com/items?itemName=georgewfraser.vscode-tree-sitter https://marketplace.visualstudio.com/items?itemName=georgewf... It works just fine, although it is not a massive improvement by any means, from the dev experience we have today. I do not know if any editors are shipping with it yet.
- Liquid_Fire 7y ago
  Presumably also Atom and GitHub's online editor, for which Tree-sitter was written.
minxomat 7y ago
Tree sitter is used in Atom and is replacing Textmate grammars. In fact, tree sitter recently got s-expression query support, which will replace the old CSS selector syntax. So atom is already in the second generation of tree sitter support. GitHub semantic (go to definition, call graph) is powered by tree sitter wrapped in an Haskell analyzer. The next version of Codemirror is using Lezer, a JS port of tree sitter. I use tree sitter in a static analysis tool that runs on our internal repos (about 4000).

ashkankiani 7y ago

I mean, to be fair, this only took me a few hours to write, and most of it was spent on the README/docs/demo, so it's not a huge deal. I do wish I had done it sooner, considering how long the Lua integration has been in Neovim. Even the trie implementation was rather easy since I used a naive algorithm (but it was fun to flex my ffi). I think treesitter would be overkill for this, however, considering the size and ffi overhead involved and any speed improvement would be very minimal while increasing the complexity. Considering that we are only told what "lines" changed by neovim right now, it would be very difficult to improve on what I have right now. I had seen mention of treesitter in the code, and I had no idea what it was, so thank you very much for enlightening me. I have a decent amount of experience writing pratt parsers and DFAs by hand, which will probably still be faster, but the incremental algorithm and other advanced features wouldn't come for free. I had been looking for more advanced reading material on parsers (I was even thinking of emailing Donald Knuth to see if I could preview chapter 7's section on parsing), and also for parsers in some of those languages, so this is very nicely timed. I'm definitely going to be reading their papers and watching the talk.