6 ms·

Atom understands your code better than ever before

46 points by guessmyname 8y ago

maxbrunsfeld 8y ago
Hello! I worked on this feature and wrote the post. I'd love to hear any feedback or answer any questions.
- Matthias247 8y ago
  Have you thought about a pure Javascript implementation of the parser, or compiling the runtime to WASM? Since a parser shouldn't really use any platform features it should be possible - and it could enable using the technology in pure browser based editors (like Monaco, CodeMirror, etc).
  - maxbrunsfeld 8y ago
    Yes! I'm very interested in making the Tree-sitter runtime and all of the parsers available as pre-built WASM modules. I think it should be straightforward to do.
- smg 8y ago
  How closely coupled is treesitter to Atom? How difficult would it be to generate binding to treesitter for languages other than Javascript? I dream of a future where elisp bindings for treesitter power syntax highlighting in Emacs.
WorldMaker 8y ago
It seems interesting to compare/contrast this Tree-Sitter toolkit/approach with the Language Server Protocol [1] work that VSCode has been taking. My gut impression is that the VSCode assumption that a language's own compiler is often the best source of truth regarding that language, and using an information "onion" of fast "traditional" syntax highlighter augmented with the ability to request a more detailed version from a language compiler/interpreter is a better approach for long-term maintenance reasons, if nothing else. Admittedly, Tree-sitter has the opportunity to centralize some of the work in incremental parsing that LSP pushes to individual language tooling, but arguably in the modern world of watch-based programming most language tooling is going to feel a need to build it regardless. The maintenance headache, though, is that Tree-sitter has its own custom grammar DSL, which likely won't look anything like or share anything with the native grammars of the languages it is parsing, and which ultimately has to settle for a lowest-common denominator approach to grammar parsing. Even if it is more sophisticated than so-called "crude" RegEx-based syntax highlighters, it can still at best be a separately maintained emulation of a language's grammar versus the LSP encouraging a language to provide its own intelligence presumably from the very same parser it uses to get work done. [1] https://docs.microsoft.com/en-us/visualstudio/extensibility/language-server-protocol?view=vs-2017 https://docs.microsoft.com/en-us/visualstudio/extensibility/...
- maxbrunsfeld 8y ago
  Tree-sitter isn't really an alternative to LSP. We think of it as solving a different set of problems. LSP is probably the best way to provide the fixed set of classic IDE features that LSP does - inline diagnostics, autocomplete, go-to definition, etc. We think that for many other features, Tree-sitter is a much cleaner solution than pushing the logic out into the language server. For example, if you want accurate syntax highlighting that's lightweight and updates immediately (as opposed to in a delayed fashion like in most IDEs), you need incremental parsing. Incremental parsing is a highly specialized problem, so writing an incremental parser in the form of a Tree-sitter grammar is much simpler than modifying a language's compiler toolchain in order to make it incremental, and bundling that whole toolchain into an app like Atom. The same goes for other features besides syntax highlighting. The syntax tree is now available via a uniform, in-process API in Atom, and is always up-to-date, so you can script the editor to manipulate your code intelligently. I'm not totally sure what kinds of things we'll end up building on top of that, but I think it's a different set of things than LSP will end up providing.
  - specialist 8y ago
    Watched your StrangeLoop presentation. Nicely done. Am now eager to play with Atom and so forth. I'm a layperson, so please forgive the noob question: I've written some LL(k) grammars (using ANTLR). I really wanted to have incremental parsing, for a two-way structured editor I envisioned for my UI DSL. But I could never wrap my head around either LR or Wagner et al's paper, much less how Wagner's algo could apply to LL grammars. Is there hope for us ANTLR people to get something like tree-sitter? (I ask tparr every few years, to see if someone smarter than me has tried. No joy. Yet.)
    - maxbrunsfeld 8y ago
      I am not super familiar with all of the variations on LL(k) parsing, but my understanding is that the reliance on multiple tokens of lookahead makes incremental parsing a bit more complex, whereas it fits very cleanly into the framework of LR(1)/GLR parsing. My hope is to make parser development with Tree-sitter easy enough that you wouldn't have to pave a new path for your structured editor (unless you wanted to) - you could just create a Tree-sitter grammar for your DSL and implement your custom editing logic as an Atom package. I have tried to make Tree-sitter more approachable than, say, Bison in a several ways. Usually, once a language's basic structure is in place, the process of adding a new feature is pretty declarative and easy. But it's still true that in the course of developing a grammar from scratch, some understanding of LR is important.
tnolet 8y ago
If this is better than the Jetbrains' IDE's I'll eat a pony. I'm a WebStorm and IntelliJ user, with Sublime Text for simple stuff. Used VsCode and Atom once or twice. Ditched them because highlighting and code completion was below par.
- vegasdew 8y ago
  Can you add details, for which languages and when did you try vscode?
  - pcr0 8y ago
    Not OP, but I've had the same experience with Ruby on Rails. I switched to RubyMine last month after using VSCode for 2 months. I don't blame VSCode, Ruby has got to be the hardest language to do static analysis for, but JetBrain's proprietary code inspection and IDE integration blows every other open source solution out of the water.
  - tnolet 8y ago
    I mostly do JavaScript and Go. But also Scala and Python somewhat longer ago.
  - a9entroy 8y ago
    I mostly code in Python. Debugging on VSCode is a nightmare. Especially if you're debugging large matrices in numpy or PyTorch. VSCode just hangs if you try to evaluate large matrices. PyCharm on the other hand works like a breeze. Not to mention the PyCharm debugger is so much better than VSCode.
- atom_arranger 8y ago
  I find WebStorm much more productive than VSCode. Drag and drop files, refactoring actions, extract exports to another file, etc. I'd like to eventually use VSCode, but so far they just don't have the same refactoring capabilities.
  - jacobr 8y ago
dpkonofa 8y ago
Does this require an extension on the file? One of my pet peeves with Sublime is that every package for beautification, syntax highlighting, and code-folding requires whatever I'm working on to be saved to a file using a specific extension to enable those features. Sometimes, I just need to copy and paste a few blocks of code and format, re-indent or any number of things but I don't necessarily want it saved to disk. I know this is a really niche situation but Atom is looking more and more promising to me and this edge case would help make that switch totally justifiable.
- gmemstr 8y ago
  I can't speak for the other two but I know syntax highlighting works fine if you bring up the command line wihtin Sublime (CTRL+SHIFT+C iirc) and then type in the language you want to highlight for - it should give you an option to switch over. I use this all the time at work - Sublime is basically my notepad
- Ajedi32 8y ago
  It goes by either file extension or the hashbang line. But I think it only make the determination when you open the file; it doesn't change the language when you edit the text. You can manually set the language pretty easily though via the command palette or by clicking the language in the bottom-right.
partycoder 8y ago
Atom is slow even on a fast computer.
finchisko 8y ago
Bit offtopic. But what is the future for Atom in hands of Microsoft? Will they continue to develop it or cease development in favor of VSCode? Or merge those two?

46 points by guessmyname 8y ago

maxbrunsfeld 8y ago

Hello! I worked on this feature and wrote the post. I'd love to hear any feedback or answer any questions.

Matthias247 8y ago
Have you thought about a pure Javascript implementation of the parser, or compiling the runtime to WASM? Since a parser shouldn't really use any platform features it should be possible - and it could enable using the technology in pure browser based editors (like Monaco, CodeMirror, etc).
- maxbrunsfeld 8y ago
  Yes! I'm very interested in making the Tree-sitter runtime and all of the parsers available as pre-built WASM modules. I think it should be straightforward to do.
smg 8y ago
How closely coupled is treesitter to Atom? How difficult would it be to generate binding to treesitter for languages other than Javascript? I dream of a future where elisp bindings for treesitter power syntax highlighting in Emacs.

WorldMaker 8y ago

It seems interesting to compare/contrast this Tree-Sitter toolkit/approach with the Language Server Protocol [1] work that VSCode has been taking. My gut impression is that the VSCode assumption that a language's own compiler is often the best source of truth regarding that language, and using an information "onion" of fast "traditional" syntax highlighter augmented with the ability to request a more detailed version from a language compiler/interpreter is a better approach for long-term maintenance reasons, if nothing else. Admittedly, Tree-sitter has the opportunity to centralize some of the work in incremental parsing that LSP pushes to individual language tooling, but arguably in the modern world of watch-based programming most language tooling is going to feel a need to build it regardless. The maintenance headache, though, is that Tree-sitter has its own custom grammar DSL, which likely won't look anything like or share anything with the native grammars of the languages it is parsing, and which ultimately has to settle for a lowest-common denominator approach to grammar parsing. Even if it is more sophisticated than so-called "crude" RegEx-based syntax highlighters, it can still at best be a separately maintained emulation of a language's grammar versus the LSP encouraging a language to provide its own intelligence presumably from the very same parser it uses to get work done. [1] https://docs.microsoft.com/en-us/visualstudio/extensibility/language-server-protocol?view=vs-2017 https://docs.microsoft.com/en-us/visualstudio/extensibility/...

maxbrunsfeld 8y ago
Tree-sitter isn't really an alternative to LSP. We think of it as solving a different set of problems. LSP is probably the best way to provide the fixed set of classic IDE features that LSP does - inline diagnostics, autocomplete, go-to definition, etc. We think that for many other features, Tree-sitter is a much cleaner solution than pushing the logic out into the language server. For example, if you want accurate syntax highlighting that's lightweight and updates immediately (as opposed to in a delayed fashion like in most IDEs), you need incremental parsing. Incremental parsing is a highly specialized problem, so writing an incremental parser in the form of a Tree-sitter grammar is much simpler than modifying a language's compiler toolchain in order to make it incremental, and bundling that whole toolchain into an app like Atom. The same goes for other features besides syntax highlighting. The syntax tree is now available via a uniform, in-process API in Atom, and is always up-to-date, so you can script the editor to manipulate your code intelligently. I'm not totally sure what kinds of things we'll end up building on top of that, but I think it's a different set of things than LSP will end up providing.
- specialist 8y ago
  Watched your StrangeLoop presentation. Nicely done. Am now eager to play with Atom and so forth. I'm a layperson, so please forgive the noob question: I've written some LL(k) grammars (using ANTLR). I really wanted to have incremental parsing, for a two-way structured editor I envisioned for my UI DSL. But I could never wrap my head around either LR or Wagner et al's paper, much less how Wagner's algo could apply to LL grammars. Is there hope for us ANTLR people to get something like tree-sitter? (I ask tparr every few years, to see if someone smarter than me has tried. No joy. Yet.)
  - maxbrunsfeld 8y ago
    I am not super familiar with all of the variations on LL(k) parsing, but my understanding is that the reliance on multiple tokens of lookahead makes incremental parsing a bit more complex, whereas it fits very cleanly into the framework of LR(1)/GLR parsing. My hope is to make parser development with Tree-sitter easy enough that you wouldn't have to pave a new path for your structured editor (unless you wanted to) - you could just create a Tree-sitter grammar for your DSL and implement your custom editing logic as an Atom package. I have tried to make Tree-sitter more approachable than, say, Bison in a several ways. Usually, once a language's basic structure is in place, the process of adding a new feature is pretty declarative and easy. But it's still true that in the course of developing a grammar from scratch, some understanding of LR is important.

tnolet 8y ago

If this is better than the Jetbrains' IDE's I'll eat a pony. I'm a WebStorm and IntelliJ user, with Sublime Text for simple stuff. Used VsCode and Atom once or twice. Ditched them because highlighting and code completion was below par.

vegasdew 8y ago
Can you add details, for which languages and when did you try vscode?
- pcr0 8y ago
  Not OP, but I've had the same experience with Ruby on Rails. I switched to RubyMine last month after using VSCode for 2 months. I don't blame VSCode, Ruby has got to be the hardest language to do static analysis for, but JetBrain's proprietary code inspection and IDE integration blows every other open source solution out of the water.
- tnolet 8y ago
  I mostly do JavaScript and Go. But also Scala and Python somewhat longer ago.
- a9entroy 8y ago
  I mostly code in Python. Debugging on VSCode is a nightmare. Especially if you're debugging large matrices in numpy or PyTorch. VSCode just hangs if you try to evaluate large matrices. PyCharm on the other hand works like a breeze. Not to mention the PyCharm debugger is so much better than VSCode.
atom_arranger 8y ago
I find WebStorm much more productive than VSCode. Drag and drop files, refactoring actions, extract exports to another file, etc. I'd like to eventually use VSCode, but so far they just don't have the same refactoring capabilities.
- jacobr 8y ago

dpkonofa 8y ago

Does this require an extension on the file? One of my pet peeves with Sublime is that every package for beautification, syntax highlighting, and code-folding requires whatever I'm working on to be saved to a file using a specific extension to enable those features. Sometimes, I just need to copy and paste a few blocks of code and format, re-indent or any number of things but I don't necessarily want it saved to disk. I know this is a really niche situation but Atom is looking more and more promising to me and this edge case would help make that switch totally justifiable.

gmemstr 8y ago
I can't speak for the other two but I know syntax highlighting works fine if you bring up the command line wihtin Sublime (CTRL+SHIFT+C iirc) and then type in the language you want to highlight for - it should give you an option to switch over. I use this all the time at work - Sublime is basically my notepad
Ajedi32 8y ago
It goes by either file extension or the hashbang line. But I think it only make the determination when you open the file; it doesn't change the language when you edit the text. You can manually set the language pretty easily though via the command palette or by clicking the language in the bottom-right.

partycoder 8y ago

Atom is slow even on a fast computer.

finchisko 8y ago

Bit offtopic. But what is the future for Atom in hands of Microsoft? Will they continue to develop it or cease development in favor of VSCode? Or merge those two?