12 ms·

Tree Sitter and the Complications of Parsing Languages

225 points by podiki 5y ago

grenoire 5y ago
I am in love with language servers, the quality of life improvement is just unreal.
- 5e92cb50239222b 5y ago
  Wait until you try a fully featured "real" IDE. The features language servers provide are only some of the many things that IDE users have had for literally decades.
  - grenoire 5y ago
    I'm old enough to have used IDEs, the issue is that my job involves dealing with multiple different languages and markup files. In turn, a general purpose editor with language servers just suits my workflow better.
    - smw 5y ago
      Try a jetbrains ide? They handle pretty much any language you can think of, including fun things like lua with ERB substitutions.
      - square_usual 5y ago
        Yeah, IntelliJ Ultimate with plugins gets pretty close to VSCode in terms of language support.
    - riffraff 5y ago
      surely any decent IDE will handle that nicely? I used to work with Eclipse and it supported everything* through plugins just nicely. * ..that I was using at the time: java, xml, python, html, jsp, javascript
  - Lio 5y ago
    Could you provide some concrete examples? I ask because modern editors can do most things people often regard as IDE only but there is still the odd gem that’s worth hearing about.
    - sa46 5y ago
      I'm not familiar with state of the art for language servers but here's common IntelliJ refactors I use across Go and TypeScript (and Java a while ago): - Add a parameter to a method signature and fill it in with a default value. - Reorder method parameters. - Extract a block of code to a method that infers the correct input and output types. The most advanced refactoring I've done with IntelliJ is structural replace for Java which can do something like: for every private method matching the name "*DoThing" defined on a class that extends Foo, insert a log line at the beginning of the method: https://www.jetbrains.com/help/idea/structural-search-and-replace.html https://www.jetbrains.com/help/idea/structural-search-and-re... I make heavy use of the "integrated" aspect of IntelliJ. One of the nicer benefits is that SQL strings are type-checked against a database schema.
      - bestouff 5y ago
        All of these are doable with LSP. I'm a big fan of the "LSP rename" action which will rename a particular semantic item (e.g. a method) across files, or the refactoring actions (e.g. change an "if let" into a "match" in Rust).
        jen20 5y ago
        Do you have an example of a language server capable of structural refactoring of the type mentioned in the GP? The “semantic rename” is table stakes from ~20 years ago in IntelliJ and ReSharper, and even Eclipse.
        unrealhoang 5y ago
        Structural refactoring as in extract function/variable? You can find that in Rust Analyzer https://rust-analyzer.github.io/manual.html#assists-code-actions https://rust-analyzer.github.io/manual.html#assists-code-act...
  - aidos 5y ago
    Sure, but a lot of us don’t like that extra overhead. LSP is great in that it’s a tool you can tap into to use in a workflow that’s best for you. More of a library than an application (not technically, but in terms of how you use it).
  - omnicognate 5y ago
    It's kind of hilarious that programmers, who learn again and again the value of decoupling and cohesion, fell so hard for the idea of an Integrated Development Environment. There's nothing about syntactic/semantic code analysis, to pick one example, that requires it to be packaged along with a particular text editor in a single big blob. Ironically, the most successful IDEs today, the Jetbrains ones, are demonstrations of this. They are built out of reusable components that are combined to produce a wide range of IDEs for different languages. LSP and DAP aren't perfect, but they're a huge step in the right direction. There's no reason people shouldn't be able to use the editor of their choice along with common tooling for different languages. The fact that IDEs had (for a while) better autocomplete, for example, than emacs wasn't because of some inherent advantage an IDE has over an editor. It's because the people that wrote the language analysis tools behind that autocomplete facility deliberately chose to package them in such a way that they could only be used with one blessed editor. It's great to see the fight back against that, and especially so to see Microsoft (of all people) embracing it with LSP, Roslyn, etc.
    - gmueckl 5y ago
      The technical design isn't the user experience. IDEs are an integrated user experience. It literally doesn't matter to the user how nicely decoupled everything is or isn't under the hood if the end results are indistinguishable. One point in favor of tight integration and against LSPs is that editing programs isn't like editing unstructured text at all and shouldn't be presented as such. There are tons of ways in which the IDE UX can be enhanced using syntactic and semantic knowledge of programs. Having a limited and standardized interface between the UI and a module providing semantic information will just hamper such innovation.
      - spinningslate 5y ago
        >Having a limited and standardized interface between the UI and a module providing semantic information will just hamper such innovation. Maybe. That can certainly be a downside of standardisation in general. However, it doesn't necessarily follow in all cases, and this is, I think, one where it doesn't. The features LSP provides are stable - and have been standard across most editors/IDEs for quite some time. Implementing them once for N editors, rather than N times, is just far more approachable (and appealing) for language tooling developers. It doesn't stop those developers (or anyone else) adding features beyond the LSP standard. But that means doing it in an editor-specific way. Which is no worse than where we were before anyway.
  - mickeyp 5y ago
    It's hard for an open source community to build features that compete with a commercial offering like that of Microsoft (or Borland in the 90s.) And the reason for that is mostly down to fragmentation: the vim guys are doing their thing; the Emacs theirs, etc. Now focus that energy into a singular project like a Language Server and the payout is likely to be many orders of magnitude greater.
    - turminal 5y ago
      Language server is a Microsoft offering
      - NateEag 5y ago
        It's an open standard started and maintained by Microsoft. I think that's a bit different from just being a "Microsoft offering".
  - rowanG077 5y ago
    I have not ever had a good experience with an IDE. They are always bloated messes that try to force you into their shitty project structure.
  - tapia 5y ago
    For me, there is no IDE feature that can compete with the experience of editing in vim/neovim. When I use any other editor I just feel like I have a hand tied. The development of LSP and tree-sitter just makes the whole experience even better.
    - nick_ 5y ago
      LSP is just a generalization of the implementations IDEs have had for decades.
  - IshKebab 5y ago
    VSCode is a "real" IDE. > The features language servers provide are only some of the many things that IDE users have had for literally decades. Yes of course, because that's what they were explicitly designed to do. The novel thing about language servers isn't that they enable code intelligence features like auto-complete and variable renaming. It's that they do so over a standard protocol that any editor or IDE (or website or CI system or ...) can use.
  - Epitom3 5y ago
    "Hey I use an IDE, im special"
aidos 5y ago
Before this conversation is railroaded by talk about language servers, as the article points out, tree sitter tends to need to be a bit closer to the environment to be effective. There’s still work to do, but having tree sitter in neovim feels like a great step forward.
- omnicognate 5y ago
  Yes, it's more for syntax highlighting where you don't want the lag of an external server and don't need the deep language analysis needed for diagnostics, refactoring, etc. I'm not sure what other use cases it would be superior to LSP for, but I'm sure there are some.
  - clktmr 5y ago
    It can also be used for text editing, e.g. changing, deleting, swapping of function arguments or any other text object defined by the language syntax.
    - yakubin 5y ago
      Cursor movement as well.
- mickeyp 5y ago
  Author here. Yes, both are very useful, with some overlap of purpose, but work well together.
  - aidos 5y ago
    Thanks for the article. Even though I'm not active on the development side in either editor, I love the idea that people are toiling away on these same sorts of enhancements in both environments (and I get the benefit in neovim).
kieckerjan 5y ago
I suppose that these days I am one of the few professional programmers who has an active dislike of syntax highlighting. I find it immensely distracting. The only stuff I allow the highlighter to touch are my comments (I turn them bold) and I consider this a somewhat frivolous indulgence. (I appreciate the complexity of the problem, btw)
- Derbasti 5y ago
  A few years ago I switched my color theme to something very simple, just as an experiment. Somehow I never found a need to change that. I highlight comments, keywords, and strings. Comment and string highlights are helpful if they contain code-like text, to make them obviously not-code. Keywords give some structure to the text. Everything else is frivolous to me. Books do not highlight verbs in green, either.
  - xyzzy_plugh 5y ago
    > Books do not highlight verbs in green, either. While I will not argue with your general point -- I also don't really need highlighting and I read a lot of plaintext code -- I wonder about this. Would this make languages easier for non-native speakers? Would improve comprehension? It's funny that the industry spends so much time on syntax highlighting for programming languages, when humanity's written languages are arguably more complex and difficult to parse and master.
    - Derbasti 5y ago
      I think the point is that books do highlight things: Headlines, italics, Capitalization. Just not silly technicalities like parts of speech.
    - unhammer 5y ago
      > Would this make languages easier for non-native speakers? Would improve comprehension? When I've been trying to learn languages, I can typically part-of-speech tag unknown words quite easily (common prefixes/suffixes/word length/sentence position give lots of information – and some of this is shared across languages as well). The comprehension difficulty is nearly always due to content words I haven't seen before (or have forgotten).
deriramdani 5y ago
Momok
dgellow 5y ago
Checkout the project page here: https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ Quite a lot of languages are already supported, it's really nice to see. I might have a use for such a library for a personal project :) You can play around with the playground here: https://tree-sitter.github.io/tree-sitter/playground https://tree-sitter.github.io/tree-sitter/playground
rcshubhadeep 5y ago
tree-sitter is a great framework. I have used it quite a bit in past. I even created a small library on top of it, called tree-hugger (https://github.com/autosoft-dev/tree-hugger https://github.com/autosoft-dev/tree-hugger) Really enjoyed their playground as well.
matklad 5y ago
> Well, because it’s gosh-darn hard to do it the right way. I think this overstates the difficulty. This of course depends a lot on the language, but for a reasonable one (not C++) you can just go and write the parser by hand. I’d ballpark this as three weeks, if you know how the thing is supposed to work. > it doesn’t have to redo the whole thing on every keypress. This is probably what makes the task seem harder than it is. Incremental parsing is nice, but not mandatory. rust-analyzer and most IntelliJ parsers re-parse the whole file on every change (IJ does incremental lexing, which is simple). > The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance. I am surprised to hear that. We never had performance problems with highlighting on the server in rust-analyzer. I remember that for Emacs specifically there were client side problems with parsing LSP JSON. > Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated. That’s not the bottleneck for syntax highlighting, typechecking is (and it’s typechecking that makes highlighting especially interesting). In general, my perception of what’s going on with proper parsing in the industry is a bit different. I’d say status quo from five years back boils down to people just getting accustomed to the way things were done. Compiler authors generally didn’t think about syntax highlighting or completions, and editors generally didn’t want to do the parsing stuff. JetBrains were the exception, as they just did the thing. In this sense, LSP was a much-needed stimulus to just start doing things properly. People were building rich IDE experiences before LSP just fine (see dart analyzer), it’s just that relatively few languages saw it as an important problem to solve at all.
- bsder 5y ago
  > I think this overstates the difficulty. This of course depends a lot on the language, but for a reasonable one (not C++) you can just go and write the parser by hand. I don't agree. Newer languages are all being designed with the constraint that the grammar should be easy to parse and not require indefinite lookahead and full compilation to get back on track after an error. That's a big change from the C/C++ heritage. It's no coincidence that "modern" languages (call it the last 10 or so years) tend to have things like explicit variable assignment (let-statement-like) and delimiters between variable and type, for example.
0x008 5y ago
Not coming from the vim/eMacs world, I fail to understand what treesitter is compare to a language server? Why would I need both?
- dbalatero 5y ago
  The article talks about why LSP servers don't typically implement syntax highlighting (performance).
  - petepete 5y ago
    Here's a great video on the topic by one of Neovim's core team https://www.youtube.com/watch?v=c17j09vY5sw https://www.youtube.com/watch?v=c17j09vY5sw
    - 0x008 5y ago
      thank's that is very helpful. I was wondering if the both were to achieve similar goals it makes no sense to run them both, but now I can educate myself.
smcameron 5y ago
> Semantic Bovinator Heh. A long time ago I wrote a video game[1] somewhat similar to Williams Defender, and casting about for some sort of "theme" for the game, I hit upon the "editor wars", the ancient storied battle between vi and emacs. You are ostensibly "vi", (a little spaceship vaguely reminiscent of the Vipers from Battlestar Galactica) cruising through system memory, evading system processes, GDB instances, etc trying to recover your ".swp" files. How to represent Emacs? Obviously, via a giant blimp! and I could display all sorts of messages on the side of the blimp, singing the praises of Emacs, and disparaging fans of vi. And the Emacs blimp had a "memory leak", which meant that pieces of the xemacs source code would literally leak out of the back end of the blimp, with the letters floating lazily away, like smoke. So that meant I had to take a look at the xemacs source, dig through it and try to find some funny bits to put in. Of course, "semantic bovinate" jumped out at me.[2] [1] https://github.com/smcameron/wordwarvi https://github.com/smcameron/wordwarvi [2] https://github.com/smcameron/wordwarvi/blob/master/wordwarvi.c#L1719 https://github.com/smcameron/wordwarvi/blob/master/wordwarvi...
- krylon 5y ago
  That is gorgeous! Thanks for sharing!
jicea 5y ago
I'm a maintainer of a cli HTTP client with a text plain file format, Hurl [1]. I would like to begin to add support for various IDE (VSCode, IntelliJ), starting from syntax highlighting, but I have hard time to start. I struggle on many "little" details, for instance: syntax error should be exactly the same in the terminal and in the IDE. Should I reimplement exactly the same parsing or should I reuse some of the cli tools parser? If I reuse it, how do I implement things given than, for instance, IntelliJ plugin are written in Java/Kotlin, while VScode plugin are Javascript/TypeScript, and Hurl is written in Rust... Very hard to figure all when it's not your core domain, [1] https://hurl.dev https://hurl.dev
- billconan 5y ago
  https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/
- IshKebab 5y ago
  If it's something simple (and it sounds like it is) then I would strongly recommend just making a single parser library that you use in both the language server and CLI. That's what I've done for my RPC format. I used Nom. Even though it's not incremental, parsing is easily fast enough to just reparse the entire document on each change. An alternative is to just use Tree Sitter as your parser for the CLI too. You won't use the incremental parsing feature in the CLI but that's fine. Supporting IntelliJ may be tricky but there is a WIP plugin that adds LSP support.
IshKebab 5y ago
> The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance. Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated. Repeat that up to 100 words per minute (or whatever your typing speed is) and you’re looking at a lot of cross-chatter that is just better suited for in-process communication. While I agree... he might be surprised to know that that is what all language servers do anyway, even if they don't provide syntax highlighting. Every keystroke gets sent over the LSP. As JSON. It's amazing it works as well as it does.

225 points by podiki 5y ago

grenoire 5y ago

I am in love with language servers, the quality of life improvement is just unreal.

5e92cb50239222b 5y ago
Wait until you try a fully featured "real" IDE. The features language servers provide are only some of the many things that IDE users have had for literally decades.
- grenoire 5y ago
  I'm old enough to have used IDEs, the issue is that my job involves dealing with multiple different languages and markup files. In turn, a general purpose editor with language servers just suits my workflow better.
  - smw 5y ago
    Try a jetbrains ide? They handle pretty much any language you can think of, including fun things like lua with ERB substitutions.
    - square_usual 5y ago
      Yeah, IntelliJ Ultimate with plugins gets pretty close to VSCode in terms of language support.
  - riffraff 5y ago
    surely any decent IDE will handle that nicely? I used to work with Eclipse and it supported everything* through plugins just nicely. * ..that I was using at the time: java, xml, python, html, jsp, javascript
- Lio 5y ago
  Could you provide some concrete examples? I ask because modern editors can do most things people often regard as IDE only but there is still the odd gem that’s worth hearing about.
  - sa46 5y ago
    I'm not familiar with state of the art for language servers but here's common IntelliJ refactors I use across Go and TypeScript (and Java a while ago): - Add a parameter to a method signature and fill it in with a default value. - Reorder method parameters. - Extract a block of code to a method that infers the correct input and output types. The most advanced refactoring I've done with IntelliJ is structural replace for Java which can do something like: for every private method matching the name "*DoThing" defined on a class that extends Foo, insert a log line at the beginning of the method: https://www.jetbrains.com/help/idea/structural-search-and-replace.html https://www.jetbrains.com/help/idea/structural-search-and-re... I make heavy use of the "integrated" aspect of IntelliJ. One of the nicer benefits is that SQL strings are type-checked against a database schema.
    - bestouff 5y ago
      All of these are doable with LSP. I'm a big fan of the "LSP rename" action which will rename a particular semantic item (e.g. a method) across files, or the refactoring actions (e.g. change an "if let" into a "match" in Rust).
      - jen20 5y ago
        Do you have an example of a language server capable of structural refactoring of the type mentioned in the GP? The “semantic rename” is table stakes from ~20 years ago in IntelliJ and ReSharper, and even Eclipse.
        unrealhoang 5y ago
        Structural refactoring as in extract function/variable? You can find that in Rust Analyzer https://rust-analyzer.github.io/manual.html#assists-code-actions https://rust-analyzer.github.io/manual.html#assists-code-act...
- aidos 5y ago
  Sure, but a lot of us don’t like that extra overhead. LSP is great in that it’s a tool you can tap into to use in a workflow that’s best for you. More of a library than an application (not technically, but in terms of how you use it).
- omnicognate 5y ago
  It's kind of hilarious that programmers, who learn again and again the value of decoupling and cohesion, fell so hard for the idea of an Integrated Development Environment. There's nothing about syntactic/semantic code analysis, to pick one example, that requires it to be packaged along with a particular text editor in a single big blob. Ironically, the most successful IDEs today, the Jetbrains ones, are demonstrations of this. They are built out of reusable components that are combined to produce a wide range of IDEs for different languages. LSP and DAP aren't perfect, but they're a huge step in the right direction. There's no reason people shouldn't be able to use the editor of their choice along with common tooling for different languages. The fact that IDEs had (for a while) better autocomplete, for example, than emacs wasn't because of some inherent advantage an IDE has over an editor. It's because the people that wrote the language analysis tools behind that autocomplete facility deliberately chose to package them in such a way that they could only be used with one blessed editor. It's great to see the fight back against that, and especially so to see Microsoft (of all people) embracing it with LSP, Roslyn, etc.
  - gmueckl 5y ago
    The technical design isn't the user experience. IDEs are an integrated user experience. It literally doesn't matter to the user how nicely decoupled everything is or isn't under the hood if the end results are indistinguishable. One point in favor of tight integration and against LSPs is that editing programs isn't like editing unstructured text at all and shouldn't be presented as such. There are tons of ways in which the IDE UX can be enhanced using syntactic and semantic knowledge of programs. Having a limited and standardized interface between the UI and a module providing semantic information will just hamper such innovation.
    - spinningslate 5y ago
      >Having a limited and standardized interface between the UI and a module providing semantic information will just hamper such innovation. Maybe. That can certainly be a downside of standardisation in general. However, it doesn't necessarily follow in all cases, and this is, I think, one where it doesn't. The features LSP provides are stable - and have been standard across most editors/IDEs for quite some time. Implementing them once for N editors, rather than N times, is just far more approachable (and appealing) for language tooling developers. It doesn't stop those developers (or anyone else) adding features beyond the LSP standard. But that means doing it in an editor-specific way. Which is no worse than where we were before anyway.
- mickeyp 5y ago
  It's hard for an open source community to build features that compete with a commercial offering like that of Microsoft (or Borland in the 90s.) And the reason for that is mostly down to fragmentation: the vim guys are doing their thing; the Emacs theirs, etc. Now focus that energy into a singular project like a Language Server and the payout is likely to be many orders of magnitude greater.
  - turminal 5y ago
    Language server is a Microsoft offering
    - NateEag 5y ago
      It's an open standard started and maintained by Microsoft. I think that's a bit different from just being a "Microsoft offering".
- rowanG077 5y ago
  I have not ever had a good experience with an IDE. They are always bloated messes that try to force you into their shitty project structure.
- tapia 5y ago
  For me, there is no IDE feature that can compete with the experience of editing in vim/neovim. When I use any other editor I just feel like I have a hand tied. The development of LSP and tree-sitter just makes the whole experience even better.
  - nick_ 5y ago
    LSP is just a generalization of the implementations IDEs have had for decades.
- IshKebab 5y ago
  VSCode is a "real" IDE. > The features language servers provide are only some of the many things that IDE users have had for literally decades. Yes of course, because that's what they were explicitly designed to do. The novel thing about language servers isn't that they enable code intelligence features like auto-complete and variable renaming. It's that they do so over a standard protocol that any editor or IDE (or website or CI system or ...) can use.
- Epitom3 5y ago
  "Hey I use an IDE, im special"

aidos 5y ago

Before this conversation is railroaded by talk about language servers, as the article points out, tree sitter tends to need to be a bit closer to the environment to be effective. There’s still work to do, but having tree sitter in neovim feels like a great step forward.

omnicognate 5y ago
Yes, it's more for syntax highlighting where you don't want the lag of an external server and don't need the deep language analysis needed for diagnostics, refactoring, etc. I'm not sure what other use cases it would be superior to LSP for, but I'm sure there are some.
- clktmr 5y ago
  It can also be used for text editing, e.g. changing, deleting, swapping of function arguments or any other text object defined by the language syntax.
  - yakubin 5y ago
    Cursor movement as well.
mickeyp 5y ago
Author here. Yes, both are very useful, with some overlap of purpose, but work well together.
- aidos 5y ago
  Thanks for the article. Even though I'm not active on the development side in either editor, I love the idea that people are toiling away on these same sorts of enhancements in both environments (and I get the benefit in neovim).

kieckerjan 5y ago

I suppose that these days I am one of the few professional programmers who has an active dislike of syntax highlighting. I find it immensely distracting. The only stuff I allow the highlighter to touch are my comments (I turn them bold) and I consider this a somewhat frivolous indulgence. (I appreciate the complexity of the problem, btw)

Derbasti 5y ago
A few years ago I switched my color theme to something very simple, just as an experiment. Somehow I never found a need to change that. I highlight comments, keywords, and strings. Comment and string highlights are helpful if they contain code-like text, to make them obviously not-code. Keywords give some structure to the text. Everything else is frivolous to me. Books do not highlight verbs in green, either.
- xyzzy_plugh 5y ago
  > Books do not highlight verbs in green, either. While I will not argue with your general point -- I also don't really need highlighting and I read a lot of plaintext code -- I wonder about this. Would this make languages easier for non-native speakers? Would improve comprehension? It's funny that the industry spends so much time on syntax highlighting for programming languages, when humanity's written languages are arguably more complex and difficult to parse and master.
  - Derbasti 5y ago
    I think the point is that books do highlight things: Headlines, italics, Capitalization. Just not silly technicalities like parts of speech.
  - unhammer 5y ago
    > Would this make languages easier for non-native speakers? Would improve comprehension? When I've been trying to learn languages, I can typically part-of-speech tag unknown words quite easily (common prefixes/suffixes/word length/sentence position give lots of information – and some of this is shared across languages as well). The comprehension difficulty is nearly always due to content words I haven't seen before (or have forgotten).

deriramdani 5y ago

Momok

dgellow 5y ago

Checkout the project page here: https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ Quite a lot of languages are already supported, it's really nice to see. I might have a use for such a library for a personal project :) You can play around with the playground here: https://tree-sitter.github.io/tree-sitter/playground https://tree-sitter.github.io/tree-sitter/playground

rcshubhadeep 5y ago

tree-sitter is a great framework. I have used it quite a bit in past. I even created a small library on top of it, called tree-hugger (https://github.com/autosoft-dev/tree-hugger https://github.com/autosoft-dev/tree-hugger) Really enjoyed their playground as well.

matklad 5y ago

> Well, because it’s gosh-darn hard to do it the right way. I think this overstates the difficulty. This of course depends a lot on the language, but for a reasonable one (not C++) you can just go and write the parser by hand. I’d ballpark this as three weeks, if you know how the thing is supposed to work. > it doesn’t have to redo the whole thing on every keypress. This is probably what makes the task seem harder than it is. Incremental parsing is nice, but not mandatory. rust-analyzer and most IntelliJ parsers re-parse the whole file on every change (IJ does incremental lexing, which is simple). > The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance. I am surprised to hear that. We never had performance problems with highlighting on the server in rust-analyzer. I remember that for Emacs specifically there were client side problems with parsing LSP JSON. > Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated. That’s not the bottleneck for syntax highlighting, typechecking is (and it’s typechecking that makes highlighting especially interesting). In general, my perception of what’s going on with proper parsing in the industry is a bit different. I’d say status quo from five years back boils down to people just getting accustomed to the way things were done. Compiler authors generally didn’t think about syntax highlighting or completions, and editors generally didn’t want to do the parsing stuff. JetBrains were the exception, as they just did the thing. In this sense, LSP was a much-needed stimulus to just start doing things properly. People were building rich IDE experiences before LSP just fine (see dart analyzer), it’s just that relatively few languages saw it as an important problem to solve at all.

bsder 5y ago
> I think this overstates the difficulty. This of course depends a lot on the language, but for a reasonable one (not C++) you can just go and write the parser by hand. I don't agree. Newer languages are all being designed with the constraint that the grammar should be easy to parse and not require indefinite lookahead and full compilation to get back on track after an error. That's a big change from the C/C++ heritage. It's no coincidence that "modern" languages (call it the last 10 or so years) tend to have things like explicit variable assignment (let-statement-like) and delimiters between variable and type, for example.

0x008 5y ago

Not coming from the vim/eMacs world, I fail to understand what treesitter is compare to a language server? Why would I need both?

dbalatero 5y ago
The article talks about why LSP servers don't typically implement syntax highlighting (performance).
- petepete 5y ago
  Here's a great video on the topic by one of Neovim's core team https://www.youtube.com/watch?v=c17j09vY5sw https://www.youtube.com/watch?v=c17j09vY5sw
  - 0x008 5y ago
    thank's that is very helpful. I was wondering if the both were to achieve similar goals it makes no sense to run them both, but now I can educate myself.

smcameron 5y ago

> Semantic Bovinator Heh. A long time ago I wrote a video game[1] somewhat similar to Williams Defender, and casting about for some sort of "theme" for the game, I hit upon the "editor wars", the ancient storied battle between vi and emacs. You are ostensibly "vi", (a little spaceship vaguely reminiscent of the Vipers from Battlestar Galactica) cruising through system memory, evading system processes, GDB instances, etc trying to recover your ".swp" files. How to represent Emacs? Obviously, via a giant blimp! and I could display all sorts of messages on the side of the blimp, singing the praises of Emacs, and disparaging fans of vi. And the Emacs blimp had a "memory leak", which meant that pieces of the xemacs source code would literally leak out of the back end of the blimp, with the letters floating lazily away, like smoke. So that meant I had to take a look at the xemacs source, dig through it and try to find some funny bits to put in. Of course, "semantic bovinate" jumped out at me.[2] [1] https://github.com/smcameron/wordwarvi https://github.com/smcameron/wordwarvi [2] https://github.com/smcameron/wordwarvi/blob/master/wordwarvi.c#L1719 https://github.com/smcameron/wordwarvi/blob/master/wordwarvi...

krylon 5y ago
That is gorgeous! Thanks for sharing!

jicea 5y ago

I'm a maintainer of a cli HTTP client with a text plain file format, Hurl [1]. I would like to begin to add support for various IDE (VSCode, IntelliJ), starting from syntax highlighting, but I have hard time to start. I struggle on many "little" details, for instance: syntax error should be exactly the same in the terminal and in the IDE. Should I reimplement exactly the same parsing or should I reuse some of the cli tools parser? If I reuse it, how do I implement things given than, for instance, IntelliJ plugin are written in Java/Kotlin, while VScode plugin are Javascript/TypeScript, and Hurl is written in Rust... Very hard to figure all when it's not your core domain, [1] https://hurl.dev https://hurl.dev

billconan 5y ago
https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/
IshKebab 5y ago
If it's something simple (and it sounds like it is) then I would strongly recommend just making a single parser library that you use in both the language server and CLI. That's what I've done for my RPC format. I used Nom. Even though it's not incremental, parsing is easily fast enough to just reparse the entire document on each change. An alternative is to just use Tree Sitter as your parser for the CLI too. You won't use the incremental parsing feature in the CLI but that's fine. Supporting IntelliJ may be tricky but there is a WIP plugin that adds LSP support.

IshKebab 5y ago

> The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance. Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated. Repeat that up to 100 words per minute (or whatever your typing speed is) and you’re looking at a lot of cross-chatter that is just better suited for in-process communication. While I agree... he might be surprised to know that that is what all language servers do anyway, even if they don't provide syntax highlighting. Every keystroke gets sent over the LSP. As JSON. It's amazing it works as well as it does.