6 ms·

The Heart of a Language Server

le-mark 3y ago
> For Rust, a file foo.rs only exists semantically if some parent file includes it via mod foo; declaration! And, in general, it’s impossible to locate the parent file automatically. Usually, for src/bar/foo.rs the parent would be src/bar.rs, but, due to #[path] attributes which override this default, this might not be true. This is where the path based package structure of Java/c# was a huge win from a static analysis perspective.
- jen20 3y ago
  C# does not have a path-based structure.
marwis 3y ago
Since semantic information had to be generated from syntax in the first place, why not retain that linkage somewhere so that it doesn't have to be searched again? Also to find usage of symbols generated by macros, can't one make compiler plugin to dump usage information somewhere (or modify compiler if impossible with plugin)? This might be a bit stale but much better than nothing.
- di4na 3y ago
  Yes and that is what these analyzers do. The problem is that your space grow really fast and that your compilers are really not built to extract that information in a few ms. Even less to regenerate only part of it based on partial input. Even less when the input may not be correct syntax. Also that linkage being kept is exactly what this post talk about. How to keep it intact through the different steps and transformations in your pipeline in a way adapted to the kind of queries you are going to need is... Actually hard and dependent on the query. Which means that adding new features to your IDE would regularly need (and actually does need) a new way to store and query that data. But yes. Reusing part of the rust compiler (or replacing some of them) in rust-analyzer is already something that happens and that maintainers work on. It is just not that easy. But yes, C# and Roslyn in general was built with that in mind. Typescript too.
brabel 3y ago
> The second one is actually less true in Rust than it is in Kotlin or C#! In those languages, each file starts with a package declaration, which immediately mounts the file at the appropriate place in the semantic model. That's the difference between languages designed to be IDE-friendly and languages designed to have flexible semantics without much concern for making analysis easy. Languages like C++, D, Lisp, Rust are terribly difficult to analyse compared to Java, Kotlin and Dart... and that's no accident. Rust only has good support (not great but ok) in IDEs because it managed to get enough people to muster the huge effort that requires. I've been trying the Kate editor (from the KDE project), for example, and it does a pretty good job of helping with Rust thanks to the Rust Analyzer LSP... all Kate had to do was provide a LSP client for that to work. The situation for D, for example, which has not had the same amount of effort (and due to the difficult to analyze nature of the language) is much worse, unfortunately... that's a big reason for me to prefer Rust over D (others being more libraries and much fewer compiler bugs), despite finding D more fun to write.
- globular-toast 3y ago
  Interesting that you put Lisp in difficult category despite the IDE experience being in another league compared to any other language I've tried.
  - brabel 3y ago
    The IDE experience with Lisp is great because of the interactivity, not because its great analysis. You do get good analysis from the language introspection capabilities itself, but you need to "manually" use that... It's just different. I know SLIME and love it, but the analysis it's capable of doing is not near the level you get with IntelliJ and Java or Kotlin. For starter, Common Lisp (and most other Lisps) don't even have static types (I know you can declare types, but most libs don't do it and SLIME doesn't use that for any IDE feature as far as I know, other than perhaps warn when you use an obviously wrong argument type), making features like advanced refactoring very difficult. Macros make it very hard to even know whether the syntax is right (try writing a LOOP in Common Lisp - SLIME won't help you).
cyocum 3y ago
Reading this, I am increasing my belief that this is very, if not exactly, like XSLT. You are searching a tree looking for parents, siblings, children, etc nodes then analysing or transforming them.
- heads 3y ago
  Visiting and transforming trees is indeed a well trodden pattern. Here’s another kind of tree walking which I’ve enjoyed working with, this year: https://libcst.readthedocs.io/en/latest/tutorial.html#Build-Visitor-or-Transformer https://libcst.readthedocs.io/en/latest/tutorial.html#Build-... It is a little less pure than the functional XSLT, and also a lot more practical.
int0x80 3y ago
>In this formulation, a language server needs to just enough analysis to drill down to a specific node. Does rust-analyzer compute semantic information on the fly and incrementally?
- matklad 3y ago
  Yes

> For Rust, a file foo.rs only exists semantically if some parent file includes it via mod foo; declaration! And, in general, it’s impossible to locate the parent file automatically. Usually, for src/bar/foo.rs the parent would be src/bar.rs, but, due to #[path] attributes which override this default, this might not be true. This is where the path based package structure of Java/c# was a huge win from a static analysis perspective.

jen20 3y ago
C# does not have a path-based structure.

marwis 3y ago

Since semantic information had to be generated from syntax in the first place, why not retain that linkage somewhere so that it doesn't have to be searched again? Also to find usage of symbols generated by macros, can't one make compiler plugin to dump usage information somewhere (or modify compiler if impossible with plugin)? This might be a bit stale but much better than nothing.

di4na 3y ago
Yes and that is what these analyzers do. The problem is that your space grow really fast and that your compilers are really not built to extract that information in a few ms. Even less to regenerate only part of it based on partial input. Even less when the input may not be correct syntax. Also that linkage being kept is exactly what this post talk about. How to keep it intact through the different steps and transformations in your pipeline in a way adapted to the kind of queries you are going to need is... Actually hard and dependent on the query. Which means that adding new features to your IDE would regularly need (and actually does need) a new way to store and query that data. But yes. Reusing part of the rust compiler (or replacing some of them) in rust-analyzer is already something that happens and that maintainers work on. It is just not that easy. But yes, C# and Roslyn in general was built with that in mind. Typescript too.

brabel 3y ago

> The second one is actually less true in Rust than it is in Kotlin or C#! In those languages, each file starts with a package declaration, which immediately mounts the file at the appropriate place in the semantic model. That's the difference between languages designed to be IDE-friendly and languages designed to have flexible semantics without much concern for making analysis easy. Languages like C++, D, Lisp, Rust are terribly difficult to analyse compared to Java, Kotlin and Dart... and that's no accident. Rust only has good support (not great but ok) in IDEs because it managed to get enough people to muster the huge effort that requires. I've been trying the Kate editor (from the KDE project), for example, and it does a pretty good job of helping with Rust thanks to the Rust Analyzer LSP... all Kate had to do was provide a LSP client for that to work. The situation for D, for example, which has not had the same amount of effort (and due to the difficult to analyze nature of the language) is much worse, unfortunately... that's a big reason for me to prefer Rust over D (others being more libraries and much fewer compiler bugs), despite finding D more fun to write.

globular-toast 3y ago
Interesting that you put Lisp in difficult category despite the IDE experience being in another league compared to any other language I've tried.
- brabel 3y ago
  The IDE experience with Lisp is great because of the interactivity, not because its great analysis. You do get good analysis from the language introspection capabilities itself, but you need to "manually" use that... It's just different. I know SLIME and love it, but the analysis it's capable of doing is not near the level you get with IntelliJ and Java or Kotlin. For starter, Common Lisp (and most other Lisps) don't even have static types (I know you can declare types, but most libs don't do it and SLIME doesn't use that for any IDE feature as far as I know, other than perhaps warn when you use an obviously wrong argument type), making features like advanced refactoring very difficult. Macros make it very hard to even know whether the syntax is right (try writing a LOOP in Common Lisp - SLIME won't help you).

cyocum 3y ago

Reading this, I am increasing my belief that this is very, if not exactly, like XSLT. You are searching a tree looking for parents, siblings, children, etc nodes then analysing or transforming them.

heads 3y ago
Visiting and transforming trees is indeed a well trodden pattern. Here’s another kind of tree walking which I’ve enjoyed working with, this year: https://libcst.readthedocs.io/en/latest/tutorial.html#Build-Visitor-or-Transformer https://libcst.readthedocs.io/en/latest/tutorial.html#Build-... It is a little less pure than the functional XSLT, and also a lot more practical.

int0x80 3y ago

>In this formulation, a language server needs to just enough analysis to drill down to a specific node. Does rust-analyzer compute semantic information on the fly and incrementally?

matklad 3y ago
Yes