7 ms·

Descriptively, yes, the shape of the language determines how you can structure an interactive compiler. There are at least three wildly different ways to do tha

Descriptively, yes, the shape of the language determines how you can structure an interactive compiler. There are at least three wildly different ways to do that: https://rust-analyzer.github.io/blog/2020/07/20/three-architectures-for-responsive-ide.html https://rust-analyzer.github.io/blog/2020/07/20/three-archit... Which one can work depends on the language in question. Java, Rust, C++ lead to different answers, and that’s not because of the “interesting” differences like borrowchecker vs gc, but rather due to the “boring” differences in name resolution and module system. Prescriptively, whether we _should_ do this (or, rather, how much), is unclear. My biased answer is that, while language designers talk a lot about making languages tooling friendly, there’s little of that actually happening (at least with “current” languages, “next” ones seem to fare better). Like, it was said that “rust macros are designed with tooling friendliness in mind”, but overall the language is pretty tooling-hostile, mostly for accidental reasons. It seems to me that if we _actually_ co-design a tooling-first language, without trying to innovate too much, but by just ensuring that existing techniques work robustly, we might arrive at a close, but meaningfully different point in the design space. Specifically, here’s my version of IDE-friendliness diff for language design: Push conditional compilation far further in the pipeline, such that it is done after all semantic analysis. This is a pre-requisite for making automated refactors which are _guaranteed_ to work. Similarly, push meta programming further down, such that code analysis doesn’t invoke user-defined code (which might be arbitrary slow), but, at the same time, meta parts can fully reflect on existing code, including resolved types. C#-style source generators are an interesting design here. Have strong signatures on micro and macro level. Macro, have well-defined compilation units with explicitly specified dag of dependencies and signature files. Micro, annotate types of the functions. Use tooling to reduce double-annotation burden. Ensure that each source file can be somewhat deeply analyzed in complete isolation. The last two points should unlock both embarrassingly parallel (distributed) compilation, and snappy completion.

avgcorrection 3y ago
Thanks for that. It seems that some newer languages (like in the last ten years) either care about (1) batch compilation speed or (2) language design mostly unconstrained from having to have fast batch compilation. It would be interesting to see a third way: language design that takes into account all kinds of future interactive tooling by laying a good groundwork. So maybe from-scratch batch compilation ends up slow, but then it turns out to be fine in practice since you can distribute builds and cache things so that you only pay at most a one-time large cost. And then what you mostly have to deal with is incremental stuff.
- firstlink 3y ago
  Interactive proof languages already work that way, to a greater or lesser extent. But, for general-purpose languages, this revictimizes the docker addicts.
the_mitsuhiko 3y ago
> there’s little of that actually happening (at least with “current” languages, “next” ones seem to fare better). I'm not too involved but I think the languages that actually develop the tooling as part of the language design have traditionally done quite well at it in language design. Delphi/C#/TypeScript all had tooling built alongside the language and it shows. Rust sadly made a ton of choices in the language that just make a lot of things hard, but enabled the type of ecosystem that exists today. So it has different tradeoffs. I'm quite amazed how well RLS is doing considering how hostile the language is to fast iteration.
- valenterry 3y ago
  Look at Kotlin to see the opposite. It was designed to be IDE friendly - in fact, it was developed by the IntelliJ folks. However, looking at it now, it seems that Kotlin is pretty much falling behind even Java in terms of features and functionality. But especially it also seems behind in a well-thought design. When looking at Rust, I think while there are still a lot of features that I would desire, it at least introduced major improvements like typeclasses (or "traits" in Rust lingo) which really help the ecosystem and libraries to flourish. If this makes it harder for the compiler and the IDE then I think it's a fair tradeoff. Kotlin had the chance to do the same, they deliberately didn't and I is already starting to hinder the language from thriving.
  - rhdunn 3y ago
    Kotlin is in a tricky place w.r.t. Java because the JVM is one of Kotlin's target platforms. If Kotlin deviated from the JVM too far, it would be harder to migrate to the newer JVM versions. That specifically applies to things like pattern matching and value classes. Value classes are constrained to single values to allow the Kotlin compiler to do the heavy lifting without involving specific JVM capabilities. It will be interesting to see how Kotlin evolves with version 2.0 and beyond, with the new compiler and stable multiplatform support.
    - valenterry 3y ago
      I don't think so. Look at Scala, which is in the same position as Kotlin. Still, it has a far superior featureset both compared to Java and Kotlin and has had pattern matching since almost forever (in fact, Java's pattern matching, sealed classes etc. are heavily inspired by Scala's). And it manages quite fine to do all these things, even before the JVM improved to support those features better in a native way.
littlestymaar 3y ago
> Push conditional compilation far further in the pipeline, such that it is done after all semantic analysis So much this. I've idea how this would work un practice (like how you're supposed to accomodate antagonistic semantics which depend on the conditionnal compilation), but I hate #[feature(X)] with passion because of how brittle they are.
ComputerGuru 3y ago
Alex, I've been reading all your posts on Zig and Rust LSPs with great fervor and appreciate everything you've done for the community with rust-analyzer. Can I ask you about clangd, the llvm/C++ lsp? It seems to work tremendously well, far better than I naively ever thought it would even on massive codebases. I was using it from its first release and the quality ramped up rather quickly. Can I bug you to write up about it from your perspective and with your insights sometime and where we could benefit from their design choices and where we can't? And of course, the limitations of their approach? It's ostensibly driven by the `compile_commands.json` produced by the compiler during a run (mapping each source file to the output .o and the command (w/ all options) that was used to generate it) but in reality that's just because C++ doesn't have its own build system so the tooling has no other way of knowing how a project is wired up. But aside from that, it's extremely fast and has fairly low latency for a language that's got a lot of the same warts that rust has (generics, monomorphization, macros, strong types, (limited) type inference, etc) especially when compared to all the other LSP offerings (although I think C++ compilation units are smaller than rust's, which has to help).
- matklad 3y ago
  I covered this a bit here: https://rust-analyzer.github.io/blog/2020/07/20/three-architectures-for-responsive-ide.html#leveraging-headers https://rust-analyzer.github.io/blog/2020/07/20/three-archit... I don't actually _know_ how exactly clangd works, especially post modules, but pre-modules C++ has a compilation model which is actually quite friendly for an IDE, because of the header files. Headers essentially explicitly encode body/interface separation that more advanced incremental compilation systems like Salsa try to recover implicitly. Looking at a slightly simplified typical C++ #include <iostream> void main() { std::cout << "Hello, World!" << std:: } what a language server could do is that it can run bog-standard phased compiler, and just _freeze_ its state after all includes are parsed&expanded. After that, any typing inside the file needs to re-analyzer just the file itself. Also, the code in included `.h` files is typically far smaller then the code in the corresponding `.cpp` files. So, a language server gets ability to analyse only the relevant code, and skip the rest, for free, it is encoded in compilation model. Compare this with Rust, where, in the general case, changing a file can affect any other file in CU in arbitrary ways (_usually_ it doesn't, but it _could_, and the complexity lies precisely in figuring out which edits are local and which are not). And CU's themselves are meatier, as macro expansion can run arbitrary code, so you need to somehow cache that, taking into account that macro expansion _anywhere_ in CU can potentially affect _anything_ in CU. So, you need full build systems a-la carte incremental compilation, which is a) hard b) different from how the command line compiler works. The second point explains the success of clangd I think: _usually_, when you have a command-line compiler and want a language server, you want to rewrite, because the architectures are too different. But because of the way C++ (and OCaml) work, for those two languages you actually can soundly re-use existing code with relatively minor modifications.
lerno 3y ago
It’s hard to push down conditional compilation too far without compromising its value, but one can still stop at a much later point than Zig does. (In fact, the fundamental choices of Zig in regards to compile time necessarily makes it hard to create good IDEa for) In my C-like (C3), I retained an #if-like construct, which while semantic (unlike C’s) still meant dealing with sections of conditional code that had to be resolved before semantic checking could commence. To simplify I removed the ability to conditionally include struct members etc. Doing more research I found D’s `version` interesting. In particular the non-block variant where the version is attached to a declaration. I ended up creating a more flexible (and therefore actually less good) version of it as an attribute (`@if(cond)`) which then may be attached to any declaration. The advantage here is that at most this leads to some symbols having two (or more) different declarations (resolvable using the `@if`), which allows a lot of analysis even if the attribute hasn’t had its argument resolved. I think this might be the way forward at the top level. Inside of functions / macros, $if and friends are fine as they often do not matter for the overall analysis.

avgcorrection 3y ago

Thanks for that. It seems that some newer languages (like in the last ten years) either care about (1) batch compilation speed or (2) language design mostly unconstrained from having to have fast batch compilation. It would be interesting to see a third way: language design that takes into account all kinds of future interactive tooling by laying a good groundwork. So maybe from-scratch batch compilation ends up slow, but then it turns out to be fine in practice since you can distribute builds and cache things so that you only pay at most a one-time large cost. And then what you mostly have to deal with is incremental stuff.

firstlink 3y ago
Interactive proof languages already work that way, to a greater or lesser extent. But, for general-purpose languages, this revictimizes the docker addicts.

the_mitsuhiko 3y ago

> there’s little of that actually happening (at least with “current” languages, “next” ones seem to fare better). I'm not too involved but I think the languages that actually develop the tooling as part of the language design have traditionally done quite well at it in language design. Delphi/C#/TypeScript all had tooling built alongside the language and it shows. Rust sadly made a ton of choices in the language that just make a lot of things hard, but enabled the type of ecosystem that exists today. So it has different tradeoffs. I'm quite amazed how well RLS is doing considering how hostile the language is to fast iteration.

valenterry 3y ago
Look at Kotlin to see the opposite. It was designed to be IDE friendly - in fact, it was developed by the IntelliJ folks. However, looking at it now, it seems that Kotlin is pretty much falling behind even Java in terms of features and functionality. But especially it also seems behind in a well-thought design. When looking at Rust, I think while there are still a lot of features that I would desire, it at least introduced major improvements like typeclasses (or "traits" in Rust lingo) which really help the ecosystem and libraries to flourish. If this makes it harder for the compiler and the IDE then I think it's a fair tradeoff. Kotlin had the chance to do the same, they deliberately didn't and I is already starting to hinder the language from thriving.
- rhdunn 3y ago
  Kotlin is in a tricky place w.r.t. Java because the JVM is one of Kotlin's target platforms. If Kotlin deviated from the JVM too far, it would be harder to migrate to the newer JVM versions. That specifically applies to things like pattern matching and value classes. Value classes are constrained to single values to allow the Kotlin compiler to do the heavy lifting without involving specific JVM capabilities. It will be interesting to see how Kotlin evolves with version 2.0 and beyond, with the new compiler and stable multiplatform support.
  - valenterry 3y ago
    I don't think so. Look at Scala, which is in the same position as Kotlin. Still, it has a far superior featureset both compared to Java and Kotlin and has had pattern matching since almost forever (in fact, Java's pattern matching, sealed classes etc. are heavily inspired by Scala's). And it manages quite fine to do all these things, even before the JVM improved to support those features better in a native way.

littlestymaar 3y ago

> Push conditional compilation far further in the pipeline, such that it is done after all semantic analysis So much this. I've idea how this would work un practice (like how you're supposed to accomodate antagonistic semantics which depend on the conditionnal compilation), but I hate #[feature(X)] with passion because of how brittle they are.

ComputerGuru 3y ago

Alex, I've been reading all your posts on Zig and Rust LSPs with great fervor and appreciate everything you've done for the community with rust-analyzer. Can I ask you about clangd, the llvm/C++ lsp? It seems to work tremendously well, far better than I naively ever thought it would even on massive codebases. I was using it from its first release and the quality ramped up rather quickly. Can I bug you to write up about it from your perspective and with your insights sometime and where we could benefit from their design choices and where we can't? And of course, the limitations of their approach? It's ostensibly driven by the `compile_commands.json` produced by the compiler during a run (mapping each source file to the output .o and the command (w/ all options) that was used to generate it) but in reality that's just because C++ doesn't have its own build system so the tooling has no other way of knowing how a project is wired up. But aside from that, it's extremely fast and has fairly low latency for a language that's got a lot of the same warts that rust has (generics, monomorphization, macros, strong types, (limited) type inference, etc) especially when compared to all the other LSP offerings (although I think C++ compilation units are smaller than rust's, which has to help).

matklad 3y ago
I covered this a bit here: https://rust-analyzer.github.io/blog/2020/07/20/three-architectures-for-responsive-ide.html#leveraging-headers https://rust-analyzer.github.io/blog/2020/07/20/three-archit... I don't actually _know_ how exactly clangd works, especially post modules, but pre-modules C++ has a compilation model which is actually quite friendly for an IDE, because of the header files. Headers essentially explicitly encode body/interface separation that more advanced incremental compilation systems like Salsa try to recover implicitly. Looking at a slightly simplified typical C++ #include <iostream> void main() { std::cout << "Hello, World!" << std:: } what a language server could do is that it can run bog-standard phased compiler, and just _freeze_ its state after all includes are parsed&expanded. After that, any typing inside the file needs to re-analyzer just the file itself. Also, the code in included `.h` files is typically far smaller then the code in the corresponding `.cpp` files. So, a language server gets ability to analyse only the relevant code, and skip the rest, for free, it is encoded in compilation model. Compare this with Rust, where, in the general case, changing a file can affect any other file in CU in arbitrary ways (_usually_ it doesn't, but it _could_, and the complexity lies precisely in figuring out which edits are local and which are not). And CU's themselves are meatier, as macro expansion can run arbitrary code, so you need to somehow cache that, taking into account that macro expansion _anywhere_ in CU can potentially affect _anything_ in CU. So, you need full build systems a-la carte incremental compilation, which is a) hard b) different from how the command line compiler works. The second point explains the success of clangd I think: _usually_, when you have a command-line compiler and want a language server, you want to rewrite, because the architectures are too different. But because of the way C++ (and OCaml) work, for those two languages you actually can soundly re-use existing code with relatively minor modifications.

lerno 3y ago

It’s hard to push down conditional compilation too far without compromising its value, but one can still stop at a much later point than Zig does. (In fact, the fundamental choices of Zig in regards to compile time necessarily makes it hard to create good IDEa for) In my C-like (C3), I retained an #if-like construct, which while semantic (unlike C’s) still meant dealing with sections of conditional code that had to be resolved before semantic checking could commence. To simplify I removed the ability to conditionally include struct members etc. Doing more research I found D’s `version` interesting. In particular the non-block variant where the version is attached to a declaration. I ended up creating a more flexible (and therefore actually less good) version of it as an attribute (`@if(cond)`) which then may be attached to any declaration. The advantage here is that at most this leads to some symbols having two (or more) different declarations (resolvable using the `@if`), which allows a lot of analysis even if the attribute hasn’t had its argument resolved. I think this might be the way forward at the top level. Inside of functions / macros, $if and friends are fine as they often do not matter for the overall analysis.