9 ms·

Rust Moving Towards an IDE-Friendly Compiler with Rust Analyzer

235 points by sethev 7y ago

_bxg1 7y ago
> Another thing is difference in handling invalid code. A traditional compiler front-end is usually organized as a progression of phases, where each phase takes an unstructured input, checks the input for validity, and, if it is indeed valid, adds more structure on top. Specifically, an error in an early phase (like parsing) usually means that the latter phase (like type checking) is not run for this bit of code at all. In other words, "correct code" is a happy case, and everything else can be treated as an error condition. In contrast, in IDE code is always broken, because the user constantly modifies it. As soon as the code is valid, the job of IDE ends and the job of the batch compiler begins. So, an IDE-oriented compiler should accommodate an incomplete and broken code, and provide IDE features, like completion, for such code. This has been one of the most frustrating things about Rust's development experience. A piece of code looks error-free, then you fix an error somewhere else, reach the next compiler phase, and an error pops up where you just were. It makes it really hard to develop something piece by piece. I'm glad they're aware of these problems and are making a concerted effort to address them.
- epage 7y ago
  Hadn't thought of that potential benefit to the rust-analyzer work: I could get borrow errors upfront even if there are irrelevant errors in other areas of code. (borrow checker seems like it is one of the last phases).
  - woah 7y ago
    That’s how you know you’re reaching the light at the end of the tunnel, when the borrow checker errors start showing up
    - empath75 7y ago
      That’s usually when I give up and start writing go, tbh.
      - echelon 7y ago
        Do you have to dump on Rust because of its learning curve? It isn't even that bad. It's not like Golang doesn't have warts of its own. I have a hard time believing you make this choice often or "usually". This seems like flat out prejudice.
      - galangalalgol 7y ago
        If go was fast enough why didn't you start out using it, or some other language that is 3x slower than rust but has gc and comfy ergonomics. C#, clojure, julia, ocaml( if you dont need threads). Or just plane Java.
    - _bxg1 7y ago
      I've started associating a shower of warnings with "success", because those only appear once all the errors have been taken care of
    - nirui 7y ago
      Or, in my case, it tells me why I have to restructure my code :( I mean, an early warning, before I actually made those mistakes would help a lot, if it can be done that is :)
- Buttons840 7y ago
  > A piece of code looks error-free, then you fix an error somewhere else, reach the next compiler phase, and an error pops up where you just were. Isn't this the case with all compilers? The most basic example being (for example) I have a syntax error on line 10, and that's the only reported problem, but then I fix the syntax error and now I have an error on line 200. I agree it is a problem, the world would be better if all languages could avoid it, but I don't expect it ever to go away. Do you believe this problem is better or worse in Rust compared to other languages?
  - edflsafoiewq 7y ago
    > Do you believe this problem is better or worse in Rust compared to other languages? Unlike other languages, Rust still has borrow checking to go after all the type errors are gone.
    - pjmlp 7y ago
      An idea currently being added to Swift, C++ (via static analysis, core guidelines lifetime profile), Ada/SPARK, D, OCaml and Haskell, originally developed in Cyclone, further explored in ATS. So that "Unlike other languages" isn't quite true. What Rust has done, was to prove it is viable to push such ideas into mainstream computing.
      - pcwalton 7y ago
        This is incorrect. Neither Cyclone nor ATS had borrow checking. In fact the ATS author at one point dismissed borrow checking as too inflexible. The borrow checker is actually original, though based on known techniques. Specifically, the original part is that it allows the per-use-site choice of aliasing vs. mutability instead of making the decision globally.
        pjmlp 7y ago
        I might be wrong on the origins, but I am certainly right that everyone else is now adopting similar algorithms. So while it is great that Rust is making these concepts mainstream, in about 5 years time, the other languages will have their New Jersey style implementations mature and available for their communities.
- earenndil 7y ago
  > Another thing is difference in handling invalid code. A traditional compiler front-end is usually organized as a progression of phases, where each phase takes an unstructured input, checks the input for validity, and, if it is indeed valid, adds more structure on top. Is this true? The way the d compiler does it is to make a special ast node type called 'error', which allows it to avoid this problem. I have also gotten error messages from c compilers that indicated they were doing something similar. Now, in the d compiler, this is not a panacea, because erroneous code can cause erroneous error messages about unreachable code, but it generally works quite well.
  - estebank 7y ago
    rustc does the same. I believe the op complaint is that rustc does not perform type and lifetime checks on blocks of code that haven't been successfully parsed.
dunkelheit 7y ago
Aleksey is true to his “avoid success at all cost” strategy, constantly emphasizing that rust-analyzer is experimental and alpha quality. Even the installation process is I think deliberately clunky. In fact in my opinion it already provides a better IDE experience than RLS and is improving daily, whereas RLS is stuck in maintenance mode.
- indemnity 7y ago
  Also, you won't find a link to the Open Collective project for rust-analyzer on the GitHub repo or website. I'm also contributing to https://opencollective.com/rust-analyzer https://opencollective.com/rust-analyzer, I want it to be a viable project.
- GolDDranks 7y ago
  That is understandable in the sense that the original RLS seemed to follow an exact opposite of a strategy: it was frequently said to be "pretty good already" and "close to 1.0", but some people experienced crashes and the code completion not working, and were obviously let down, which led to even some animosity towards the project.
- matklad 7y ago
  Being alpha quality and providing better experience than RLS are not incompatible. rust-analyzer is definitely alpha-quality at the moment.
  - dunkelheit 7y ago
    Sure, it is a good idea to be upfront about the current state of the project. Still I think I am not mistaken in my impression that you don't wish for rust-analyzer to become popular too early as that would put some unwanted constraints on experimentation. BTW, thanks the project and good luck! I use it daily and it is awesome.
    - matklad 7y ago
      >Still I think I am not mistaken in my impression that you don't wish for rust-analyzer to become popular too early as that would put some unwanted constraints on experimentation. You are not mistaken indeed. More specifically, I do want to maintain freedom of pushing completely broken code. It's not necessary about popularity, it's about user expectations. And I also don't allocate as much time as I could into things that directly increase popularity. Like, the "install extension from the market place" could have been done almost a month ago, but I still haven't got to it. It's not that I am deliberately pushing back against such improvements, I just don't push them forward to actively.
xiphias2 7y ago
It would be great to have a timeline as well: when does the project plans to superseed the current RLS? What's the goal for end of 2020? At the same time I know how hard is to estimate projects.
- nindalf 7y ago
  From what I've read, the maintainers don't yet want to discuss when rust-analyzer will be the recommended IDE experience. As a user, I can say it's pretty good and you should use it today if you're writing Rust code. So to me, it doesn't matter whether the docs recommend it, or if it's distributed by default with the compiler. I just use it and tell everyone I know to use it.
rictic 7y ago
What would a compiler for a language like Rust or C++ look like if it was optimized for development iteration speed first? Could you produce binaries that were good enough to be useful for testing and developing? You'd end up sharing a lot of the same infrastructure that you'd want for an LSP server, basically to do as much preprocessing as you could on the initial compilation and then work incrementally on top of that. You'd also of course still want your globally-optimizing batch mode compiler for production, but for many projects it seems like such a thing could produce good enough binaries for development, and greatly improve engineer productivity.
- dropdrive 7y ago
  Slightly tangential but one thing I absolutely love about sbt/play framework is the frigging development loop. You start play using `play ~run`, save changes and by the time you've switched to your browser, the source has been compiled to `.class` and you are ready to hit f5. It's almost magical, DESPITE scala being slow as snail at compilation. The scala metals LSP server also has better than enterprise grade documentation. https://scalameta.org/metals/docs/editors/vim.html https://scalameta.org/metals/docs/editors/vim.html
  - switchbak 7y ago
    The Bloop project is doing amazing work as well in making Scala compilation leagues faster (even faster than SBT). I hear Scala 3 should help significantly in this area as well. I wish this would be a higher priority generally, developer time being as expensive as it is.
- the8472 7y ago
  That's pretty much what rust-analyzer aims to be.
- badsectoracula 7y ago
  It'd look like Delphi i'd guess, at least the earlier versions. Delphi 2 is stupidly fast, almost instant compilation on a contemporary PC that runs at around 200MHz with 16MB of RAM and practically instant (in that it is impossible to notice it) in any modern PC (a synthetic benchmark i wrote some weeks ago had it compile a bit above 10MLOC in 5.56 seconds on my 3700X CPU, which is an average priced consumer CPU - on a single thread). While it doesn't have the niceties added in later versions and in Free Pascal (like dynamic arrays, generics, anonymous functions and a few other things) it still has more than enough features to create big complex applications (e.g. object oriented language, rich RTTI that allows automatic serialization, properties, native string support, optional reference counting, real module support). In terms of generated code... well, it isn't exactly great (it came last on this benchmark i wrote some time ago http://runtimeterror.com/tools/raybench/ http://runtimeterror.com/tools/raybench/ that compares some C and Pascal compilers - i mainly wrote it for retrocoding, but threw in some modern compilers as well - EDIT: i misremembered, it didn't came last, Borland C++ 5.0 came last, in fact it came slightly above Free Pascal 3.0.4 with its default settings, though by enabling 64bit and modern instructions FPC was able to generate much faster code). But then again, it is written in itself and as i wrote, it is stupidly fast. So it does the work perfectly fine for most tasks - as long as you don't need brute force number crunching. Of course i'm not advocating the use of Delphi 2 (and i do not know how it compares with modern Delphi - last time i tried out the free version they have the entire IDE felt very weird and sluggish). However the fact that it exists is a proof that you can have a very fast compiler and IDE for a rich language that produces very good code - maybe not the best possible code, but good enough for a large majority of tasks.
unnouinceput 7y ago
Quote: "The main thing is that the command line (or batch) compiler is primarily optimized for throughput (compiling N thousands lines of code per second), while an IDE compiler is optimized for latency (showing correct completion variants in M milliseconds after user typed new fragment of code)" And then there's Delphi, doing both.
- int_19h 7y ago
  Delphi could pull this off by being a language designed to be easy to parse (and often unnecessarily restrictive or verbose as a result). This goes all the way back to Turbo Pascal - while Borland's DOS C compilers were also very fast, their Pascal was unbeatable because of very straightforward parsing, and true separate compilation of units.
  - unnouinceput 7y ago
    *can pull this off. To this day Delphi is stupidly fast when compiling. Despite lately being bloated with stuff it didn't really need, but hey! C/C++ standard is all the rage so why not.
Dowwie 7y ago
I am so grateful for the effort that people are putting into the rust-analyzer project. It keeps getting better. I wonder whether an entire class of challenges that the project faces is attributed to electron?
- nindalf 7y ago
  I use rust-analyzer and I don't think it has any issues related to electron.
- matklad 7y ago
  > I wonder whether an entire class of challenges that the project faces is attributed to electron? No, not at all.
- maccam94 7y ago
  You might be confusing rust-analyzer with coc[1]? 1: https://github.com/neoclide/coc.nvim https://github.com/neoclide/coc.nvim
CameronNemo 7y ago
I have been using racer and neomake with a lot of success, fwiw.

235 points by sethev 7y ago

> Another thing is difference in handling invalid code. A traditional compiler front-end is usually organized as a progression of phases, where each phase takes an unstructured input, checks the input for validity, and, if it is indeed valid, adds more structure on top. Specifically, an error in an early phase (like parsing) usually means that the latter phase (like type checking) is not run for this bit of code at all. In other words, "correct code" is a happy case, and everything else can be treated as an error condition. In contrast, in IDE code is always broken, because the user constantly modifies it. As soon as the code is valid, the job of IDE ends and the job of the batch compiler begins. So, an IDE-oriented compiler should accommodate an incomplete and broken code, and provide IDE features, like completion, for such code. This has been one of the most frustrating things about Rust's development experience. A piece of code looks error-free, then you fix an error somewhere else, reach the next compiler phase, and an error pops up where you just were. It makes it really hard to develop something piece by piece. I'm glad they're aware of these problems and are making a concerted effort to address them.

epage 7y ago
Hadn't thought of that potential benefit to the rust-analyzer work: I could get borrow errors upfront even if there are irrelevant errors in other areas of code. (borrow checker seems like it is one of the last phases).
- woah 7y ago
  That’s how you know you’re reaching the light at the end of the tunnel, when the borrow checker errors start showing up
  - empath75 7y ago
    That’s usually when I give up and start writing go, tbh.
    - echelon 7y ago
      Do you have to dump on Rust because of its learning curve? It isn't even that bad. It's not like Golang doesn't have warts of its own. I have a hard time believing you make this choice often or "usually". This seems like flat out prejudice.
    - galangalalgol 7y ago
      If go was fast enough why didn't you start out using it, or some other language that is 3x slower than rust but has gc and comfy ergonomics. C#, clojure, julia, ocaml( if you dont need threads). Or just plane Java.
  - _bxg1 7y ago
    I've started associating a shower of warnings with "success", because those only appear once all the errors have been taken care of
  - nirui 7y ago
    Or, in my case, it tells me why I have to restructure my code :( I mean, an early warning, before I actually made those mistakes would help a lot, if it can be done that is :)
Buttons840 7y ago
> A piece of code looks error-free, then you fix an error somewhere else, reach the next compiler phase, and an error pops up where you just were. Isn't this the case with all compilers? The most basic example being (for example) I have a syntax error on line 10, and that's the only reported problem, but then I fix the syntax error and now I have an error on line 200. I agree it is a problem, the world would be better if all languages could avoid it, but I don't expect it ever to go away. Do you believe this problem is better or worse in Rust compared to other languages?
- edflsafoiewq 7y ago
  > Do you believe this problem is better or worse in Rust compared to other languages? Unlike other languages, Rust still has borrow checking to go after all the type errors are gone.
  - pjmlp 7y ago
    An idea currently being added to Swift, C++ (via static analysis, core guidelines lifetime profile), Ada/SPARK, D, OCaml and Haskell, originally developed in Cyclone, further explored in ATS. So that "Unlike other languages" isn't quite true. What Rust has done, was to prove it is viable to push such ideas into mainstream computing.
    - pcwalton 7y ago
      This is incorrect. Neither Cyclone nor ATS had borrow checking. In fact the ATS author at one point dismissed borrow checking as too inflexible. The borrow checker is actually original, though based on known techniques. Specifically, the original part is that it allows the per-use-site choice of aliasing vs. mutability instead of making the decision globally.
      - pjmlp 7y ago
        I might be wrong on the origins, but I am certainly right that everyone else is now adopting similar algorithms. So while it is great that Rust is making these concepts mainstream, in about 5 years time, the other languages will have their New Jersey style implementations mature and available for their communities.
earenndil 7y ago
> Another thing is difference in handling invalid code. A traditional compiler front-end is usually organized as a progression of phases, where each phase takes an unstructured input, checks the input for validity, and, if it is indeed valid, adds more structure on top. Is this true? The way the d compiler does it is to make a special ast node type called 'error', which allows it to avoid this problem. I have also gotten error messages from c compilers that indicated they were doing something similar. Now, in the d compiler, this is not a panacea, because erroneous code can cause erroneous error messages about unreachable code, but it generally works quite well.
- estebank 7y ago
  rustc does the same. I believe the op complaint is that rustc does not perform type and lifetime checks on blocks of code that haven't been successfully parsed.

dunkelheit 7y ago

Aleksey is true to his “avoid success at all cost” strategy, constantly emphasizing that rust-analyzer is experimental and alpha quality. Even the installation process is I think deliberately clunky. In fact in my opinion it already provides a better IDE experience than RLS and is improving daily, whereas RLS is stuck in maintenance mode.

indemnity 7y ago
Also, you won't find a link to the Open Collective project for rust-analyzer on the GitHub repo or website. I'm also contributing to https://opencollective.com/rust-analyzer https://opencollective.com/rust-analyzer, I want it to be a viable project.
GolDDranks 7y ago
That is understandable in the sense that the original RLS seemed to follow an exact opposite of a strategy: it was frequently said to be "pretty good already" and "close to 1.0", but some people experienced crashes and the code completion not working, and were obviously let down, which led to even some animosity towards the project.
matklad 7y ago
Being alpha quality and providing better experience than RLS are not incompatible. rust-analyzer is definitely alpha-quality at the moment.
- dunkelheit 7y ago
  Sure, it is a good idea to be upfront about the current state of the project. Still I think I am not mistaken in my impression that you don't wish for rust-analyzer to become popular too early as that would put some unwanted constraints on experimentation. BTW, thanks the project and good luck! I use it daily and it is awesome.
  - matklad 7y ago
    >Still I think I am not mistaken in my impression that you don't wish for rust-analyzer to become popular too early as that would put some unwanted constraints on experimentation. You are not mistaken indeed. More specifically, I do want to maintain freedom of pushing completely broken code. It's not necessary about popularity, it's about user expectations. And I also don't allocate as much time as I could into things that directly increase popularity. Like, the "install extension from the market place" could have been done almost a month ago, but I still haven't got to it. It's not that I am deliberately pushing back against such improvements, I just don't push them forward to actively.

xiphias2 7y ago

It would be great to have a timeline as well: when does the project plans to superseed the current RLS? What's the goal for end of 2020? At the same time I know how hard is to estimate projects.

nindalf 7y ago
From what I've read, the maintainers don't yet want to discuss when rust-analyzer will be the recommended IDE experience. As a user, I can say it's pretty good and you should use it today if you're writing Rust code. So to me, it doesn't matter whether the docs recommend it, or if it's distributed by default with the compiler. I just use it and tell everyone I know to use it.

rictic 7y ago

What would a compiler for a language like Rust or C++ look like if it was optimized for development iteration speed first? Could you produce binaries that were good enough to be useful for testing and developing? You'd end up sharing a lot of the same infrastructure that you'd want for an LSP server, basically to do as much preprocessing as you could on the initial compilation and then work incrementally on top of that. You'd also of course still want your globally-optimizing batch mode compiler for production, but for many projects it seems like such a thing could produce good enough binaries for development, and greatly improve engineer productivity.

dropdrive 7y ago
Slightly tangential but one thing I absolutely love about sbt/play framework is the frigging development loop. You start play using `play ~run`, save changes and by the time you've switched to your browser, the source has been compiled to `.class` and you are ready to hit f5. It's almost magical, DESPITE scala being slow as snail at compilation. The scala metals LSP server also has better than enterprise grade documentation. https://scalameta.org/metals/docs/editors/vim.html https://scalameta.org/metals/docs/editors/vim.html
- switchbak 7y ago
  The Bloop project is doing amazing work as well in making Scala compilation leagues faster (even faster than SBT). I hear Scala 3 should help significantly in this area as well. I wish this would be a higher priority generally, developer time being as expensive as it is.
the8472 7y ago
That's pretty much what rust-analyzer aims to be.
badsectoracula 7y ago
It'd look like Delphi i'd guess, at least the earlier versions. Delphi 2 is stupidly fast, almost instant compilation on a contemporary PC that runs at around 200MHz with 16MB of RAM and practically instant (in that it is impossible to notice it) in any modern PC (a synthetic benchmark i wrote some weeks ago had it compile a bit above 10MLOC in 5.56 seconds on my 3700X CPU, which is an average priced consumer CPU - on a single thread). While it doesn't have the niceties added in later versions and in Free Pascal (like dynamic arrays, generics, anonymous functions and a few other things) it still has more than enough features to create big complex applications (e.g. object oriented language, rich RTTI that allows automatic serialization, properties, native string support, optional reference counting, real module support). In terms of generated code... well, it isn't exactly great (it came last on this benchmark i wrote some time ago http://runtimeterror.com/tools/raybench/ http://runtimeterror.com/tools/raybench/ that compares some C and Pascal compilers - i mainly wrote it for retrocoding, but threw in some modern compilers as well - EDIT: i misremembered, it didn't came last, Borland C++ 5.0 came last, in fact it came slightly above Free Pascal 3.0.4 with its default settings, though by enabling 64bit and modern instructions FPC was able to generate much faster code). But then again, it is written in itself and as i wrote, it is stupidly fast. So it does the work perfectly fine for most tasks - as long as you don't need brute force number crunching. Of course i'm not advocating the use of Delphi 2 (and i do not know how it compares with modern Delphi - last time i tried out the free version they have the entire IDE felt very weird and sluggish). However the fact that it exists is a proof that you can have a very fast compiler and IDE for a rich language that produces very good code - maybe not the best possible code, but good enough for a large majority of tasks.

unnouinceput 7y ago

Quote: "The main thing is that the command line (or batch) compiler is primarily optimized for throughput (compiling N thousands lines of code per second), while an IDE compiler is optimized for latency (showing correct completion variants in M milliseconds after user typed new fragment of code)" And then there's Delphi, doing both.

int_19h 7y ago
Delphi could pull this off by being a language designed to be easy to parse (and often unnecessarily restrictive or verbose as a result). This goes all the way back to Turbo Pascal - while Borland's DOS C compilers were also very fast, their Pascal was unbeatable because of very straightforward parsing, and true separate compilation of units.
- unnouinceput 7y ago
  *can pull this off. To this day Delphi is stupidly fast when compiling. Despite lately being bloated with stuff it didn't really need, but hey! C/C++ standard is all the rage so why not.

Dowwie 7y ago

I am so grateful for the effort that people are putting into the rust-analyzer project. It keeps getting better. I wonder whether an entire class of challenges that the project faces is attributed to electron?

nindalf 7y ago
I use rust-analyzer and I don't think it has any issues related to electron.
matklad 7y ago
> I wonder whether an entire class of challenges that the project faces is attributed to electron? No, not at all.
maccam94 7y ago
You might be confusing rust-analyzer with coc[1]? 1: https://github.com/neoclide/coc.nvim https://github.com/neoclide/coc.nvim

CameronNemo 7y ago

I have been using racer and neomake with a lot of success, fwiw.