5 ms·

I'm not a compiler writer myself, but my impression is that designing a compiler "the traditional way" and designing one "the IDE way" look really different. Mo

I'm not a compiler writer myself, but my impression is that designing a compiler "the traditional way" and designing one "the IDE way" look really different. Most people who learned to write compilers 10 or more years ago learned the traditional way. As a new language, Rust benefited from a lot of obvious-in-hindsight things, but "design your compiler for an IDE first" just hadn't reached quite the same level of obviousness by 2010.

munificent 7y ago
You are right. Even today, my impression is that most CS students taking compilers and PL courses are still taught in the "batch mode compile the world" way. Most textbooks still assume that too. It's somewhat understandable because the way you architect a compiler for an IDE is a lot more complex. It's basically everything that's hard about a compiler plus everything that's hard about data validation, and everything that's hard about caching and cache invalidation. But, yeah, there's a big gap between industry and academia with regards to how to architect an IDE/compiler.
- vaylian 7y ago
  I think IDEs make things look more complicated than they actually are. Especially new students have trouble understanding what the difference between a programming language and a programming environment is, because they are exposed to them as one unit.
_bxg1 7y ago
As someone who knows very little about compiler implementation but has some interest in it, I'm curious what makes the difference between the two in terms of architecture
- pjmlp 7y ago
  Basically the IDE one needs to take into account that your program is broken all the time, yet you want code completion for everything else that is actually correct. Also it needs to respond immediately after asking for completion, as anything beyond 2s is frustrating development experience. You also want to get real time errors and warnings, just for the parts that are actually broken, not a wall of text like many batch compilers that fail to understand the remaining of the file. Also you want to be able to do code refactorings, regardless of the compilation state. So basically you want a Smalltalk/Lisp Machines like experience. Anders has a nice interview about this, https://channel9.msdn.com/Blogs/Seth-Juarez/Anders-Hejlsberg-on-Modern-Compiler-Construction https://channel9.msdn.com/Blogs/Seth-Juarez/Anders-Hejlsberg...
  - _bxg1 7y ago
    So would you typically maintain a separate compiler for each use-case, or try and make one that serves both?
    - steveklabnik 7y ago
      IMHO, one that serves both. The IDE case is a superset of the batch case.
- estebank 7y ago
  It not only informs the architecture but also the language design. For IDEs you want quick response even if it is not perfect, for a compiler you want correctness over everything else. The language having syntactic redundancy can help an IDE parser recover gracefully from a typo or missing tokens. Having semantic negative space can let an IDE recognize intent for code that looks correct from extrapolated understanding that is actually not semantically correct, and suggest solutions. rustc already tries to do all of these strategies, but blows the latency budget because it still prioritizes correctness over everything else.
- dralley 7y ago
  There's a decent explanation in here: https://www.infoq.com/news/2020/01/rust-analyser-ide-support/ https://www.infoq.com/news/2020/01/rust-analyser-ide-support...
- oconnor663 7y ago
  My understanding is that the "traditional way" is based on a number of "passes". In my head it looks like: 1) Parse all the code. 2) Assemble the set of all types. 3) Typecheck all the code. 4) Translate the parse tree into unoptimized machine code. 5) Optimize all that machine code. This is oversimplifying things, as in practice there are intermediate representations between the parse tree and the machine code. And optimization itself usually involves multiple passes of different kinds. But anyway, the key point here is that, if you change any of the code, you have to run all of these steps all over again. (I'm oversimplifying again. Maybe you only have to rerun them for a given "compilation unit", but that's bad enough.) This is the opposite of what you want for an IDE. There, you want to say "I just changed this function. Please recompile the absolute minimum necessary to tell me whether my change works." To answer a question like that efficiently, you have to rearchitect the whole compiler from being pass-based to being query-based, so you can give it instructions like "please update the type of just this expression".
robocat 7y ago
Anders Hejlsberg explains explains how “the IDE” way compiler works. First ~10 minutes is traditional compiler background. Then about ~8 minutes before he starts talking about how an IDE focused compiler works: https://www.youtube.com/watch?v=wSdV1M7n4gQ https://www.youtube.com/watch?v=wSdV1M7n4gQ
estebank 7y ago
Keep in mind that Rust was already trying to figure out what the language itself was supposed to be and even if the whole lifetime analysis would work once they landed on it as a possible solution. Making a compiler suitable for an IDE from the beginning is much harder (but can inform language design). If you compare the output from rustc 1.0 to any recent version you will see how much of the work needed for a production compiler is at best tangentially related to what people consider the raison d'être for a compiler.
bsder 7y ago
> As a new language, Rust benefited from a lot of obvious-in-hindsight things, but "design your compiler for an IDE first" just hadn't reached quite the same level of obviousness by 2010. The whole idea of a "language server protocol" is relatively new. And it's why VSCode is absolutely killing it in a lot of spaces. A "language server protocol" is ... kind of obvious? But it requires a lot of power behind it. It also requires languages whose grammars are optimized in such a way that they don't have to compile the universe to figure things out (see: C and C++) How did IntelliJ handle things? Did they do something analogous?
- Argorak 7y ago
  To my knowledge, IntelliJ has a base framework for handling many languages build towards an internal protocol.

munificent 7y ago

You are right. Even today, my impression is that most CS students taking compilers and PL courses are still taught in the "batch mode compile the world" way. Most textbooks still assume that too. It's somewhat understandable because the way you architect a compiler for an IDE is a lot more complex. It's basically everything that's hard about a compiler plus everything that's hard about data validation, and everything that's hard about caching and cache invalidation. But, yeah, there's a big gap between industry and academia with regards to how to architect an IDE/compiler.

vaylian 7y ago
I think IDEs make things look more complicated than they actually are. Especially new students have trouble understanding what the difference between a programming language and a programming environment is, because they are exposed to them as one unit.

_bxg1 7y ago

As someone who knows very little about compiler implementation but has some interest in it, I'm curious what makes the difference between the two in terms of architecture

pjmlp 7y ago
Basically the IDE one needs to take into account that your program is broken all the time, yet you want code completion for everything else that is actually correct. Also it needs to respond immediately after asking for completion, as anything beyond 2s is frustrating development experience. You also want to get real time errors and warnings, just for the parts that are actually broken, not a wall of text like many batch compilers that fail to understand the remaining of the file. Also you want to be able to do code refactorings, regardless of the compilation state. So basically you want a Smalltalk/Lisp Machines like experience. Anders has a nice interview about this, https://channel9.msdn.com/Blogs/Seth-Juarez/Anders-Hejlsberg-on-Modern-Compiler-Construction https://channel9.msdn.com/Blogs/Seth-Juarez/Anders-Hejlsberg...
- _bxg1 7y ago
  So would you typically maintain a separate compiler for each use-case, or try and make one that serves both?
  - steveklabnik 7y ago
    IMHO, one that serves both. The IDE case is a superset of the batch case.
estebank 7y ago
It not only informs the architecture but also the language design. For IDEs you want quick response even if it is not perfect, for a compiler you want correctness over everything else. The language having syntactic redundancy can help an IDE parser recover gracefully from a typo or missing tokens. Having semantic negative space can let an IDE recognize intent for code that looks correct from extrapolated understanding that is actually not semantically correct, and suggest solutions. rustc already tries to do all of these strategies, but blows the latency budget because it still prioritizes correctness over everything else.
dralley 7y ago
There's a decent explanation in here: https://www.infoq.com/news/2020/01/rust-analyser-ide-support/ https://www.infoq.com/news/2020/01/rust-analyser-ide-support...
oconnor663 7y ago
My understanding is that the "traditional way" is based on a number of "passes". In my head it looks like: 1) Parse all the code. 2) Assemble the set of all types. 3) Typecheck all the code. 4) Translate the parse tree into unoptimized machine code. 5) Optimize all that machine code. This is oversimplifying things, as in practice there are intermediate representations between the parse tree and the machine code. And optimization itself usually involves multiple passes of different kinds. But anyway, the key point here is that, if you change any of the code, you have to run all of these steps all over again. (I'm oversimplifying again. Maybe you only have to rerun them for a given "compilation unit", but that's bad enough.) This is the opposite of what you want for an IDE. There, you want to say "I just changed this function. Please recompile the absolute minimum necessary to tell me whether my change works." To answer a question like that efficiently, you have to rearchitect the whole compiler from being pass-based to being query-based, so you can give it instructions like "please update the type of just this expression".

robocat 7y ago

Anders Hejlsberg explains explains how “the IDE” way compiler works. First ~10 minutes is traditional compiler background. Then about ~8 minutes before he starts talking about how an IDE focused compiler works: https://www.youtube.com/watch?v=wSdV1M7n4gQ https://www.youtube.com/watch?v=wSdV1M7n4gQ

estebank 7y ago

Keep in mind that Rust was already trying to figure out what the language itself was supposed to be and even if the whole lifetime analysis would work once they landed on it as a possible solution. Making a compiler suitable for an IDE from the beginning is much harder (but can inform language design). If you compare the output from rustc 1.0 to any recent version you will see how much of the work needed for a production compiler is at best tangentially related to what people consider the raison d'être for a compiler.

bsder 7y ago

> As a new language, Rust benefited from a lot of obvious-in-hindsight things, but "design your compiler for an IDE first" just hadn't reached quite the same level of obviousness by 2010. The whole idea of a "language server protocol" is relatively new. And it's why VSCode is absolutely killing it in a lot of spaces. A "language server protocol" is ... kind of obvious? But it requires a lot of power behind it. It also requires languages whose grammars are optimized in such a way that they don't have to compile the universe to figure things out (see: C and C++) How did IntelliJ handle things? Did they do something analogous?

Argorak 7y ago
To my knowledge, IntelliJ has a base framework for handling many languages build towards an internal protocol.