7 ms·

The rust compiler now has features that few or none C and C++ compilers have: incremental compilation within a single translation unit, pipelined compilation, m

The rust compiler now has features that few or none C and C++ compilers have: incremental compilation within a single translation unit, pipelined compilation, multi-threaded and lazy query-based compilation, ... Implementing each of these features have required whole program refactorings in a large-scale codebase performed by few individuals while hundreds of other developers where simultaneously evolving the software and implementing new features. Having been a C++ programmer for over 10 years, none of these refactorings would have payed off in C++ because of how time consuming it would have been to track all bugs introduced by them. Yet they do pay off in Rust because of my favourite Rust feature: the ability to refactor large-scale software without breaking anything: if a non-functional change compiles, "it works" for some definition of "works" that's much better than what most other languages give you (no memory unsafety, no data-races, no segfaults...). Very few programming languages have this feature, and no low-level programming language except for Rust has it. Software design is not an upfront-only task, and Rust let's you iterate on software design as you better understand the problem domain or the constraints and requirements change without having to rewrite things from scratch.

nindalf 7y ago
The rust compiler team continues to set ambitious goals. Last I checked they wanted to extract much of the compiler front end into independent crates that can be reused by the Language Server. That's a pretty daunting refactoring, but the way they're going seems like it's achievable.
- derefr 7y ago
  The one thing I’ve always wanted from a language runtime is for the data structures used by the compiler to be exposed in the stdlib. E.g. every compiler uses control flow graphs; so why doesn’t every language give me a batteries-included digraph ADT, that has every feature you’d need to implement a control flow graph, such that the compiler’s CFG is just a stdlib digraph instance? It’d be a great boon to writing your own compilers, interpreters, JITs and static-analysis tools in said language. (One language that does some of this is Erlang, but AFAIK the stdlib data structures like digraphs, sets-of-sets, etc. aren’t actually the ones the compiler uses, but are rather there for use by static verification tools like Dialyzer. Which means that the Erlang digraph doesn’t know how to topsort itself, even though there’s a module in the Erlang compiler application that does topsort on digraphs. Still feels like being a second-class citizen relative to the runtime’s favoured compiler.)
  - moomin 7y ago
    The thing is a) these structures change all the time and b) published APIs shouldn't change all the time. Which is why Erlang uses a public model and a private model and why most don't bother. Doesn't mean I wouldn't like it myself, but it is a genuine drag on continued compiler development.
    - derefr 7y ago
      There are business-logic data structures, and then there are formal data-structures from compiler theory. I only really want access to the latter. The nice thing about these formal data structures (or formal ADTs, I should say, though the papers themselves never tend to think of themselves as introducing an ADT) is that successive papers that find better algorithms, or make small changes to the data structure, pretty much always hold the set of operations required of the data structure constant (since that’s the only way to be able to compare these successive implementations on their performance/complexity.) So a “family” of papers about a given data structure is really a family of papers about an implicit ADT whose defined interface is the set of operations from the first paper that later papers compare themselves on. Personally, I feel like all “formal” data structures of this type could benefit from stdlib inclusion (e.g. interval trees; ropes; CRDTs; etc.) but that’s just a personal preference. But when a compiler is a built-in part of a runtime, with its modules already exposed as stdlib modules in a sense (being there as modules in a package namespace that you don’t have to retrieve, and which is always locked to the version of the compiler installed) such that people can actually reach in and use the compiler’s ADT for formal data-structure Foo—and people do!—and yet this growing dependency doesn’t cause the language maintainers to become worried about how they’re creating an implicit stdlib module by exposing the compiler internals in this way, but forgetting to track its usage for breaking changes... then I start to feel like it’s a bit ridiculous. To me, language projects should be run under the Microsoft philosophy: you don’t get to decide what your stable interfaces are. If your customers are all using and depending on feature X, then feature X should be treated as a stable interface. It’s up to you as a language maintainer to discover your stable interface requirements, and promote the things people are “treating as stable”, into being managed as stable in a language-evolutionary project-management sense.
pjmlp 7y ago
Visual C++ is part of those few. Rust is still far from Delphi, Eiffel, .NET Native experience though. Although it is great that it keeps improving.
- jacobush 7y ago
  Visual C++ may be great, but in practice, if you use it with MSBuild, you can't even build modules in parallel without manually tuning how many cores to allocate.
  - pjmlp 7y ago
    Might be, but it already helps and there is IncrediBuild as part of the package.
- fluffything 7y ago
  Is there a blog post about Visual C++ being multi-threaded end-to-end ? (e.g. doing parsing, type-checking, etc. of a single TU in parallel using all cores ?) That would be super interesting to read because it is mainly a C++ compiler and some C++ features like two-phase lookup and macros make it quite hard to do things in parallel. You have to do things in a certain order but I suppose that if its query-based as well it will work. The only option I know is /CGTHREADS but that only uses multiple-threads for optimizations and code generation which is something that the Rust compiler has been able to do for a very long time (e.g. there these are called codegen-units and LLVM supports these so it is quite trivial for a frontend to do so as well).
  - zone411 7y ago
    As the documentation says: "The /MP (Build with Multiple Processes) compiler flag specifies the number of cl.exe processes that simultaneously compile the source files. The /cgthreads option specifies the number of threads used by each cl.exe process." I've never used /CGTHREADS but /MP is working well (except for the precompliled header file).
    - 7y ago
tkoskine 7y ago
> ... the ability to refactor large-scale software without breaking anything ... and no low-level programming language except for Rust has it. Ada has had the same characteristics since '83. Thanks to strict typing (among other features), Ada programmers have enjoyed "safe" code refactoring for decades. But it is nice that new languages like Rust are finally picking similar ideas and design choices.
- sfkdjf9j3j 7y ago
  Ada languished for years with expensive, proprietary compilers and a community that largely ignored open source culture. Is it any surprise that Rust is having more success breaking into the modern mainstream software world?
  - hu3 7y ago
    I fail to see how your parent implied surprise.
    - jplayer01 7y ago
      I don't feel the comment was made in an adversarial way. Just pointing out some reasons why Ada didn't become widespread. There are plenty of decent languages with great high-level constructs and features that failed to gain/maintain meaningful industry penetration after C hit the stage. I'd say Ada is just another victim of that success, along with the issues he mentions. It's unfortunate, but that's how it is. Maybe Rust can reach a wider audience in a way Ada was unable to, while providing similar safety guarantees.
mrec 7y ago
> Yet they do pay off in Rust because of my favourite Rust feature: the ability to refactor large-scale software without breaking anything I think crater also deserves some credit here - given how much Rust source is in Cargo, it's very useful to be able to run a refactored compiler against all those packages and see which don't compile or start failing their unit tests. [1] https://github.com/rust-lang/crater https://github.com/rust-lang/crater
winter_blue 7y ago
Wow, this is really encouraging. I was debating whether to use Rust or Kotlin for developing a new programming language, and after reading your comment, I'm leaning a little bit closer to Rust. I had previously broken down the pros for Kotlin to 6 points: 1. You get access to all of the JVM libraries. 2. You don't have to work within the confines of the borrow checker. 3. Kotlin “Common” targets 3 platforms: LLVM, JVM, and JS (whereas, Rust only targets LLVM). 4. Kotlin is probably a more terse language, and suitable for doing algorithm / problem-solving interviews, and therefore a good one to be fluent in. 5. Kotlin's greater industry traction means that it might be more useful professionally. Outside of Android, I've also heard of servers/back-ends being written in Kotlin. 6. ANTLR is well-documented, compared to LALRPOP (the best existing Rust parser generator), and ANTLR targets/generates code for several mainstream languages (versus only Rust with LALRPOP). ANTLR is also probably a more useful skill to have for future jobs/projects. But despite of all the pluses of using Kotlin, I'm still leaning a bit closer to Rust, because of all the good things I'm hearing about it.

nindalf 7y ago

The rust compiler team continues to set ambitious goals. Last I checked they wanted to extract much of the compiler front end into independent crates that can be reused by the Language Server. That's a pretty daunting refactoring, but the way they're going seems like it's achievable.

derefr 7y ago
The one thing I’ve always wanted from a language runtime is for the data structures used by the compiler to be exposed in the stdlib. E.g. every compiler uses control flow graphs; so why doesn’t every language give me a batteries-included digraph ADT, that has every feature you’d need to implement a control flow graph, such that the compiler’s CFG is just a stdlib digraph instance? It’d be a great boon to writing your own compilers, interpreters, JITs and static-analysis tools in said language. (One language that does some of this is Erlang, but AFAIK the stdlib data structures like digraphs, sets-of-sets, etc. aren’t actually the ones the compiler uses, but are rather there for use by static verification tools like Dialyzer. Which means that the Erlang digraph doesn’t know how to topsort itself, even though there’s a module in the Erlang compiler application that does topsort on digraphs. Still feels like being a second-class citizen relative to the runtime’s favoured compiler.)
- moomin 7y ago
  The thing is a) these structures change all the time and b) published APIs shouldn't change all the time. Which is why Erlang uses a public model and a private model and why most don't bother. Doesn't mean I wouldn't like it myself, but it is a genuine drag on continued compiler development.
  - derefr 7y ago
    There are business-logic data structures, and then there are formal data-structures from compiler theory. I only really want access to the latter. The nice thing about these formal data structures (or formal ADTs, I should say, though the papers themselves never tend to think of themselves as introducing an ADT) is that successive papers that find better algorithms, or make small changes to the data structure, pretty much always hold the set of operations required of the data structure constant (since that’s the only way to be able to compare these successive implementations on their performance/complexity.) So a “family” of papers about a given data structure is really a family of papers about an implicit ADT whose defined interface is the set of operations from the first paper that later papers compare themselves on. Personally, I feel like all “formal” data structures of this type could benefit from stdlib inclusion (e.g. interval trees; ropes; CRDTs; etc.) but that’s just a personal preference. But when a compiler is a built-in part of a runtime, with its modules already exposed as stdlib modules in a sense (being there as modules in a package namespace that you don’t have to retrieve, and which is always locked to the version of the compiler installed) such that people can actually reach in and use the compiler’s ADT for formal data-structure Foo—and people do!—and yet this growing dependency doesn’t cause the language maintainers to become worried about how they’re creating an implicit stdlib module by exposing the compiler internals in this way, but forgetting to track its usage for breaking changes... then I start to feel like it’s a bit ridiculous. To me, language projects should be run under the Microsoft philosophy: you don’t get to decide what your stable interfaces are. If your customers are all using and depending on feature X, then feature X should be treated as a stable interface. It’s up to you as a language maintainer to discover your stable interface requirements, and promote the things people are “treating as stable”, into being managed as stable in a language-evolutionary project-management sense.

pjmlp 7y ago

Visual C++ is part of those few. Rust is still far from Delphi, Eiffel, .NET Native experience though. Although it is great that it keeps improving.

jacobush 7y ago
Visual C++ may be great, but in practice, if you use it with MSBuild, you can't even build modules in parallel without manually tuning how many cores to allocate.
- pjmlp 7y ago
  Might be, but it already helps and there is IncrediBuild as part of the package.
fluffything 7y ago
Is there a blog post about Visual C++ being multi-threaded end-to-end ? (e.g. doing parsing, type-checking, etc. of a single TU in parallel using all cores ?) That would be super interesting to read because it is mainly a C++ compiler and some C++ features like two-phase lookup and macros make it quite hard to do things in parallel. You have to do things in a certain order but I suppose that if its query-based as well it will work. The only option I know is /CGTHREADS but that only uses multiple-threads for optimizations and code generation which is something that the Rust compiler has been able to do for a very long time (e.g. there these are called codegen-units and LLVM supports these so it is quite trivial for a frontend to do so as well).
- zone411 7y ago
  As the documentation says: "The /MP (Build with Multiple Processes) compiler flag specifies the number of cl.exe processes that simultaneously compile the source files. The /cgthreads option specifies the number of threads used by each cl.exe process." I've never used /CGTHREADS but /MP is working well (except for the precompliled header file).
  - 7y ago

tkoskine 7y ago

> ... the ability to refactor large-scale software without breaking anything ... and no low-level programming language except for Rust has it. Ada has had the same characteristics since '83. Thanks to strict typing (among other features), Ada programmers have enjoyed "safe" code refactoring for decades. But it is nice that new languages like Rust are finally picking similar ideas and design choices.

sfkdjf9j3j 7y ago
Ada languished for years with expensive, proprietary compilers and a community that largely ignored open source culture. Is it any surprise that Rust is having more success breaking into the modern mainstream software world?
- hu3 7y ago
  I fail to see how your parent implied surprise.
  - jplayer01 7y ago
    I don't feel the comment was made in an adversarial way. Just pointing out some reasons why Ada didn't become widespread. There are plenty of decent languages with great high-level constructs and features that failed to gain/maintain meaningful industry penetration after C hit the stage. I'd say Ada is just another victim of that success, along with the issues he mentions. It's unfortunate, but that's how it is. Maybe Rust can reach a wider audience in a way Ada was unable to, while providing similar safety guarantees.

mrec 7y ago

> Yet they do pay off in Rust because of my favourite Rust feature: the ability to refactor large-scale software without breaking anything I think crater also deserves some credit here - given how much Rust source is in Cargo, it's very useful to be able to run a refactored compiler against all those packages and see which don't compile or start failing their unit tests. [1] https://github.com/rust-lang/crater https://github.com/rust-lang/crater

winter_blue 7y ago

Wow, this is really encouraging. I was debating whether to use Rust or Kotlin for developing a new programming language, and after reading your comment, I'm leaning a little bit closer to Rust. I had previously broken down the pros for Kotlin to 6 points: 1. You get access to all of the JVM libraries. 2. You don't have to work within the confines of the borrow checker. 3. Kotlin “Common” targets 3 platforms: LLVM, JVM, and JS (whereas, Rust only targets LLVM). 4. Kotlin is probably a more terse language, and suitable for doing algorithm / problem-solving interviews, and therefore a good one to be fluent in. 5. Kotlin's greater industry traction means that it might be more useful professionally. Outside of Android, I've also heard of servers/back-ends being written in Kotlin. 6. ANTLR is well-documented, compared to LALRPOP (the best existing Rust parser generator), and ANTLR targets/generates code for several mainstream languages (versus only Rust with LALRPOP). ANTLR is also probably a more useful skill to have for future jobs/projects. But despite of all the pluses of using Kotlin, I'm still leaning a bit closer to Rust, because of all the good things I'm hearing about it.