3 ms·

How much is slow compilation caused by the compiler, and how much is it caused by the crates ecosystem and the many, many, many dependencies a project gains? I

by jmmv 3y ago

How much is slow compilation caused by the compiler, and how much is it caused by the crates ecosystem and the many, many, many dependencies a project gains? I'm genuinely curious because a couple of projects I have written are not that large per se, but I don't like their compilation times too much. What I have observed is that I have ended with 100-200 (indirect) dependencies, and compilation slows down the closer it gets to the "leaf" of the process. In other words: by the time compilation reaches my own code, my code seems to take much longer to compile relative to the intermediate (larger) crates. And it's not link time (I've switched to mold).

iknowstuff 3y ago
Most of those are trivially paralllelized so it's not a problem. It gets slower towards the end because e.g. your crate needs all of its dependencies in place before it can be compiled.
darksaints 3y ago
This brings up a potentially huge optimization waiting to happen. When you have a Rust dependency, you compile the whole crate before you can even start compiling your code. But because of the depth of the ecosystem, there is a lot of code packaged into crates that may not need to be compiled...the crate might only used for a single function. There are probably dozens of ways to do this, but a quick parse and analysis of the target code could eliminate a large amount of dependency compilation.
- epage 3y ago
  I do know there is vague / high-level interest in this kind of approach though I expect this would require a lot of work from the compiler, especially when it comes to figuring out what traits need to be compiled. Already, rust splits crate compilation into two passes where dependents are only blocked on the first pass and linking on the second. Maybe an intermediate is to do the first pass everywhere and then use that to decide what can be dropped from the second pass.
  - pas 3y ago
    traits or not, isn't it all compiled down to impls? and if some are not called, they can be dropped (dead code elimination basically)? of course this would require tracking the full call graph, right? (which the compiler might or might not do already; or might not be able to do due to memory constraints.)

by jmmv 3y ago

iknowstuff 3y ago

Most of those are trivially paralllelized so it's not a problem. It gets slower towards the end because e.g. your crate needs all of its dependencies in place before it can be compiled.

darksaints 3y ago

This brings up a potentially huge optimization waiting to happen. When you have a Rust dependency, you compile the whole crate before you can even start compiling your code. But because of the depth of the ecosystem, there is a lot of code packaged into crates that may not need to be compiled...the crate might only used for a single function. There are probably dozens of ways to do this, but a quick parse and analysis of the target code could eliminate a large amount of dependency compilation.

epage 3y ago
I do know there is vague / high-level interest in this kind of approach though I expect this would require a lot of work from the compiler, especially when it comes to figuring out what traits need to be compiled. Already, rust splits crate compilation into two passes where dependents are only blocked on the first pass and linking on the second. Maybe an intermediate is to do the first pass everywhere and then use that to decide what can be dropped from the second pass.
- pas 3y ago
  traits or not, isn't it all compiled down to impls? and if some are not called, they can be dropped (dead code elimination basically)? of course this would require tracking the full call graph, right? (which the compiler might or might not do already; or might not be able to do due to memory constraints.)