42 ms·

Why is my Rust build so slow?

298 points by jacobwg 5y ago

_ma_rit_231 5y ago
The long compile times are actually one of Rust's problems I could live with. Having to depend on so many third party libs is a much bigger issue for long term maintainability and security. Also the time choosing the libs is not neglectible. At work we currently developing a mid size project (CLI and HTTP API) with Go and the standard lib is simply amazing.
- OtomotO 5y ago
  The difference between unmaintained code in a stdlib and unmaintained code in a third party lib is artificial. I want the stdlib to be as lean as possible. Also, as you just read, crates are compilation modules, hence generally this means more crates, lead to faster compile times.
  - pjmlp 5y ago
    Until cargo supports binary dependencies this is only true for incremental builds. Microsoft has to ship binary libraries with build scripts to work around it, https://github.com/microsoft/windows-rs/tree/master/.windows https://github.com/microsoft/windows-rs/tree/master/.windows
- brabel 5y ago
  Long compile times and having lots of small crates in your dependency tree, surprisingly, are related in that, as the author mentions in the article, splitting libraries into separate crates makes the compilation of each individual crate much faster because crates you depend on are compiled in parallel, and don't ever need to be re-compiled after the first compilation (so hot builds become much faster) while the current crate being compiled becomes naturally much smaller, hence faster to compile, when you pull chunks of it into other crates.
marmada 5y ago
I loved this article! Super informative. Very interesting that the author managed to hit a pathological case in rustc.
- masklinn 5y ago
  > Very interesting that the author managed to hit a pathological case in rustc. FWIW lots of people hit this one, because entire pretty-base libraries got impacted by the regression: it hit everything which makes significant use of deeply nested "decorator" types, like iterator, future, and stream combinators. Lime hit it because Warp is a big, big user of large nested decorator types (its entire routing and response system is based on that), pretty much every user of Warp with more than a handful of routes in their project hit this issue.
charcircuit 5y ago
Rust's slow compiles are such a turn off for me. Like why does it take tens of seconds to recompile when I am just changing a single number in a file? Does it really need to waste so much of my time to change a single byte in the output binary?
- jiggawatts 5y ago
  I believe incremental mode is still off by default.
  - CameronNemo 5y ago
    Not for debug builds, according to the article.
- yjftsjthsd-h 5y ago
  I think speed is separate from caching, albeit overlapping. If the compiler can see exactly what changes your code will have in the final binary, then it can do very very little work. However, it would also suffice for the compiler to just be so fast that it can do a complete compile from zero to finished in some acceptable time (tcc and I think Go favor this). Those are different goals, or at least different ways of achieving the desired outcome.
- sjakwjsjsisj 5y ago
  Use bazel? Individual files won't be faster but it's been a wonder for whole project compilations
  - charcircuit 5y ago
    This could be interesting to look into. It looks like there already exists a project that can generate BUILD files for external crates. My 2 main gripes with Bazel was that it was a pain having to rewrite the build system for all of my dependencies and that it's claim of being reproducible is weak in the sense that there are not even warnings when you use resources from the base system (eg. compilers can include files from the base system which may not be the same / exist on another) (this is a problem since I wanted to use a header from the dependency I built with Bazel and not something already on my system)
CameronNemo 5y ago
This article is fairly strange. The author keeps changing settings to make the release profile more like a debug profile. Wouldn't I want LTO for releases? Also this is a really long article with a lot of cool tricks and interesting information. Like the author's binary crate, they might consider splitting this article apart. Or at least adding a TOC!
- fulafel 5y ago
  You might want fat LTO for releases if the the lost dev productivity and second order effects from that are worth the gains, often they are not.
- akavel 5y ago
  As I read it, the flag changes are to find the source of the slowdown. I.e. if disabling LTO for a moment would improve the compilation speed drastically, then I probably still want it enabled but may now want to see if there are ways to "please" the LTO, to massage the input to be somehow more LTO-friendly and allow it to work faster - or find some other ways to optimize LTO (like how they experiment with alternative linkers in other parts of the article). It helps to understand whether the biggest gains can be had from focusing on optimizing this particular area. Also it helps quickly and roughly establish what's the "best case" improvement they can probably achieve, as presumably the debug-build set of options is bent on compilation speed. Still, in the end the biggest performance gain seems to have been found faaar away from LTO in this case.
- saurik 5y ago
  (Note: I am a long time systems software developer but I don't use Rust, so while this made perfect sense to me maybe it shouldn't have due to some Rust thing I don't know about.) I'm guessing the issue is how there is really a stage between "debug" and "release" where you need the code to run fast enough to be worth testing and using but you don't need to eek out then final 0.5% of performance by cranking all the settings. When working on such projects you only drop down to a "debug build" if you end up entirely screwed trying to figure out why something is broken because the optimizer has messed up the debugger, and you care OK with the lower performance as you've already isolated the issue.
howdydoo 5y ago
I've always wondered why useful flags like `-Z timings` are not available on stable. I guess they're worried about the output changing in a future stable release, but I don't think anyone would expect the reports to look exactly the same and list exactly the same compilation phases, etc, for all eternity. I think that for diagnostic features, people would be just fine with a loose stability guarantee that allows the output to be improved in the future.
- est31 5y ago
  > I think that for diagnostic features, people would be just fine with a loose stability guarantee The issue with this is that the output might be parsed by downstream tools that assume a special format. They might break by a change in the output.
  - spockz 5y ago
    It sounds like these tools should be using some machine format. Perhaps json or something as opposed to human readable strings. Then again, also the layout of the json might change but at least that can potentially be done in a backwards compatible manner.
- JoshTriplett 5y ago
  I've been working on stabilizing some options like this, with exactly that approach: the functionality is stable but the exact details aren't. For instance, instrument-coverage will work like that, generating coverage data that requires the current version of LLVM coverage tools to work with. timings doesn't seem especially hard to stabilize; I'll take a look at it.
  - est31 5y ago
    But you can't just say that the flag is stable while the output may change. Third party tools might rely on the output. This is okay for things read by humans (say the error messages) but fails for things that tools are parsing, like data needed for code coverage tools. Stabilization of instrument-coverage, as proposed, is a bad idea in my opinion. I'm sad that the concerns have not been addressed: https://github.com/rust-lang/rust/pull/90132#issuecomment-949736508 https://github.com/rust-lang/rust/pull/90132#issuecomment-94...
PragmaticPulp 5y ago
I had a hunch that Warp would be the answer when I started reading. I hit the same issue and worked around it the same way (boxing). My build times also went from 1-2 minutes down to a couple seconds. Generally speaking, most of my Rust work has been reasonably fast on modern hardware. Waiting 3-5 seconds for incremental builds isn't a big deal. I've only seen the excessive compile times with cold starts or in weird edge cases like the Warp issue.
twblalock 5y ago
This much complexity and the tribal knowledge required to bypass it, so early in the the life of a programming language, is a really bad sign. It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users. You might react to this by saying that slow build times are not a big deal. That's a mistake -- iteration is key to productivity and other languages do it better.
- lpapez 5y ago
  That is because the languages are different and choose different tradeoffs. The tradeoffs might make a language inapplicable in your domain. Rust has always been marketed as a "systems programming language". C++ is Rust's closes competitor in this domain space and it suffers from terribly slow compiling as well. I would say that due to headers, massive portability baggage and a lack of a standard build system, compiling C++ is even worse. And that is a language that has been around for 30+ years. I agree that build times are important (i.e. PR build takes us 30 minutes which is insane), but sometimes you don't really have a choice. When you do have a choice, then by all means reap the benefits of fast compilation that languages such as Go or C can offer.
  - saurik 5y ago
    C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.
    - codeflo 5y ago
      That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.
menaerus 5y ago
Whenever I read articles like this one, it leaves me wondering in what parallel reality do I live in. Or I could put it differently and say that authors making blogposts complaining about full rebuild cycle taking staggering 2 minutes live in a pink bubble. Everyday reality in somewhat complex industry-placed product is 10x that if you're lucky. More often than not it's few hours before you're able to build the whole thing.
- lordnacho 5y ago
  Same here, I don't get what the complaint is. A friend at a major bank says they used to take 7 hours to build their C++ trading system before someone cleaned it up. 2m to build a release and a few seconds to build incremental seems fine to me. Most of the time when I'm coding I just want the compiler to tell me if the syntax is wrong, and it is pretty snappy with rust.
  - xbar 5y ago
    I do not think that's the point. Let's say you had a build that took 7 hours but if you took all the steps the author did, you would be able to build it in 30 minutes locally for debugging. Would that be interesting? That's about the amount of relative improvement the author saw. Size of input varies, finding solutions to unnecessarily long compiles in rust are useful across many of them.
- masklinn 5y ago
  I think it helps to have some of the not-exactly-stated background: their build did not originally take 2mn, it suddenly shot up to this from around 20s seemingly overnight, so they investigated what happened, and as that's lime they also produced a nice article out of it. That's my assumption anyway because that's about what I went through, minus the blog post: had a project I was working on, at one point I noticed the clean builds were getting really long, we're talking >10mn. Initially I figured it might be Serde (as that project is really serde-heavy) and the compile time had crept up on me unnoticed, so I moved all the serialized and deserialized types in a sub-crate in case that did anything... and it did not. Then I busted out the -Ztimings getting about the same result Lime did (ungodly amount of time spent producing the binary for no clear reason), tried fiddling with some compile-time options (went nowhere near as deep there, I looked at -Z self-profile fairly quickly IIRC), finally found the evaluate_obligation stuff. From there I didn't bust out the profiler to see what was what, instead through I don't quite remember which chain got me to issue 91598 (https://github.com/rust-lang/rust/issues/91598 https://github.com/rust-lang/rust/issues/91598). Added a follow to the issue and various PRs proposed for it, and locked the project to 1.56.
lordnacho 5y ago
Very long but interesting article. What's the reason that boxing a few things shortens the compile time?
- alexchamberlain 5y ago
  Warp uses a lot of generics, which can produce extremely deep and long types. In turn, these cause a particular compile stage to take a long time (> 40 seconds). Boxing disguises/splits the chain by taking a pointer to a trait object, which doesn't require the same depth of checks. (This is especially important if the algorithm is superlinear, which is implied but not stated in the article.)
  - Smaug123 5y ago
    (The underlying issue is https://github.com/rust-lang/rust/issues/91598 https://github.com/rust-lang/rust/issues/91598.)
  - estebank 5y ago
    The problem is not the length of the types, but rather the number of "obligations" introduced by each type. If you have Foo<Bar<Baz>>, and each one of those types has a single trait bound, the resulting type now has >3 obligations to evaluate (> because you also have Sized and WF checks for them as well). The deeper the type depth, the more potential obligations you end up with, which have in some cases can have exponential evaluation. We have caching in place to only evaluate a given obligation once, but it can't always kick in. I'm sure there are still things we can improve there, though.
    - lordnacho 5y ago
      Still not sure I get it. If I box a generic, why does the time to evaluate the bounds change?
      - estebank 5y ago
        If you have T and box it, a new type Box<T> is created. The compiler not only has to do typeck to make sure that you're using Box<T> in both sides of an assignment, it also has to evaluate the bound on Box (in this case it's only that it is Well-Formed, an internal concept, because Box accepts unsized types, struct Box<A: ?Sized>), but also evaluate all the bounds on T. If T: Foo, and the type you used for T is Bar<X: Qux>, then by the time rustc actually has to confirm the type Box<Bar<X>> is correct, it has to check that Bar<X>: Foo and X: Qux, but you never wrote those close to the Box, these flow from other definitions. Multiply for every type with an explicit or implicit where clause (T: Foo, where <A as B>::C: D, associated types, etc.).
bullen 5y ago
Ok, I didn't read all that but that is 100W of power. My solution to be able to work on a 5W Raspberry 4 is to use C (arrays for cache misses), compile with GCC to a .so and hot-deploy that into my engine. That way I have a zero dependencies engine (only stb_ttf and kuba_zip) that compiles from scratch in 30 seconds and the game .so takes ~1 second. I don't even have to use a build system or many cores to compile. Simple and low power. Downvotes without commenting, proves my point.
holoduke 5y ago
I came from java and compared to that even our slowest rust build time is faster than I have ever experienced in java. I remember we had a Java springboot application who had a buildtime of 8 minutes on my 2015 MacBook. I am so happy I don't have to deal with this anymore.
- Ygg2 5y ago
  Damn. Are you me? I had the same experience.
- xpressvideoz 5y ago
  That's more to do with Spring, not Java. The compilation time of Java is infinitely faster than that of Rust.
  - Ygg2 5y ago
    Well, in this case. It had to do with Warp hitting a regression in compiler. Spring is one of most if not THE ubiquitous piece of Java software. All Java shop I worked for, had Spring Comparing Spring+Java to Rust+Warp is imo fair game.
    - pjmlp 5y ago
      Meanwhile others of us have never used Spring in production, I have been on the JEE camp since the BEA and WebSphere 5.1 days.
  - dragonelite 5y ago
    For pretty 90 plus % of jobs you're pretty much confined to the Spring ecosystem. Same with C# and asp.net separating the two would be really weird.
    - pjmlp 5y ago
      Hello from the 10%.
  - 5y ago
Ono-Sendai 5y ago
tldr
- jakobnissen 5y ago
  A rust project with 420 deps but only 7k LOC suddenly took 2 minutes to compile with cache. Turns out it's mostly a compiler regression that is already fixed. The blog shows all kinds of profiling and analysis of the Rust compiler.
flohofwoe 5y ago
Very interesting and detailed article how to profile the Rust toolchain, but TBH, after the reveal that the project uses several hundred dependencies my first thought was: "well, there's your problem". One thing or another will always be broken in such a complex scenario.
EVa5I7bHFq9mnYK 5y ago
Split your code into separately compiled modules (DLLs or whatever it is these days) and minimize interface surface between those modules.
marco_craveiro 5y ago
This is a great post, thanks for sharing.
swsieber 5y ago
Another article about rust diagnosing rust compile times: https://matklad.github.io/2021/09/04/fast-rust-builds.html https://matklad.github.io/2021/09/04/fast-rust-builds.html
Klasiaster 5y ago
Instead of using sccache and turning off incremental compilation it could help to change the approach and retain the build folder from the main branch CI builds for all other builds, since both sccache and incremental compilation are kind of overlapping with incremental compilation being the integrated solution while sccache seems to be more like a workaround?
zenlot 5y ago
"In another language, say, C or C++, we might write a Makefile by hand. What? No. Nobody does that anymore." Oh these Rustaceans or how they call themselves. Look at the gaming industry, if you're playing any AAA/AA game it's most likely C++, there's some market share on C# (where Unity is used), but still the whole gaming industry is C++, and it's bigger than movie/music industries combined. So yes, people "do that", and it's not going to change anytime soon.
- alexchamberlain 5y ago
  I think the OP meant no one writes Makefiles manually; you might use CMake or ninja, for example.
  - jjgreen 5y ago
    What an odd (and demonstrably false) thing for the OP to say, Makefiles are a delight.
jl2718 5y ago
I’ve always wondered why compiled languages can’t fall back to interpreted execution for debugging.

298 points by jacobwg 5y ago

The long compile times are actually one of Rust's problems I could live with. Having to depend on so many third party libs is a much bigger issue for long term maintainability and security. Also the time choosing the libs is not neglectible. At work we currently developing a mid size project (CLI and HTTP API) with Go and the standard lib is simply amazing.

OtomotO 5y ago
The difference between unmaintained code in a stdlib and unmaintained code in a third party lib is artificial. I want the stdlib to be as lean as possible. Also, as you just read, crates are compilation modules, hence generally this means more crates, lead to faster compile times.
- pjmlp 5y ago
  Until cargo supports binary dependencies this is only true for incremental builds. Microsoft has to ship binary libraries with build scripts to work around it, https://github.com/microsoft/windows-rs/tree/master/.windows https://github.com/microsoft/windows-rs/tree/master/.windows
brabel 5y ago
Long compile times and having lots of small crates in your dependency tree, surprisingly, are related in that, as the author mentions in the article, splitting libraries into separate crates makes the compilation of each individual crate much faster because crates you depend on are compiled in parallel, and don't ever need to be re-compiled after the first compilation (so hot builds become much faster) while the current crate being compiled becomes naturally much smaller, hence faster to compile, when you pull chunks of it into other crates.

marmada 5y ago

I loved this article! Super informative. Very interesting that the author managed to hit a pathological case in rustc.

masklinn 5y ago
> Very interesting that the author managed to hit a pathological case in rustc. FWIW lots of people hit this one, because entire pretty-base libraries got impacted by the regression: it hit everything which makes significant use of deeply nested "decorator" types, like iterator, future, and stream combinators. Lime hit it because Warp is a big, big user of large nested decorator types (its entire routing and response system is based on that), pretty much every user of Warp with more than a handful of routes in their project hit this issue.

charcircuit 5y ago

Rust's slow compiles are such a turn off for me. Like why does it take tens of seconds to recompile when I am just changing a single number in a file? Does it really need to waste so much of my time to change a single byte in the output binary?

jiggawatts 5y ago
I believe incremental mode is still off by default.
- CameronNemo 5y ago
  Not for debug builds, according to the article.
yjftsjthsd-h 5y ago
I think speed is separate from caching, albeit overlapping. If the compiler can see exactly what changes your code will have in the final binary, then it can do very very little work. However, it would also suffice for the compiler to just be so fast that it can do a complete compile from zero to finished in some acceptable time (tcc and I think Go favor this). Those are different goals, or at least different ways of achieving the desired outcome.
sjakwjsjsisj 5y ago
Use bazel? Individual files won't be faster but it's been a wonder for whole project compilations
- charcircuit 5y ago
  This could be interesting to look into. It looks like there already exists a project that can generate BUILD files for external crates. My 2 main gripes with Bazel was that it was a pain having to rewrite the build system for all of my dependencies and that it's claim of being reproducible is weak in the sense that there are not even warnings when you use resources from the base system (eg. compilers can include files from the base system which may not be the same / exist on another) (this is a problem since I wanted to use a header from the dependency I built with Bazel and not something already on my system)

CameronNemo 5y ago

This article is fairly strange. The author keeps changing settings to make the release profile more like a debug profile. Wouldn't I want LTO for releases? Also this is a really long article with a lot of cool tricks and interesting information. Like the author's binary crate, they might consider splitting this article apart. Or at least adding a TOC!

fulafel 5y ago
You might want fat LTO for releases if the the lost dev productivity and second order effects from that are worth the gains, often they are not.
akavel 5y ago
As I read it, the flag changes are to find the source of the slowdown. I.e. if disabling LTO for a moment would improve the compilation speed drastically, then I probably still want it enabled but may now want to see if there are ways to "please" the LTO, to massage the input to be somehow more LTO-friendly and allow it to work faster - or find some other ways to optimize LTO (like how they experiment with alternative linkers in other parts of the article). It helps to understand whether the biggest gains can be had from focusing on optimizing this particular area. Also it helps quickly and roughly establish what's the "best case" improvement they can probably achieve, as presumably the debug-build set of options is bent on compilation speed. Still, in the end the biggest performance gain seems to have been found faaar away from LTO in this case.
saurik 5y ago
(Note: I am a long time systems software developer but I don't use Rust, so while this made perfect sense to me maybe it shouldn't have due to some Rust thing I don't know about.) I'm guessing the issue is how there is really a stage between "debug" and "release" where you need the code to run fast enough to be worth testing and using but you don't need to eek out then final 0.5% of performance by cranking all the settings. When working on such projects you only drop down to a "debug build" if you end up entirely screwed trying to figure out why something is broken because the optimizer has messed up the debugger, and you care OK with the lower performance as you've already isolated the issue.

howdydoo 5y ago

I've always wondered why useful flags like `-Z timings` are not available on stable. I guess they're worried about the output changing in a future stable release, but I don't think anyone would expect the reports to look exactly the same and list exactly the same compilation phases, etc, for all eternity. I think that for diagnostic features, people would be just fine with a loose stability guarantee that allows the output to be improved in the future.

est31 5y ago
> I think that for diagnostic features, people would be just fine with a loose stability guarantee The issue with this is that the output might be parsed by downstream tools that assume a special format. They might break by a change in the output.
- spockz 5y ago
  It sounds like these tools should be using some machine format. Perhaps json or something as opposed to human readable strings. Then again, also the layout of the json might change but at least that can potentially be done in a backwards compatible manner.
JoshTriplett 5y ago
I've been working on stabilizing some options like this, with exactly that approach: the functionality is stable but the exact details aren't. For instance, instrument-coverage will work like that, generating coverage data that requires the current version of LLVM coverage tools to work with. timings doesn't seem especially hard to stabilize; I'll take a look at it.
- est31 5y ago
  But you can't just say that the flag is stable while the output may change. Third party tools might rely on the output. This is okay for things read by humans (say the error messages) but fails for things that tools are parsing, like data needed for code coverage tools. Stabilization of instrument-coverage, as proposed, is a bad idea in my opinion. I'm sad that the concerns have not been addressed: https://github.com/rust-lang/rust/pull/90132#issuecomment-949736508 https://github.com/rust-lang/rust/pull/90132#issuecomment-94...

PragmaticPulp 5y ago

I had a hunch that Warp would be the answer when I started reading. I hit the same issue and worked around it the same way (boxing). My build times also went from 1-2 minutes down to a couple seconds. Generally speaking, most of my Rust work has been reasonably fast on modern hardware. Waiting 3-5 seconds for incremental builds isn't a big deal. I've only seen the excessive compile times with cold starts or in weird edge cases like the Warp issue.

twblalock 5y ago

This much complexity and the tribal knowledge required to bypass it, so early in the the life of a programming language, is a really bad sign. It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users. You might react to this by saying that slow build times are not a big deal. That's a mistake -- iteration is key to productivity and other languages do it better.

lpapez 5y ago
That is because the languages are different and choose different tradeoffs. The tradeoffs might make a language inapplicable in your domain. Rust has always been marketed as a "systems programming language". C++ is Rust's closes competitor in this domain space and it suffers from terribly slow compiling as well. I would say that due to headers, massive portability baggage and a lack of a standard build system, compiling C++ is even worse. And that is a language that has been around for 30+ years. I agree that build times are important (i.e. PR build takes us 30 minutes which is insane), but sometimes you don't really have a choice. When you do have a choice, then by all means reap the benefits of fast compilation that languages such as Go or C can offer.
- saurik 5y ago
  C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.
  - codeflo 5y ago
    That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.

menaerus 5y ago

Whenever I read articles like this one, it leaves me wondering in what parallel reality do I live in. Or I could put it differently and say that authors making blogposts complaining about full rebuild cycle taking staggering 2 minutes live in a pink bubble. Everyday reality in somewhat complex industry-placed product is 10x that if you're lucky. More often than not it's few hours before you're able to build the whole thing.

lordnacho 5y ago
Same here, I don't get what the complaint is. A friend at a major bank says they used to take 7 hours to build their C++ trading system before someone cleaned it up. 2m to build a release and a few seconds to build incremental seems fine to me. Most of the time when I'm coding I just want the compiler to tell me if the syntax is wrong, and it is pretty snappy with rust.
- xbar 5y ago
  I do not think that's the point. Let's say you had a build that took 7 hours but if you took all the steps the author did, you would be able to build it in 30 minutes locally for debugging. Would that be interesting? That's about the amount of relative improvement the author saw. Size of input varies, finding solutions to unnecessarily long compiles in rust are useful across many of them.
masklinn 5y ago
I think it helps to have some of the not-exactly-stated background: their build did not originally take 2mn, it suddenly shot up to this from around 20s seemingly overnight, so they investigated what happened, and as that's lime they also produced a nice article out of it. That's my assumption anyway because that's about what I went through, minus the blog post: had a project I was working on, at one point I noticed the clean builds were getting really long, we're talking >10mn. Initially I figured it might be Serde (as that project is really serde-heavy) and the compile time had crept up on me unnoticed, so I moved all the serialized and deserialized types in a sub-crate in case that did anything... and it did not. Then I busted out the -Ztimings getting about the same result Lime did (ungodly amount of time spent producing the binary for no clear reason), tried fiddling with some compile-time options (went nowhere near as deep there, I looked at -Z self-profile fairly quickly IIRC), finally found the evaluate_obligation stuff. From there I didn't bust out the profiler to see what was what, instead through I don't quite remember which chain got me to issue 91598 (https://github.com/rust-lang/rust/issues/91598 https://github.com/rust-lang/rust/issues/91598). Added a follow to the issue and various PRs proposed for it, and locked the project to 1.56.

lordnacho 5y ago

Very long but interesting article. What's the reason that boxing a few things shortens the compile time?

alexchamberlain 5y ago
Warp uses a lot of generics, which can produce extremely deep and long types. In turn, these cause a particular compile stage to take a long time (> 40 seconds). Boxing disguises/splits the chain by taking a pointer to a trait object, which doesn't require the same depth of checks. (This is especially important if the algorithm is superlinear, which is implied but not stated in the article.)
- Smaug123 5y ago
  (The underlying issue is https://github.com/rust-lang/rust/issues/91598 https://github.com/rust-lang/rust/issues/91598.)
- estebank 5y ago
  The problem is not the length of the types, but rather the number of "obligations" introduced by each type. If you have Foo<Bar<Baz>>, and each one of those types has a single trait bound, the resulting type now has >3 obligations to evaluate (> because you also have Sized and WF checks for them as well). The deeper the type depth, the more potential obligations you end up with, which have in some cases can have exponential evaluation. We have caching in place to only evaluate a given obligation once, but it can't always kick in. I'm sure there are still things we can improve there, though.
  - lordnacho 5y ago
    Still not sure I get it. If I box a generic, why does the time to evaluate the bounds change?
    - estebank 5y ago
      If you have T and box it, a new type Box<T> is created. The compiler not only has to do typeck to make sure that you're using Box<T> in both sides of an assignment, it also has to evaluate the bound on Box (in this case it's only that it is Well-Formed, an internal concept, because Box accepts unsized types, struct Box<A: ?Sized>), but also evaluate all the bounds on T. If T: Foo, and the type you used for T is Bar<X: Qux>, then by the time rustc actually has to confirm the type Box<Bar<X>> is correct, it has to check that Bar<X>: Foo and X: Qux, but you never wrote those close to the Box, these flow from other definitions. Multiply for every type with an explicit or implicit where clause (T: Foo, where <A as B>::C: D, associated types, etc.).

bullen 5y ago

Ok, I didn't read all that but that is 100W of power. My solution to be able to work on a 5W Raspberry 4 is to use C (arrays for cache misses), compile with GCC to a .so and hot-deploy that into my engine. That way I have a zero dependencies engine (only stb_ttf and kuba_zip) that compiles from scratch in 30 seconds and the game .so takes ~1 second. I don't even have to use a build system or many cores to compile. Simple and low power. Downvotes without commenting, proves my point.

holoduke 5y ago

I came from java and compared to that even our slowest rust build time is faster than I have ever experienced in java. I remember we had a Java springboot application who had a buildtime of 8 minutes on my 2015 MacBook. I am so happy I don't have to deal with this anymore.

Ygg2 5y ago
Damn. Are you me? I had the same experience.
xpressvideoz 5y ago
That's more to do with Spring, not Java. The compilation time of Java is infinitely faster than that of Rust.
- Ygg2 5y ago
  Well, in this case. It had to do with Warp hitting a regression in compiler. Spring is one of most if not THE ubiquitous piece of Java software. All Java shop I worked for, had Spring Comparing Spring+Java to Rust+Warp is imo fair game.
  - pjmlp 5y ago
    Meanwhile others of us have never used Spring in production, I have been on the JEE camp since the BEA and WebSphere 5.1 days.
- dragonelite 5y ago
  For pretty 90 plus % of jobs you're pretty much confined to the Spring ecosystem. Same with C# and asp.net separating the two would be really weird.
  - pjmlp 5y ago
    Hello from the 10%.
- 5y ago

Ono-Sendai 5y ago

tldr

jakobnissen 5y ago
A rust project with 420 deps but only 7k LOC suddenly took 2 minutes to compile with cache. Turns out it's mostly a compiler regression that is already fixed. The blog shows all kinds of profiling and analysis of the Rust compiler.

flohofwoe 5y ago

Very interesting and detailed article how to profile the Rust toolchain, but TBH, after the reveal that the project uses several hundred dependencies my first thought was: "well, there's your problem". One thing or another will always be broken in such a complex scenario.

EVa5I7bHFq9mnYK 5y ago

Split your code into separately compiled modules (DLLs or whatever it is these days) and minimize interface surface between those modules.

marco_craveiro 5y ago

This is a great post, thanks for sharing.

swsieber 5y ago

Another article about rust diagnosing rust compile times: https://matklad.github.io/2021/09/04/fast-rust-builds.html https://matklad.github.io/2021/09/04/fast-rust-builds.html

Klasiaster 5y ago

Instead of using sccache and turning off incremental compilation it could help to change the approach and retain the build folder from the main branch CI builds for all other builds, since both sccache and incremental compilation are kind of overlapping with incremental compilation being the integrated solution while sccache seems to be more like a workaround?

zenlot 5y ago

"In another language, say, C or C++, we might write a Makefile by hand. What? No. Nobody does that anymore." Oh these Rustaceans or how they call themselves. Look at the gaming industry, if you're playing any AAA/AA game it's most likely C++, there's some market share on C# (where Unity is used), but still the whole gaming industry is C++, and it's bigger than movie/music industries combined. So yes, people "do that", and it's not going to change anytime soon.

alexchamberlain 5y ago
I think the OP meant no one writes Makefiles manually; you might use CMake or ninja, for example.
- jjgreen 5y ago
  What an odd (and demonstrably false) thing for the OP to say, Makefiles are a delight.

jl2718 5y ago

I’ve always wondered why compiled languages can’t fall back to interpreted execution for debugging.