14 ms·

This much complexity and the tribal knowledge required to bypass it, so early in the the life of a programming language, is a really bad sign. It means the com

This much complexity and the tribal knowledge required to bypass it, so early in the the life of a programming language, is a really bad sign. It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users. You might react to this by saying that slow build times are not a big deal. That's a mistake -- iteration is key to productivity and other languages do it better.

lpapez 5y ago
That is because the languages are different and choose different tradeoffs. The tradeoffs might make a language inapplicable in your domain. Rust has always been marketed as a "systems programming language". C++ is Rust's closes competitor in this domain space and it suffers from terribly slow compiling as well. I would say that due to headers, massive portability baggage and a lack of a standard build system, compiling C++ is even worse. And that is a language that has been around for 30+ years. I agree that build times are important (i.e. PR build takes us 30 minutes which is insane), but sometimes you don't really have a choice. When you do have a choice, then by all means reap the benefits of fast compilation that languages such as Go or C can offer.
- saurik 5y ago
  C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.
  - codeflo 5y ago
    That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.
    - nmfisher 5y ago
      Care to elaborate further on how to best structure a C++ project for fast builds?
      - codeflo 5y ago
        It’s a large topic, but the most important goal is to minimize the amount of code that’s transitively included in headers. That can mean avoiding templates, or (if you’re willing to do the work) using explicit instantiations in one source file only. Use the PImpl pattern so that users of a class don’t need to know the declarations of its private members. And whenever reasonable, design your method signatures so that objects are passed by pointer/reference — this means you can work with forward declarations instead of always having to include the full class declaration. Do all of that only where reasonable. Small headers usually don’t matter much, and avoiding dependencies on headers that users are likely to include anyway (like <string>) doesn’t win anything either.
      - dataflow 5y ago
        Avoid over-templating, don't inline expensive functions, put expensive #includes inside .cpp files instead of headers, use internal linkage whenever possible, etc. Doable in theory, painful to get there in practice if you've already dug yourself into a hole, and you can't get out of said hole without convincing the rest of your team/organization to change their coding practices accordingly.
      - jcelerier 5y ago
        Here's my edit-compile-run cycle in dev configuration: https://streamable.com/az397y https://streamable.com/az397y I use Qt, boost, lots of templates. - clang as a compiler - mold as a linker - PCH (easy with cmake) - plugin architecture for the software ; plugins are dynamic libraries when developing (everything is linked statically for releases which take of course much longer to build with lto, etc.) - building on Linux (it's seriously slower on Windows, when using the exact same compiler on the exact same SSD)
        mastax 5y ago
  - spockz 5y ago
    I rarely touch C(++) but every time I have to compile something a bit more complicated than the simplest of projects it is noticeably slow. Other languages I use, scala and go and rust, have higher initial cost but seem to suffer less when the sources grow. A recent example, linkerd proxy compiled in a few minutes, while it took me hours to compile envoy proxy. Then there are the stories in how ridiculously long it takes to build Chrome or the V8 engine. I’m really not sold that real world c(++) projects are fast to build.
    - ithkuil 5y ago
      I have a vague memory that at a previous job I was building envoy proxy in a build automation pipeline and it regularly OOMed. While troubleshooting I noticed that most of the time and memory was spent compiling one or two generated C++ files; they contained many many many type parameters. When you have to generate templates something is wrong.
  - mtzet 5y ago
    That's a bit misleading though. The problem with header files is that they tend to get (textually) included in lots of compilation units, and hence rebuilt a lot of times. I did an experiment on a codebase at work where turning on unity builds -- so effectively everything is a single c++ file -- resulted in a 2x speedup in real time, and a 12x reduction in 'user' time. Of course, such approaches lead to slower hot rebuilds. There's trade-offs worthy of a book. But the point is that C++ is only massively parallel by adding a lot of extra work. This problem could be overcome by reducing the complexity of header files, but that's generally difficult if template-types are used. In general, the solution is to write code that's closer to C than C++ in the header files. In particular using opaque pointers to hide data. We're actually seeing a very similiar problem in the linked article. The chief problem is monomorphization (~templates) and the solution is to use boxing (~data hiding via opaque pointers).
    - deleted 5y ago
      [deleted]
- menaerus 5y ago
  To be more precise, rust is a competitor to C++ and not the other way around. Also, compiling C++ is _not_ worse than compiling rust projects but again the other way around. Although compilation times are a hog in both of these languages, it is well known that this is actually one of the biggest rust pain points. In general, long compilation times are mostly attributed by the complexity of things (algorithms) that compilers are trying to do for you in order to produce machine code and not because there's a lack of standardized build system (a bad idea) nor because of existence of headers. In particular, biggest "offender" in C++ compilation model is almost always due to the usage of template metaprogramming whereas in rust I'd envision it's the compile time guarantees that must be offered through mechanisms such as borrow checker and alike.
  - jokethrowaway 5y ago
    I disagree that it's a pain point in C++ or in Rust. You're generally just doing a lot more stuff and shifting the pain to incremental debug builds (so that the slow path doesn't need recompilation) is a perfectly viable solution. Template metaprogramming has never been the pain point in C++ for me because nobody wanted to use templates (after a certain points), huge deps are. Anecdotally, the slow builds I've experienced in Rust were case sof overzealous Serialize implementation (aka, people adding Serialize to everything, just in case) or deps on large external projects (often in C++). Coming from C++, where you either have someone who knows how to debug build times, I think the comparison with rust is unfair. cargo-timings is pretty nice and gives you a hint on what to look for. This article actually shows debugging build times is not that hard. I did that for a personal project (which compiled part of OpenCV) and I'm certainly no ninja. I remember attempting similar tasks in C++ and getting completely lost.
    - menaerus 5y ago
      I live and breathe C++ full time for more than a decade now. Compiling C++ _has_ always been pain point and big efficiency killer. Especially if you come from other languages (I don't but I enjoy learning other stuff too). It doesn't matter if you're directly writing your own template code or not because you're always gonna pull it in indirectly, even in most simplistic and inrealistic case when your sole dependency is C++ runtime library. OTOH debugging C++ build times is no more than recompiling your code with -ftime-trace and inspecting it with ninjatracing, which btw is a great tool. However, getting the data as you see is not the real challenge here but in how to use this data to optimize your build pipeline while keeping the functionality and not introducing more technical burden. That is very difficult and for many projects questionable if the benefits will outweigh the total cost invested in it.
- pjmlp 5y ago
  On most corporations C++ third party libraries are consumed as binary dependencies, hence the large compile times aren't as serious as in Rust.
- mplanchard 5y ago
  I work on a large rust project professionally, and we have tried to make build times a priority for CI. I’m just noting a couple of things that helped us a lot, in case they’re useful. - sccache is pretty much essential - beefiness: GitLab’s default runners for example are way too underpowered and made our builds take like 30 minutes (as opposed to 5 on a beefy machine in CircleCI). We’re working on switching to self-hosted runners for GL - replacing diesel with sqlx saved us a fair bit of time - caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory - lld linking was another relatively small win, but a win nonetheless - splitting things up into smaller libraries and binaries (all in one workspace) helped us a fair bit, since building/testing some targets can now totally skip some libraries, but I imagine YMMV
  - pdimitar 5y ago
    > caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory Can you please clarify this point? Caching how?
    - mplanchard 5y ago
      Persisting it across jobs, using whatever your CI framework lets you do. We cache this stuff using the workspace cargo.lock checksum as the primary cache key
- deleted 5y ago
  [deleted]
- lenkite 5y ago
  "compiling C++ is even worse" In my experience, this has been the other way around. C++ gives you many tools to optimise compilation and when one one takes advantage of them (precompiled headers/shared libs/no boost, etc), the edit-compile-test cycle is fairly fast. Rust is consistently slow to compile and not much idea of how to go about optimising compile speed. Both languages are snails compared to Go, of-course.
celeduc 5y ago
Build times matter a lot. I've spent a lot of my career working to reduce build and test cycles to improve productivity. Rust is the one environment where no matter how much time we threw at the problem it never got better.
geewee 5y ago
Often when building compilers you trade the speed of compilation against the speed of execution[1], and Rust has (traditionally) chosen the speed of executiong over making the compiler fast. This won't necessarily always be so, there's e.g. a new compiler backend called cranelift in the works, that makes compilations 30%ish faster, and in return the code isn't as optimized - which is good for a faster inner cycle. https://pingcap.com/blog/rust-compilation-model-calamity#rusts-designs-for-poor-compilation-time https://pingcap.com/blog/rust-compilation-model-calamity#rus...
- saurik 5y ago
  The biggest issue cited in this article is a regression in the type checker, which is an open bug that apparently is just lower priority than whatever else the Rust team is working on instead of ensuring compilation time is fast. Only once that was worked around did they article really start having to pay attention to the code generation and linking stages, and the issues there seem to mostly be about lack of parallelism caused by the "crates" model combined with link time optimizations and incremental compiles being off by default. You don't need to make a fundamentally faster code generator that makes worse output to deal with this: the performance losses you get from parallel, incremental, and separate compilation techniques are so minimal that only the most serious C++ projects go out of their way to combine translation units and activate LTO.
  - Ar-Curunir 5y ago
    Contrary to what you say, rust compilation speed is a top priority for rustc developers. Rust compiler times have gone down YoY every year for the past few years. See, eg, https://nnethercote.github.io/2021/11/12/the-rust-compiler-has-gotten-faster-again.html https://nnethercote.github.io/2021/11/12/the-rust-compiler-h...
    - pdimitar 5y ago
      Then this means that the developer ecosystem continuously finds ways to swallow the improvements in speed. :(
codeflo 5y ago
I agree, and even as someone who holds Rust in high regard, I’m becoming increasingly frustrated by this. There doesn’t seem to be one clear root of this problem. Some of it is the language not being designed with ease of compilation in mind, unlike let’s say, to pick an extreme example, Go. Some of it is the tooling and default settings, especially around incremental compilation, linkers etc. People are working on this, but it takes time. But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio. In the C++ world, the popular “boost” library has a similar problem, and I know shops that avoid it, or large parts of it, simply because the compile times would be unbearable. I really hope that parts of the Rust community start to make similar decisions.
- vladvasiliu 5y ago
  > But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio. But isn't this the right trade off for something like Tokio which is at the base of applications which expect to be fast? Or else those applications wouldn't bother with async runtimes and what-have-you. There's also the fact that during the dev cycle, you'll compile tokio & friends once and be done with it. And for CI pipelines, there's caching. I'm however not a professional developer, so I'm curious if I'm missing something.
  - gameswithgo 5y ago
    Yes, a slow compiletime of a dependency is less painful since you don’t do full recompiles that often, but things can spiral out of control if you have say 50 dependencies and many of them are real slow and you are on your old dual core laptop and find it takes hours, and so on.
    - ModernMech 5y ago
izacus 5y ago
Rust seems to have inherited two favorite C++ development process features: long compilation times and incomprehensible error messages that aren't really related to the actual issue (e.g. just how C++ barfed out a bunch of template errors, borrow checker really loves to barf out very obscure error messages when there's an issue with lifetimes).
- estebank 5y ago
  If you encounter incomprehensible error messages, I would encourage you to file a ticket at http://github.com/rust-lang/rust/issues http://github.com/rust-lang/rust/issues. We consider them bugs and are diligent about dealing with them.
  - ogoffart 5y ago
    The worst error message was basically no error message. After I made some big refactoring to my project, cargo would just exit with: error: could not compile `the_project`. And nothing else. No hint of what the problem is. It then took me a long time to try to comment out code here and there to find out what was causing the error. When I finally found out the problem, I took the time to fill a bug report. I don't know if anyone looked at it. https://github.com/rust-lang/rust/issues/73026 https://github.com/rust-lang/rust/issues/73026
    - estebank 5y ago
      Thank you for the great report. It indeed seems to have fallen through the cracks, with no one touching it (it's missing the T-compiler tag, which means I didn't triage it) since it was first cathegorized. I wouldn't call that lack of output representative of the experience of using rustc though, if I'm allowed to let my pride on my work flare up a bit.
      - ogoffart 5y ago
        Sorry, I did not mean to say that this was representative of the overall experience. On the contrary, I'm usually quite satisfied with the compiler messages. Thanks for your work!
maxwell86 5y ago
Rust 1.58 has a bug that introduced a compile time regression, and the knowledge required to bypass it is to just switch to the preview of the next release, which is trivial to set up. So I really don’t know what you are talking about in your comment. What’s hard about this?
- masklinn 5y ago
  > Rust 1.58 Do you mean 1.57? As soon as I saw the final binary taking an ungodly amount of time I knew what it was going to be, because I hit it myself, in 1.57, using warp. Though there are also a few separate older regressions (at least one in 1.56, though I gather somewhat less critical / troublesome). > the knowledge required to bypass it is to just switch to the preview of the next release Switching to the previous tends to be more reliable and a better idea, really.
misnome 5y ago
This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems. Anyone who doesn't know about many of these tools will probably find them useful (or at least useful to know that they exist) in the future. Besides, for iteration people are probably in debug, not release mode, which the article mentions is initially 19s vs 2m+. As far as I can see the main conclusion seems to be that there was a compiler regression causing the issues, which is fixed in nightly. I don't think many users would need to go through these steps to diagnose the problem beyond "wait for a newer compiler release". In general I don't think being slightly slower in a systems language in cases where you've explicitly asked for full optimisations is that much of a problem when there are simple ways to mitigate the feedback loop (e.g. debug compilations). Would I like these things to be faster? Sure. But with the size of the rust projects I use, it's not noticeable slower than the large established C++ projects I have to deal with (that aren't tuned for compilation speed).
- ithkuil 5y ago
  Rust debug builds are also unbearably slow. I wouldn't complain that much if it was just a problems of having full optimizations on
  - singron 5y ago
    For perf sensitive projects, I set opt-level=1 in debug for at least the dependencies. Edit: I realized you might have meant that debug compiles still take a long time. In that case, obviously this won't help.
- MuffinFlavored 5y ago
  > This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems. I'd argue that when it comes to doing work, I only want to deep-dive into my code. Not third party libraries that "vow" to have been tested + performant, let alone the language/build tools themselves.
  - 5y ago
forrestthewoods 5y ago
> It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users. What languages do it best? JavaScript is a dystopia of stacked bundlers. Python dependencies and deployment is a toxic superfund wasteland. You’re not wrong that it’s a problem. But as far as I can tell build+deployment is an unsolved problem for non-hobby projects in most, if not all, languages.
- hu3 5y ago
  Go is known for having really fast compilations. Almost Pascal levels of fast.
  - couchand 5y ago
    Go is also known for treating developers like cattle, so, I'll take Rust's trade-off, thanks.
    - yawaramin 5y ago
      OCaml is a powerful, expressive language with generics from day one, and Go-like build speeds.
      - pdimitar 5y ago
        All that remains is for it to gain a build tool like mix or cargo. Futzing around with esy works but takes more time than I'm willing to regularly invest in it. Super excited about 5.00 and multicore!
        yawaramin 5y ago
        I really go and forth on this. By separating the package manager and build tool, they leave the door open for either to be swapped out (which is what esy does, incidentally). The flexibility comes at a price of complexity, but it usually works well after climbing the learning curve.
jeffparsons 5y ago
I'm very happy with my overall experience using Rust, but I would never say "slow build times are not a big deal". I can work around them just fine by myself because it's the kind of thing I'm used to doing, but as more of my team (especially more junior members) start using it for projects, the slowness becomes a big deal. And I don't see that as a problem with my team at all — it's very much a problem with Rust. Especially if you're doing the kind of work where you're switching fairly quickly between different projects (bug blitz, cross-cutting platform work, etc.), you simply can't afford to be wasting time trying to work around bad-by-default compiler performance. That said, I don't think that the level of slowness we're talking about here is actually an intrinsic property of Rust. Specifically, I think there are a handful of strategies that would really improve things: - For teams, dead-simple out-of-the-box sccache integration, and officially supported Terraform (etc.) projects for setting it up in your corporate cloud environment. Then your "cold" builds for a project you've never touched before could be a lot quicker. - A set of inbuilt use-case-driven named profiles for Cargo beyond just "debug" and "release", which you could set as the default for your "debug" or "release" builds. I'm thinking things like "prioritise-runtime-performance-and-incremental-build-time" (maybe with a better name) that sets the right kinds of flags to make that work, given that all those things are achievable if you tweak flags manually. - Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here. Maybe it could just be used where possible? - Binary repositories. I think people are too quick to dismiss these on the grounds that different features enabled on a crate makes it produce a very different output. You could cache the three most common configurations or something. TL;DR: I think just not a lot of time has gone into tackling this at a high level, because there have been other priorities, and there are whole lot of directions that could be explored that might make a big difference.
- ThomasWinwood 5y ago
  > Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here. mold supports precisely one platform: x86_64 Linux. It also doesn't support linker scripts beyond what is necessary to link libc, and given the creator honestly suggested the suckless practice of editing and recompiling the source code as a possible replacement for them I have my doubts that it'll have a satisfactory replacement for, say, the things I'm currently doing. (Admittedly those things are in ARMv4T rather than x86_64, but still.)
Narretz 5y ago
The two biggest time savings where accomplished by using a version without the regression bug, and using incremental builds in debug mode for local dev. Bugs can be introduced in any language, so the only gotcha to be fixed is that maybe incremental compile should be the default.
- swiftcoder 5y ago
  Note that incremental builds are the default for debug builds. For reasons that are not entirely elaborated in the article, the author is using release builds for local development.
  - estebank 5y ago
    In some verticals the only way to properly exercise and test the resulting binary is in release mode. Think things like DBs, game engines, etc. It's why good debug information in release is needed.

lpapez 5y ago

That is because the languages are different and choose different tradeoffs. The tradeoffs might make a language inapplicable in your domain. Rust has always been marketed as a "systems programming language". C++ is Rust's closes competitor in this domain space and it suffers from terribly slow compiling as well. I would say that due to headers, massive portability baggage and a lack of a standard build system, compiling C++ is even worse. And that is a language that has been around for 30+ years. I agree that build times are important (i.e. PR build takes us 30 minutes which is insane), but sometimes you don't really have a choice. When you do have a choice, then by all means reap the benefits of fast compilation that languages such as Go or C can offer.

saurik 5y ago
C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.
- codeflo 5y ago
  That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.
  - nmfisher 5y ago
    Care to elaborate further on how to best structure a C++ project for fast builds?
    - codeflo 5y ago
      It’s a large topic, but the most important goal is to minimize the amount of code that’s transitively included in headers. That can mean avoiding templates, or (if you’re willing to do the work) using explicit instantiations in one source file only. Use the PImpl pattern so that users of a class don’t need to know the declarations of its private members. And whenever reasonable, design your method signatures so that objects are passed by pointer/reference — this means you can work with forward declarations instead of always having to include the full class declaration. Do all of that only where reasonable. Small headers usually don’t matter much, and avoiding dependencies on headers that users are likely to include anyway (like <string>) doesn’t win anything either.
    - dataflow 5y ago
      Avoid over-templating, don't inline expensive functions, put expensive #includes inside .cpp files instead of headers, use internal linkage whenever possible, etc. Doable in theory, painful to get there in practice if you've already dug yourself into a hole, and you can't get out of said hole without convincing the rest of your team/organization to change their coding practices accordingly.
    - jcelerier 5y ago
      Here's my edit-compile-run cycle in dev configuration: https://streamable.com/az397y https://streamable.com/az397y I use Qt, boost, lots of templates. - clang as a compiler - mold as a linker - PCH (easy with cmake) - plugin architecture for the software ; plugins are dynamic libraries when developing (everything is linked statically for releases which take of course much longer to build with lto, etc.) - building on Linux (it's seriously slower on Windows, when using the exact same compiler on the exact same SSD)
      - mastax 5y ago
- spockz 5y ago
  I rarely touch C(++) but every time I have to compile something a bit more complicated than the simplest of projects it is noticeably slow. Other languages I use, scala and go and rust, have higher initial cost but seem to suffer less when the sources grow. A recent example, linkerd proxy compiled in a few minutes, while it took me hours to compile envoy proxy. Then there are the stories in how ridiculously long it takes to build Chrome or the V8 engine. I’m really not sold that real world c(++) projects are fast to build.
  - ithkuil 5y ago
    I have a vague memory that at a previous job I was building envoy proxy in a build automation pipeline and it regularly OOMed. While troubleshooting I noticed that most of the time and memory was spent compiling one or two generated C++ files; they contained many many many type parameters. When you have to generate templates something is wrong.
- mtzet 5y ago
  That's a bit misleading though. The problem with header files is that they tend to get (textually) included in lots of compilation units, and hence rebuilt a lot of times. I did an experiment on a codebase at work where turning on unity builds -- so effectively everything is a single c++ file -- resulted in a 2x speedup in real time, and a 12x reduction in 'user' time. Of course, such approaches lead to slower hot rebuilds. There's trade-offs worthy of a book. But the point is that C++ is only massively parallel by adding a lot of extra work. This problem could be overcome by reducing the complexity of header files, but that's generally difficult if template-types are used. In general, the solution is to write code that's closer to C than C++ in the header files. In particular using opaque pointers to hide data. We're actually seeing a very similiar problem in the linked article. The chief problem is monomorphization (~templates) and the solution is to use boxing (~data hiding via opaque pointers).
  - deleted 5y ago
    [deleted]
menaerus 5y ago
To be more precise, rust is a competitor to C++ and not the other way around. Also, compiling C++ is _not_ worse than compiling rust projects but again the other way around. Although compilation times are a hog in both of these languages, it is well known that this is actually one of the biggest rust pain points. In general, long compilation times are mostly attributed by the complexity of things (algorithms) that compilers are trying to do for you in order to produce machine code and not because there's a lack of standardized build system (a bad idea) nor because of existence of headers. In particular, biggest "offender" in C++ compilation model is almost always due to the usage of template metaprogramming whereas in rust I'd envision it's the compile time guarantees that must be offered through mechanisms such as borrow checker and alike.
- jokethrowaway 5y ago
  I disagree that it's a pain point in C++ or in Rust. You're generally just doing a lot more stuff and shifting the pain to incremental debug builds (so that the slow path doesn't need recompilation) is a perfectly viable solution. Template metaprogramming has never been the pain point in C++ for me because nobody wanted to use templates (after a certain points), huge deps are. Anecdotally, the slow builds I've experienced in Rust were case sof overzealous Serialize implementation (aka, people adding Serialize to everything, just in case) or deps on large external projects (often in C++). Coming from C++, where you either have someone who knows how to debug build times, I think the comparison with rust is unfair. cargo-timings is pretty nice and gives you a hint on what to look for. This article actually shows debugging build times is not that hard. I did that for a personal project (which compiled part of OpenCV) and I'm certainly no ninja. I remember attempting similar tasks in C++ and getting completely lost.
  - menaerus 5y ago
    I live and breathe C++ full time for more than a decade now. Compiling C++ _has_ always been pain point and big efficiency killer. Especially if you come from other languages (I don't but I enjoy learning other stuff too). It doesn't matter if you're directly writing your own template code or not because you're always gonna pull it in indirectly, even in most simplistic and inrealistic case when your sole dependency is C++ runtime library. OTOH debugging C++ build times is no more than recompiling your code with -ftime-trace and inspecting it with ninjatracing, which btw is a great tool. However, getting the data as you see is not the real challenge here but in how to use this data to optimize your build pipeline while keeping the functionality and not introducing more technical burden. That is very difficult and for many projects questionable if the benefits will outweigh the total cost invested in it.
pjmlp 5y ago
On most corporations C++ third party libraries are consumed as binary dependencies, hence the large compile times aren't as serious as in Rust.
mplanchard 5y ago
I work on a large rust project professionally, and we have tried to make build times a priority for CI. I’m just noting a couple of things that helped us a lot, in case they’re useful. - sccache is pretty much essential - beefiness: GitLab’s default runners for example are way too underpowered and made our builds take like 30 minutes (as opposed to 5 on a beefy machine in CircleCI). We’re working on switching to self-hosted runners for GL - replacing diesel with sqlx saved us a fair bit of time - caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory - lld linking was another relatively small win, but a win nonetheless - splitting things up into smaller libraries and binaries (all in one workspace) helped us a fair bit, since building/testing some targets can now totally skip some libraries, but I imagine YMMV
- pdimitar 5y ago
  > caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory Can you please clarify this point? Caching how?
  - mplanchard 5y ago
    Persisting it across jobs, using whatever your CI framework lets you do. We cache this stuff using the workspace cargo.lock checksum as the primary cache key
deleted 5y ago
[deleted]
lenkite 5y ago
"compiling C++ is even worse" In my experience, this has been the other way around. C++ gives you many tools to optimise compilation and when one one takes advantage of them (precompiled headers/shared libs/no boost, etc), the edit-compile-test cycle is fairly fast. Rust is consistently slow to compile and not much idea of how to go about optimising compile speed. Both languages are snails compared to Go, of-course.

celeduc 5y ago

Build times matter a lot. I've spent a lot of my career working to reduce build and test cycles to improve productivity. Rust is the one environment where no matter how much time we threw at the problem it never got better.

geewee 5y ago

Often when building compilers you trade the speed of compilation against the speed of execution[1], and Rust has (traditionally) chosen the speed of executiong over making the compiler fast. This won't necessarily always be so, there's e.g. a new compiler backend called cranelift in the works, that makes compilations 30%ish faster, and in return the code isn't as optimized - which is good for a faster inner cycle. https://pingcap.com/blog/rust-compilation-model-calamity#rusts-designs-for-poor-compilation-time https://pingcap.com/blog/rust-compilation-model-calamity#rus...

saurik 5y ago
The biggest issue cited in this article is a regression in the type checker, which is an open bug that apparently is just lower priority than whatever else the Rust team is working on instead of ensuring compilation time is fast. Only once that was worked around did they article really start having to pay attention to the code generation and linking stages, and the issues there seem to mostly be about lack of parallelism caused by the "crates" model combined with link time optimizations and incremental compiles being off by default. You don't need to make a fundamentally faster code generator that makes worse output to deal with this: the performance losses you get from parallel, incremental, and separate compilation techniques are so minimal that only the most serious C++ projects go out of their way to combine translation units and activate LTO.
- Ar-Curunir 5y ago
  Contrary to what you say, rust compilation speed is a top priority for rustc developers. Rust compiler times have gone down YoY every year for the past few years. See, eg, https://nnethercote.github.io/2021/11/12/the-rust-compiler-has-gotten-faster-again.html https://nnethercote.github.io/2021/11/12/the-rust-compiler-h...
  - pdimitar 5y ago
    Then this means that the developer ecosystem continuously finds ways to swallow the improvements in speed. :(

codeflo 5y ago

I agree, and even as someone who holds Rust in high regard, I’m becoming increasingly frustrated by this. There doesn’t seem to be one clear root of this problem. Some of it is the language not being designed with ease of compilation in mind, unlike let’s say, to pick an extreme example, Go. Some of it is the tooling and default settings, especially around incremental compilation, linkers etc. People are working on this, but it takes time. But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio. In the C++ world, the popular “boost” library has a similar problem, and I know shops that avoid it, or large parts of it, simply because the compile times would be unbearable. I really hope that parts of the Rust community start to make similar decisions.

vladvasiliu 5y ago
> But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio. But isn't this the right trade off for something like Tokio which is at the base of applications which expect to be fast? Or else those applications wouldn't bother with async runtimes and what-have-you. There's also the fact that during the dev cycle, you'll compile tokio & friends once and be done with it. And for CI pipelines, there's caching. I'm however not a professional developer, so I'm curious if I'm missing something.
- gameswithgo 5y ago
  Yes, a slow compiletime of a dependency is less painful since you don’t do full recompiles that often, but things can spiral out of control if you have say 50 dependencies and many of them are real slow and you are on your old dual core laptop and find it takes hours, and so on.
  - ModernMech 5y ago

izacus 5y ago

Rust seems to have inherited two favorite C++ development process features: long compilation times and incomprehensible error messages that aren't really related to the actual issue (e.g. just how C++ barfed out a bunch of template errors, borrow checker really loves to barf out very obscure error messages when there's an issue with lifetimes).

estebank 5y ago
If you encounter incomprehensible error messages, I would encourage you to file a ticket at http://github.com/rust-lang/rust/issues http://github.com/rust-lang/rust/issues. We consider them bugs and are diligent about dealing with them.
- ogoffart 5y ago
  The worst error message was basically no error message. After I made some big refactoring to my project, cargo would just exit with: error: could not compile `the_project`. And nothing else. No hint of what the problem is. It then took me a long time to try to comment out code here and there to find out what was causing the error. When I finally found out the problem, I took the time to fill a bug report. I don't know if anyone looked at it. https://github.com/rust-lang/rust/issues/73026 https://github.com/rust-lang/rust/issues/73026
  - estebank 5y ago
    Thank you for the great report. It indeed seems to have fallen through the cracks, with no one touching it (it's missing the T-compiler tag, which means I didn't triage it) since it was first cathegorized. I wouldn't call that lack of output representative of the experience of using rustc though, if I'm allowed to let my pride on my work flare up a bit.
    - ogoffart 5y ago
      Sorry, I did not mean to say that this was representative of the overall experience. On the contrary, I'm usually quite satisfied with the compiler messages. Thanks for your work!

maxwell86 5y ago

Rust 1.58 has a bug that introduced a compile time regression, and the knowledge required to bypass it is to just switch to the preview of the next release, which is trivial to set up. So I really don’t know what you are talking about in your comment. What’s hard about this?

masklinn 5y ago
> Rust 1.58 Do you mean 1.57? As soon as I saw the final binary taking an ungodly amount of time I knew what it was going to be, because I hit it myself, in 1.57, using warp. Though there are also a few separate older regressions (at least one in 1.56, though I gather somewhat less critical / troublesome). > the knowledge required to bypass it is to just switch to the preview of the next release Switching to the previous tends to be more reliable and a better idea, really.

misnome 5y ago

This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems. Anyone who doesn't know about many of these tools will probably find them useful (or at least useful to know that they exist) in the future. Besides, for iteration people are probably in debug, not release mode, which the article mentions is initially 19s vs 2m+. As far as I can see the main conclusion seems to be that there was a compiler regression causing the issues, which is fixed in nightly. I don't think many users would need to go through these steps to diagnose the problem beyond "wait for a newer compiler release". In general I don't think being slightly slower in a systems language in cases where you've explicitly asked for full optimisations is that much of a problem when there are simple ways to mitigate the feedback loop (e.g. debug compilations). Would I like these things to be faster? Sure. But with the size of the rust projects I use, it's not noticeable slower than the large established C++ projects I have to deal with (that aren't tuned for compilation speed).

ithkuil 5y ago
Rust debug builds are also unbearably slow. I wouldn't complain that much if it was just a problems of having full optimizations on
- singron 5y ago
  For perf sensitive projects, I set opt-level=1 in debug for at least the dependencies. Edit: I realized you might have meant that debug compiles still take a long time. In that case, obviously this won't help.
MuffinFlavored 5y ago
> This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems. I'd argue that when it comes to doing work, I only want to deep-dive into my code. Not third party libraries that "vow" to have been tested + performant, let alone the language/build tools themselves.
- 5y ago

forrestthewoods 5y ago

> It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users. What languages do it best? JavaScript is a dystopia of stacked bundlers. Python dependencies and deployment is a toxic superfund wasteland. You’re not wrong that it’s a problem. But as far as I can tell build+deployment is an unsolved problem for non-hobby projects in most, if not all, languages.

hu3 5y ago
Go is known for having really fast compilations. Almost Pascal levels of fast.
- couchand 5y ago
  Go is also known for treating developers like cattle, so, I'll take Rust's trade-off, thanks.
  - yawaramin 5y ago
    OCaml is a powerful, expressive language with generics from day one, and Go-like build speeds.
    - pdimitar 5y ago
      All that remains is for it to gain a build tool like mix or cargo. Futzing around with esy works but takes more time than I'm willing to regularly invest in it. Super excited about 5.00 and multicore!
      - yawaramin 5y ago
        I really go and forth on this. By separating the package manager and build tool, they leave the door open for either to be swapped out (which is what esy does, incidentally). The flexibility comes at a price of complexity, but it usually works well after climbing the learning curve.

jeffparsons 5y ago

I'm very happy with my overall experience using Rust, but I would never say "slow build times are not a big deal". I can work around them just fine by myself because it's the kind of thing I'm used to doing, but as more of my team (especially more junior members) start using it for projects, the slowness becomes a big deal. And I don't see that as a problem with my team at all — it's very much a problem with Rust. Especially if you're doing the kind of work where you're switching fairly quickly between different projects (bug blitz, cross-cutting platform work, etc.), you simply can't afford to be wasting time trying to work around bad-by-default compiler performance. That said, I don't think that the level of slowness we're talking about here is actually an intrinsic property of Rust. Specifically, I think there are a handful of strategies that would really improve things: - For teams, dead-simple out-of-the-box sccache integration, and officially supported Terraform (etc.) projects for setting it up in your corporate cloud environment. Then your "cold" builds for a project you've never touched before could be a lot quicker. - A set of inbuilt use-case-driven named profiles for Cargo beyond just "debug" and "release", which you could set as the default for your "debug" or "release" builds. I'm thinking things like "prioritise-runtime-performance-and-incremental-build-time" (maybe with a better name) that sets the right kinds of flags to make that work, given that all those things are achievable if you tweak flags manually. - Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here. Maybe it could just be used where possible? - Binary repositories. I think people are too quick to dismiss these on the grounds that different features enabled on a crate makes it produce a very different output. You could cache the three most common configurations or something. TL;DR: I think just not a lot of time has gone into tackling this at a high level, because there have been other priorities, and there are whole lot of directions that could be explored that might make a big difference.

ThomasWinwood 5y ago
> Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here. mold supports precisely one platform: x86_64 Linux. It also doesn't support linker scripts beyond what is necessary to link libc, and given the creator honestly suggested the suckless practice of editing and recompiling the source code as a possible replacement for them I have my doubts that it'll have a satisfactory replacement for, say, the things I'm currently doing. (Admittedly those things are in ARMv4T rather than x86_64, but still.)

Narretz 5y ago

The two biggest time savings where accomplished by using a version without the regression bug, and using incremental builds in debug mode for local dev. Bugs can be introduced in any language, so the only gotcha to be fixed is that maybe incremental compile should be the default.

swiftcoder 5y ago
Note that incremental builds are the default for debug builds. For reasons that are not entirely elaborated in the article, the author is using release builds for local development.
- estebank 5y ago
  In some verticals the only way to properly exercise and test the resulting binary is in release mode. Think things like DBs, game engines, etc. It's why good debug information in release is needed.