15 ms·

Reducing Rust Incremental Compilation Times on macOS

186 points by yannikyeo 6y ago

xenadu02 6y ago
I'm not sure why Rust doesn't emit the debug info in the Mach-O. The executable can still be moved to another system and debugged but it doesn't require a second pass to generate a separate dSYM.
- the_mitsuhiko 6y ago
  The dsym is basically just a macho anyways. It used to just emit it as part of the main binary and then asked dsymutil to split it off. If you do want to retain it entirely you can set `split-debuginfo` to `off` and it will remain in the final executable. The advantage of the `unpacked` version is that the debug info just stays associated with the original object files instead of the executable. This makes handling them harder obviously but is good enough for a lot of use cases.
- JacobCarlborg 6y ago
  The system linker strips debug info from the executable. The way the debugger finds debug info is that the debug info is available in the object files. A reference to the object files is added to the executable. The debugger can find the object files and read the debug info from the object files. I have no experience with Rust but in opinion this should be the default behavior for debug builds, no need to generate a dSYM file. The dSYM file is used if you want to ship debug info with your final release build to customers. There's no need for dSYM during regular development workflows. It's possible to get the linker to keep the debug info by tweaking the section attributes. By default all debug info related sections have the `S_ATTR_DEBUG` flag. If that is replaced with the `S_REGULAR` flag the linker will keep those sections. The DMD D compiler does this for the `__debug_line` section [1][2][3]. This allows for uncaught exceptions to print a stack trace with filenames and line numbers. Of course, DMD uses a custom backend so this change was easy. Rust which relies on LLVM would probably need a fork. [1] https://github.com/dlang/dmd/pull/8168 https://github.com/dlang/dmd/pull/8168 [2] https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da14f7ec120207cbd/src/dmd/backend/dwarfdbginf.d#L15-L26 https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da1... [3] https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da14f7ec120207cbd/src/dmd/backend/dwarfdbginf.d#L520-L522 https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da1...
  - IshKebab 6y ago
    > There's no need for dSYM during regular development workflows. I'm not sure that's true. I was not getting line numbers in backtraces in a Rust program I developed because I copied the executable to another directory. I had to add a symlink to the dSYM directory to make it work.
  - mhh__ 6y ago
    Didn't know dmd did that.
imron 6y ago
> Dev rebuilds lasting 14 seconds was getting me a bit worried I wish my dev builds only took 14 seconds...
- snake_case 6y ago
  Author here. How long do your dev builds take, and what is it that you are building? Curious what kind of improvement you get with split-debuginfo enabled! This project is a very small hyper/tokio backend API. However, I also work on a CLI tool [1] I created which has very fast recompile times, usually less than a second. Mostly because it's pretty light on dependencies. [1]: https://github.com/jakedeichert/mask https://github.com/jakedeichert/mask
  - vlovich123 6y ago
    In C++ land I’ve experienced the gamut from multi-minute clean builds to 30 minutes to 1.5 hours on the latest & greatest CPUs, even with ccache. 14s sounds like paradise to me.
    - O_H_E 6y ago
      Honest question, is there really no way to know whether what you wrote is working or not without waiting 20min-1H?? That sounds like hell. Where these incremental compilation?
      - vlovich123 5y ago
        That’s usually clean builds. Incremental builds can be in the instant to 1-5 min depending on what line of code you change (deploy a new compiler and you’re probably rebuilding everything).
        O_H_E 5y ago
        Oh ok, that makes much more sense.
  - 6y ago
pjmlp 6y ago
On the other hand I just gave up expecting sscache to build on my travel netbook, > 400 dependencies, really? apt-get install rustlibXYZ-dev can't come soon enough.
- ChrisSD 6y ago
  sscache has pre-built binaries on their releases page. Or you can do: cargo install --no-default-features To avoid building support for all storage backends. You can then add back any backend you're actually using.
  - pjmlp 6y ago
    Thanks, I failed to find how to download them, besides the note that a login is expected, I could not find the artifacts among GitHub workflows. However, the point is actually how it is a big hurdle to overcome having to compile crates with npm like dependency trees all from scratch, vs other compiled languages integrated into the distribution package repositories.
Decabytes 6y ago
Are long compile times just a product of Rust’ design? Or is there an issue with the compiler. Based on my experience with rust I would say it’s the former. It’s doing so many more checks for you then other compilers to prevent lots of problems that it’s bound to be slow. I appreciate that they continue to speed up compilation/add flags. If the compiler gets slower every release it will eventually become unusable. I think the long compilation times will continue to get worse in the future as the rust compiler gets better at checking for errors, and I’m not sure what can really be done about that. One thing that irks me is that they throw around the phrase zero cost abstraction a lot. And I just don’t buy it. Sure I get it, lots of things can be monomorphized so we don’t pay a runtime cost, but there is still a large compilation hit to do that, and it’s not zero cost to the programmer. One need only to look at Chandler Carruth’s cppcon talk “there are no zero cost abstractions” to understand that.
- ChrisSD 6y ago
  But that's just playing with definitions. Rephrase it as "zero runtime cost abstraction" and you get back to the intended meaning. Incidentally your first question is one that's very amenable to testing. When does the Rust compiler spend most of it's time? Is it at the checking stage?
  - optymizer 6y ago
    Oh come on now, do you and the parent really have to be those guys who go "well technically they're not zero cost because the programmer has to wait"? Yes, you're technically correct, but also being unreasonable. It was pretty clear that "zero cost abstractions" refers to the runtime aspect. That is the default meaning, there's nothing to "get back to".
    - ChrisSD 6y ago
      Huh? That was exactly my point.
  - aw1621107 6y ago
    > When does the Rust compiler spend most of it's time? Is it at the checking stage? rustc has a self-profiler that can be used to answer this question [0], as well as a mode that times compiler passes [1]. There's no single reason the Rust compiler is slow, as it depends quite heavily on the code being compiled. For some codebases, LLVM code takes up most of the time; in other codebases (e.g., extremely generic-heavy codebases), it'll be checking-related passes. [0]: https://github.com/rust-lang/measureme/blob/master/summarize/Readme.md https://github.com/rust-lang/measureme/blob/master/summarize... [1]: https://wiki.alopex.li/WhereRustcSpendsItsTime https://wiki.alopex.li/WhereRustcSpendsItsTime
orf 6y ago
Why is this not the default? What’s the downside?
- est31 6y ago
  I think because split-debuginfo is recently stabilized (Nov 2020). https://github.com/rust-lang/rust/pull/79570 https://github.com/rust-lang/rust/pull/79570 Maybe the default will be switched in the future.
  - irh 6y ago
    It's already switched on nightly. https://github.com/rust-lang/cargo/pull/9298 https://github.com/rust-lang/cargo/pull/9298 https://doc.rust-lang.org/nightly/cargo/reference/profiles.html#split-debuginfo https://doc.rust-lang.org/nightly/cargo/reference/profiles.h...
est31 6y ago
You can also turn off debuginfo completely. Personally, someone who does printf debugging, I mainly need it to debug segfaults, which are really rare in Rust. Sometimes the call stack of a panic is useful as well, but if I need debuginfo I can just re-enable it. https://github.com/est31/cargo-udeps/commit/e550d93c7a6d756e639ecd7f117ea9f9328d3b38 https://github.com/est31/cargo-udeps/commit/e550d93c7a6d756e...
- snake_case 6y ago
  Author here. I just tried disabling debug info (debug = 0) for the first time and it looks like my recompile times shave off another 500-1000ms which isn't bad!
  - est31 6y ago
    Glad to have been of help!
person_of_color 6y ago
Rustaceans discover there’s no free lunch, about 10 years after the C++ templaters
nindalf 6y ago
Compile times in rustc have been steadily improving with time, as shown here - https://arewefastyet.rs https://arewefastyet.rs. Every release doesn't make every workload faster, but over a long time horizon, the effect is clear. Rust 1.34 was released in April 2019 and since then many crates have become 33-50% faster to compile, depending on the hardware and the compiler mode (clean/incremental, check/debug/release). Interestingly, the speedup mentioned in OP won't show up in these charts because that's a change on macOS and these benchmarks were recorded on Linux. What is expected to be a gamechanger is the release of cranelift in 2021 or 2022. It's an alternate debug backend that promises much faster debug builds.

186 points by yannikyeo 6y ago

I'm not sure why Rust doesn't emit the debug info in the Mach-O. The executable can still be moved to another system and debugged but it doesn't require a second pass to generate a separate dSYM.

the_mitsuhiko 6y ago
The dsym is basically just a macho anyways. It used to just emit it as part of the main binary and then asked dsymutil to split it off. If you do want to retain it entirely you can set `split-debuginfo` to `off` and it will remain in the final executable. The advantage of the `unpacked` version is that the debug info just stays associated with the original object files instead of the executable. This makes handling them harder obviously but is good enough for a lot of use cases.
JacobCarlborg 6y ago
The system linker strips debug info from the executable. The way the debugger finds debug info is that the debug info is available in the object files. A reference to the object files is added to the executable. The debugger can find the object files and read the debug info from the object files. I have no experience with Rust but in opinion this should be the default behavior for debug builds, no need to generate a dSYM file. The dSYM file is used if you want to ship debug info with your final release build to customers. There's no need for dSYM during regular development workflows. It's possible to get the linker to keep the debug info by tweaking the section attributes. By default all debug info related sections have the `S_ATTR_DEBUG` flag. If that is replaced with the `S_REGULAR` flag the linker will keep those sections. The DMD D compiler does this for the `__debug_line` section [1][2][3]. This allows for uncaught exceptions to print a stack trace with filenames and line numbers. Of course, DMD uses a custom backend so this change was easy. Rust which relies on LLVM would probably need a fork. [1] https://github.com/dlang/dmd/pull/8168 https://github.com/dlang/dmd/pull/8168 [2] https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da14f7ec120207cbd/src/dmd/backend/dwarfdbginf.d#L15-L26 https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da1... [3] https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da14f7ec120207cbd/src/dmd/backend/dwarfdbginf.d#L520-L522 https://github.com/dlang/dmd/blob/33406c205b76a8c2b5fb918da1...
- IshKebab 6y ago
  > There's no need for dSYM during regular development workflows. I'm not sure that's true. I was not getting line numbers in backtraces in a Rust program I developed because I copied the executable to another directory. I had to add a symlink to the dSYM directory to make it work.
- mhh__ 6y ago
  Didn't know dmd did that.

imron 6y ago

> Dev rebuilds lasting 14 seconds was getting me a bit worried I wish my dev builds only took 14 seconds...

snake_case 6y ago
Author here. How long do your dev builds take, and what is it that you are building? Curious what kind of improvement you get with split-debuginfo enabled! This project is a very small hyper/tokio backend API. However, I also work on a CLI tool [1] I created which has very fast recompile times, usually less than a second. Mostly because it's pretty light on dependencies. [1]: https://github.com/jakedeichert/mask https://github.com/jakedeichert/mask
- vlovich123 6y ago
  In C++ land I’ve experienced the gamut from multi-minute clean builds to 30 minutes to 1.5 hours on the latest & greatest CPUs, even with ccache. 14s sounds like paradise to me.
  - O_H_E 6y ago
    Honest question, is there really no way to know whether what you wrote is working or not without waiting 20min-1H?? That sounds like hell. Where these incremental compilation?
    - vlovich123 5y ago
      That’s usually clean builds. Incremental builds can be in the instant to 1-5 min depending on what line of code you change (deploy a new compiler and you’re probably rebuilding everything).
      - O_H_E 5y ago
        Oh ok, that makes much more sense.
- 6y ago

pjmlp 6y ago

On the other hand I just gave up expecting sscache to build on my travel netbook, > 400 dependencies, really? apt-get install rustlibXYZ-dev can't come soon enough.

ChrisSD 6y ago
sscache has pre-built binaries on their releases page. Or you can do: cargo install --no-default-features To avoid building support for all storage backends. You can then add back any backend you're actually using.
- pjmlp 6y ago
  Thanks, I failed to find how to download them, besides the note that a login is expected, I could not find the artifacts among GitHub workflows. However, the point is actually how it is a big hurdle to overcome having to compile crates with npm like dependency trees all from scratch, vs other compiled languages integrated into the distribution package repositories.

Decabytes 6y ago

Are long compile times just a product of Rust’ design? Or is there an issue with the compiler. Based on my experience with rust I would say it’s the former. It’s doing so many more checks for you then other compilers to prevent lots of problems that it’s bound to be slow. I appreciate that they continue to speed up compilation/add flags. If the compiler gets slower every release it will eventually become unusable. I think the long compilation times will continue to get worse in the future as the rust compiler gets better at checking for errors, and I’m not sure what can really be done about that. One thing that irks me is that they throw around the phrase zero cost abstraction a lot. And I just don’t buy it. Sure I get it, lots of things can be monomorphized so we don’t pay a runtime cost, but there is still a large compilation hit to do that, and it’s not zero cost to the programmer. One need only to look at Chandler Carruth’s cppcon talk “there are no zero cost abstractions” to understand that.

ChrisSD 6y ago
But that's just playing with definitions. Rephrase it as "zero runtime cost abstraction" and you get back to the intended meaning. Incidentally your first question is one that's very amenable to testing. When does the Rust compiler spend most of it's time? Is it at the checking stage?
- optymizer 6y ago
  Oh come on now, do you and the parent really have to be those guys who go "well technically they're not zero cost because the programmer has to wait"? Yes, you're technically correct, but also being unreasonable. It was pretty clear that "zero cost abstractions" refers to the runtime aspect. That is the default meaning, there's nothing to "get back to".
  - ChrisSD 6y ago
    Huh? That was exactly my point.
- aw1621107 6y ago
  > When does the Rust compiler spend most of it's time? Is it at the checking stage? rustc has a self-profiler that can be used to answer this question [0], as well as a mode that times compiler passes [1]. There's no single reason the Rust compiler is slow, as it depends quite heavily on the code being compiled. For some codebases, LLVM code takes up most of the time; in other codebases (e.g., extremely generic-heavy codebases), it'll be checking-related passes. [0]: https://github.com/rust-lang/measureme/blob/master/summarize/Readme.md https://github.com/rust-lang/measureme/blob/master/summarize... [1]: https://wiki.alopex.li/WhereRustcSpendsItsTime https://wiki.alopex.li/WhereRustcSpendsItsTime

orf 6y ago

Why is this not the default? What’s the downside?

est31 6y ago
I think because split-debuginfo is recently stabilized (Nov 2020). https://github.com/rust-lang/rust/pull/79570 https://github.com/rust-lang/rust/pull/79570 Maybe the default will be switched in the future.
- irh 6y ago
  It's already switched on nightly. https://github.com/rust-lang/cargo/pull/9298 https://github.com/rust-lang/cargo/pull/9298 https://doc.rust-lang.org/nightly/cargo/reference/profiles.html#split-debuginfo https://doc.rust-lang.org/nightly/cargo/reference/profiles.h...

est31 6y ago

You can also turn off debuginfo completely. Personally, someone who does printf debugging, I mainly need it to debug segfaults, which are really rare in Rust. Sometimes the call stack of a panic is useful as well, but if I need debuginfo I can just re-enable it. https://github.com/est31/cargo-udeps/commit/e550d93c7a6d756e639ecd7f117ea9f9328d3b38 https://github.com/est31/cargo-udeps/commit/e550d93c7a6d756e...

snake_case 6y ago
Author here. I just tried disabling debug info (debug = 0) for the first time and it looks like my recompile times shave off another 500-1000ms which isn't bad!
- est31 6y ago
  Glad to have been of help!

person_of_color 6y ago

Rustaceans discover there’s no free lunch, about 10 years after the C++ templaters

nindalf 6y ago

Compile times in rustc have been steadily improving with time, as shown here - https://arewefastyet.rs https://arewefastyet.rs. Every release doesn't make every workload faster, but over a long time horizon, the effect is clear. Rust 1.34 was released in April 2019 and since then many crates have become 33-50% faster to compile, depending on the hardware and the compiler mode (clean/incremental, check/debug/release). Interestingly, the speedup mentioned in OP won't show up in these charts because that's a change on macOS and these benchmarks were recorded on Linux. What is expected to be a gamechanger is the release of cranelift in 2021 or 2022. It's an alternate debug backend that promises much faster debug builds.