4 ms·

I have a rather large side project in Rust using cargo workspaces and compilation times do not bother me in the slightest, because I'm running incremental build

by qppo 6y ago

I have a rather large side project in Rust using cargo workspaces and compilation times do not bother me in the slightest, because I'm running incremental builds for everything. On the other hand I tried introducing Rust for a small part of a larger ecosystem and the cold compile times were so bad we rewrote the functionality in C. It shaved minutes off our CI build times, which costs actual money.

asdkhadsj 6y ago
> On the other hand I tried introducing Rust for a small part of a larger ecosystem and the cold compile times were so bad we rewrote the functionality in C. It shaved minutes off our CI build times, which costs actual money. Yea, we have this issue (as a shop now using Rust for all of our backend). I have a couple of hacks in place to cache the majority of the build thankfully, only needing to compile source code unless something changes. When our build cache works our builds take ~60s. When it doesn't, ~15m.
- qppo 6y ago
  To be fair to Rust, this is certainly true of other languages. C++ builds can also be compile time hogs. What did you wind up doing to cache your builds? I've tried a few different hacks but none have stuck.
  - pjmlp 6y ago
    Indeed, but unless you are doing some crazy meta-programming, you can keep reusing the same binary libraries, as inside the same company most projects tend to have similar build flags anyway. Alternatively it is also quite common to use dynamic libraries, or stuff like COM, XPC.
  - jfkebwjsbx 6y ago
    C++ build times get high only if you start messing with the type system and ask for it to be recompiled every single time again and again. Rust is slower overall, which is why people tend to complain. And if you start messing around like in C++, then you get even crazier times. But in neither case it is a dealbreaker compared to other languages. Go proponents claim compilation speed is everything, which is suspicious. I do not need to run my code immediately if I am programming in C++ or Rust. And if I am really doing something that requires interactivity, I should be doing it another way, not recompiling every time...
    - Insanity 6y ago
      Having a fast feedback loop helps with staying in the flow. Compile times need to be short for this.
      - jfkebwjsbx 6y ago
        What is "staying in the flow"? Recompiling fast helps the most when learning to program, but not for actual applications with some complexity. Many applications do not even have meaningful output by just running it, for instance they may take a long time to compute something meaningful.
        gameswithgo 6y ago
        it depends on the problem domain. when working on say a desktop app with graphical ui features it can be very useful to be able to change/experiment quickly. with an n tier web application you wont often be able to do that anyway.
        pjmlp 6y ago
        And this is why in its current state you wouldn't be seeing something like RustUI or Rust Playgrounds.
        pdimitar 6y ago
        > What is "staying in the flow"? Run the code (with some debug logging statements, very likely), find a small mistake, make a few characters worth of correction, press your IDE's keyboard shortcut for recompilation and re-running. If the last step is slow you can lose momentum and motivation to iterate quickly on the problem at hand. Sure it doesn't apply to a lot of projects, that's true. But it's not charitable to claim it only applies when learning.
      - rmdashrfstar 6y ago
        That’s why you write unit and integration tests :) That’s how you get a fast feedback loop, instead of looking for it in compilation
    - 6y ago
  - asdkhadsj 6y ago
    Oh yea, I wasn't picking on Rust. If anything I tend to defend Rust haha. As far as what we did to cache, nothing fancy - using Docker build layers. I add my Cargo files (lock/toml), include a stub source lib.rs or main.rs to make it build with a fake source, and then build the project. This builds all the dependencies. It also builds a fake binary/lib for your project, so you need to strip that from the target directory. Something like `rm -rf target/your?project?name*` (I use ? to wildcard _ and -) If you do that in one layer, your dependencies will be cached with that docker image. In the next layer you can add your source like normal, compile it, and you'll be set. We lose our cache frequently though because we're not taking special care to centralize or persist the layer cache. We should, for sanity.
    - giovannibonetti 6y ago
      > We lose our cache frequently though because we're not taking special care to centralize or persist the layer cache. We should, for sanity. Do you use digests (@sha256:...) in the Dockerfile source image (FROM ...) to ensure you are always using the same layer? If not, that is probably the reason why your cache is falling so often.
      - asdkhadsj 6y ago
        Nope, but that shouldn't be required I'd think? It's not locally, for example - as long as each layers resulting hash has not changed the cache is reused. Likewise on CI, if I repeatedly build the caches never miss, it works great. It's more a problem of sometimes, after a days or a week, the layers seem to be gone. But never in-between your pushes to a PR, for example, those always seem to stay. I think that's happening is our garbage collection on old docker layers is being too aggressive. But because it works so well between commits, pushes to CI and etc - I don't worry about it. The majority of the time I want a cache to work, it works. So it's been a low priority thing for me to fix haha. edit: Oh, and I forgot, we may have CI jobs running on different machines. Which of course would also miss caches, since we're not persisting the layers on our registry. I'm not positive on this one though, since like I said it never seems to fail between commit pushes _(say to a PR during review, dev, etc)_. /shrug
MaxBarraclough 6y ago
How do the Mozilla folks cope with these build-time issues?
- tick_tock_tick 6y ago
  Not a ton of Rust in firefox yet.
  - steveklabnik 6y ago
    2.6 million LOC according to https://4e6.github.io/firefox-lang-stats/ https://4e6.github.io/firefox-lang-stats/ (this includes vendored deps, but that still gets built, of course)
- zelly 6y ago
  because, it's checks notes % curl -Lo mozilla.zip https://hg.mozilla.org/mozilla-central/archive/tip.zip % unzip mozilla.zip && cd mozilla-central-* % find . -type f \( -iname '*.cpp' -o -iname '*.cc' -o -iname '*.cxx' -o -iname '*.c' -o -iname '*.h' -o -iname '*.hpp' -o -iname '*.hxx' -o -iname '*.hh' \) | wc -l 32347 % find . -type f -name '*.rs' | wc -l 7669 % find . -type f -iname '*.js' | wc -l 70101 ...still mostly C and C++ and more JavaScript than Rust.
  - orf 6y ago
    Does that include any of the dependencies? [1] 1. https://hg.mozilla.org/mozilla-central/file/tip/Cargo.toml https://hg.mozilla.org/mozilla-central/file/tip/Cargo.toml
    - gpm 6y ago
      Cargo.lock is probably the more useful file for listing dependencies https://hg.mozilla.org/mozilla-central/file/tip/Cargo.lock https://hg.mozilla.org/mozilla-central/file/tip/Cargo.lock
- KwanEsq 6y ago
  1) Front-end engineers don't compile any Rust or C++ etc. code, instead they use artifact builds which download pre-built binary artifacts from Mozilla's continuous integration builds, and their local builds only "compile" the HTML/JS/CSS code. 2) They use sccache locally for caching binary builds artifacts. 3) In-office they use stuff like distributed compiles/a beefy compiler machine on the network rather than necessarily the one they are using
antpls 6y ago
Maybe that's a good opportunity to learn about Bazel remote cache server (although that also costs money and time to operate)

by qppo 6y ago

asdkhadsj 6y ago

> On the other hand I tried introducing Rust for a small part of a larger ecosystem and the cold compile times were so bad we rewrote the functionality in C. It shaved minutes off our CI build times, which costs actual money. Yea, we have this issue (as a shop now using Rust for all of our backend). I have a couple of hacks in place to cache the majority of the build thankfully, only needing to compile source code unless something changes. When our build cache works our builds take ~60s. When it doesn't, ~15m.

qppo 6y ago
To be fair to Rust, this is certainly true of other languages. C++ builds can also be compile time hogs. What did you wind up doing to cache your builds? I've tried a few different hacks but none have stuck.
- pjmlp 6y ago
  Indeed, but unless you are doing some crazy meta-programming, you can keep reusing the same binary libraries, as inside the same company most projects tend to have similar build flags anyway. Alternatively it is also quite common to use dynamic libraries, or stuff like COM, XPC.
- jfkebwjsbx 6y ago
  C++ build times get high only if you start messing with the type system and ask for it to be recompiled every single time again and again. Rust is slower overall, which is why people tend to complain. And if you start messing around like in C++, then you get even crazier times. But in neither case it is a dealbreaker compared to other languages. Go proponents claim compilation speed is everything, which is suspicious. I do not need to run my code immediately if I am programming in C++ or Rust. And if I am really doing something that requires interactivity, I should be doing it another way, not recompiling every time...
  - Insanity 6y ago
    Having a fast feedback loop helps with staying in the flow. Compile times need to be short for this.
    - jfkebwjsbx 6y ago
      What is "staying in the flow"? Recompiling fast helps the most when learning to program, but not for actual applications with some complexity. Many applications do not even have meaningful output by just running it, for instance they may take a long time to compute something meaningful.
      - gameswithgo 6y ago
        it depends on the problem domain. when working on say a desktop app with graphical ui features it can be very useful to be able to change/experiment quickly. with an n tier web application you wont often be able to do that anyway.
      - pjmlp 6y ago
        And this is why in its current state you wouldn't be seeing something like RustUI or Rust Playgrounds.
      - pdimitar 6y ago
        > What is "staying in the flow"? Run the code (with some debug logging statements, very likely), find a small mistake, make a few characters worth of correction, press your IDE's keyboard shortcut for recompilation and re-running. If the last step is slow you can lose momentum and motivation to iterate quickly on the problem at hand. Sure it doesn't apply to a lot of projects, that's true. But it's not charitable to claim it only applies when learning.
    - rmdashrfstar 6y ago
      That’s why you write unit and integration tests :) That’s how you get a fast feedback loop, instead of looking for it in compilation
  - 6y ago
- asdkhadsj 6y ago
  Oh yea, I wasn't picking on Rust. If anything I tend to defend Rust haha. As far as what we did to cache, nothing fancy - using Docker build layers. I add my Cargo files (lock/toml), include a stub source lib.rs or main.rs to make it build with a fake source, and then build the project. This builds all the dependencies. It also builds a fake binary/lib for your project, so you need to strip that from the target directory. Something like `rm -rf target/your?project?name*` (I use ? to wildcard _ and -) If you do that in one layer, your dependencies will be cached with that docker image. In the next layer you can add your source like normal, compile it, and you'll be set. We lose our cache frequently though because we're not taking special care to centralize or persist the layer cache. We should, for sanity.
  - giovannibonetti 6y ago
    > We lose our cache frequently though because we're not taking special care to centralize or persist the layer cache. We should, for sanity. Do you use digests (@sha256:...) in the Dockerfile source image (FROM ...) to ensure you are always using the same layer? If not, that is probably the reason why your cache is falling so often.
    - asdkhadsj 6y ago
      Nope, but that shouldn't be required I'd think? It's not locally, for example - as long as each layers resulting hash has not changed the cache is reused. Likewise on CI, if I repeatedly build the caches never miss, it works great. It's more a problem of sometimes, after a days or a week, the layers seem to be gone. But never in-between your pushes to a PR, for example, those always seem to stay. I think that's happening is our garbage collection on old docker layers is being too aggressive. But because it works so well between commits, pushes to CI and etc - I don't worry about it. The majority of the time I want a cache to work, it works. So it's been a low priority thing for me to fix haha. edit: Oh, and I forgot, we may have CI jobs running on different machines. Which of course would also miss caches, since we're not persisting the layers on our registry. I'm not positive on this one though, since like I said it never seems to fail between commit pushes _(say to a PR during review, dev, etc)_. /shrug

MaxBarraclough 6y ago

How do the Mozilla folks cope with these build-time issues?

tick_tock_tick 6y ago
Not a ton of Rust in firefox yet.
- steveklabnik 6y ago
  2.6 million LOC according to https://4e6.github.io/firefox-lang-stats/ https://4e6.github.io/firefox-lang-stats/ (this includes vendored deps, but that still gets built, of course)
zelly 6y ago
because, it's checks notes % curl -Lo mozilla.zip https://hg.mozilla.org/mozilla-central/archive/tip.zip % unzip mozilla.zip && cd mozilla-central-* % find . -type f \( -iname '*.cpp' -o -iname '*.cc' -o -iname '*.cxx' -o -iname '*.c' -o -iname '*.h' -o -iname '*.hpp' -o -iname '*.hxx' -o -iname '*.hh' \) | wc -l 32347 % find . -type f -name '*.rs' | wc -l 7669 % find . -type f -iname '*.js' | wc -l 70101 ...still mostly C and C++ and more JavaScript than Rust.
- orf 6y ago
  Does that include any of the dependencies? [1] 1. https://hg.mozilla.org/mozilla-central/file/tip/Cargo.toml https://hg.mozilla.org/mozilla-central/file/tip/Cargo.toml
  - gpm 6y ago
    Cargo.lock is probably the more useful file for listing dependencies https://hg.mozilla.org/mozilla-central/file/tip/Cargo.lock https://hg.mozilla.org/mozilla-central/file/tip/Cargo.lock
KwanEsq 6y ago
1) Front-end engineers don't compile any Rust or C++ etc. code, instead they use artifact builds which download pre-built binary artifacts from Mozilla's continuous integration builds, and their local builds only "compile" the HTML/JS/CSS code. 2) They use sccache locally for caching binary builds artifacts. 3) In-office they use stuff like distributed compiles/a beefy compiler machine on the network rather than necessarily the one they are using

antpls 6y ago

Maybe that's a good opportunity to learn about Bazel remote cache server (although that also costs money and time to operate)