5 ms·

It looks like the author is most bothered by compile times of dependencies. Cargo needs to do better with shared caches (so you compile each dep at most once p

by pornel 7y ago

It looks like the author is most bothered by compile times of dependencies. Cargo needs to do better with shared caches (so you compile each dep at most once per machine) or ability to get precompiled crates (so you don't even compile it). Incremental improvements of compiler speed or trimming of individual dependencies won't bring the 10x improvement it needs.

holy_city 7y ago
You can kind of hack a shared cache using cargo workspaces (put all your rust projects in one directory as part of a workspace) but that is far from ideal. All your projects will share a target directory and Cargo.lock. I agree though. I don't mind binary sizes in the single megabytes for a release build, and I don't mind clean/rebuild build times that are a few dozen seconds. I do mind the 500MB of build artifacts in a single target directory, multiplied by all the projects I have that share the same version of serde/winapi/syn/quote/insert-common-dependency-here.
leggomylibro 7y ago
One of the biggest issues that I face with Rust is that its builds are enormous, and I often work on machines with limited disc space. The actual binary sizes are fine - even with embedded devices that have <1MB of program space, there doesn't seem to be much bloat. But I need to remember to clean every project when I finish working on it, because otherwise the build folders will eat 100s-1000s of megabytes each on a 16-32GB partition. I really appreciate Rust's inclusive approach to learning and teaching, but I can't justify using it for education on ultra-affordable machines for that reason. People often scatter one-off projects all over the place as they learn, and when they run out of space it takes a long time to clean everything up. So I would also be very in favor of some sort of simple shared package cache.
- asgeir 7y ago
  Have you considered doing something like this: cargo build --target-dir ~/build-artifacts/$(basename $(pwd)) Then you can have something like this in your ~/.config/user-tmpfiles.d/clean_build.conf d /home/user/build-artifacts - - - 1d - You can then do cleanup by calling systemd-tmpfiles --user Or you can use the tmpfiles timer systemctl enable --user --now systemd-tmpfiles-clean.timer
  - leggomylibro 7y ago
    That sounds like an interesting idea, thanks! Can that '--target-dir' argument be added in the build.rs script?
    - asgeir 7y ago
      It looks like you can set it using a CARGO_TARGET_DIR environment variable or build.target-dir config value. So I guess you could create a $HOME/.cargo/config file with the following in it (letting you skip the --target-dir part): [build] target-dir = "/home/user/build-artifacts" But I don't know if the different projects will trample each other that way. If that's the case you could just go with a .cargo/config per-project targeting a subdirectory of the build-artifacts directory. https://doc.rust-lang.org/cargo/commands/cargo-build.html https://doc.rust-lang.org/cargo/commands/cargo-build.html https://doc.rust-lang.org/cargo/reference/config.html https://doc.rust-lang.org/cargo/reference/config.html
maxmcd 7y ago
Can you not do this by setting CARGO_TARGET_DIR to something universal? Maybe there is a downside here that I haven't considered.
- est31 7y ago
  One downside is shared locks. You can't compile multiple projects the same time. Don't know of any other downsides.
Scuds 7y ago
Cargo compiles dependencies in parallel so if you have lots of cores you'll hear your fans spin up. I see a lot of difference between my i5 laptop and i9 desktop.
- Macha 7y ago
  As a case in point, I have a project that uses amethyst and nalgebra (and their 200 transitive dependencies). On upgrading from an OC i5-4670k to a r9-3900x, my compile times for a clean release build went from 20 minutes to 2.
  - lasagnaphil 7y ago
    But two minutes is still a lot of time to wait for a build, especially if you’re doing gamedev and want to prototype something fast. It seems nalgebra is the culprit here: because Rust doesn’t yet support const generics, it has to use some hacky type-level metaprogramming to represent numbers, and that will definitely destroy build times.
    - dpc_pw 7y ago
      2 minutes for a full build from scratch. Incremental builds afterwards take seconds, though unfortunately linking of big projects can still sometimes takes up to around a minute.
      - pjmlp 7y ago
        My toy project that I ported from a Gtkmm article done in the days of "The C/C++ Users Journal" takes around 25 minutes to build from scratch on a Asus 1215B netbook (dual core, 8GB, HDD). The original code, after being migrated to an up to date version of Gtkmm, takes a couple of seconds with GCC 7, not more than one minute if at all. The big difference? I don't need to compile from scratch all the 3rd party dependencies. With every release from Rust I do a clean build to assert how much it has improved. It was much worse, so congrats on the work achieved thus far, but it is still a pain to set up a project from scratch.
_nhynes 7y ago
Have you tried sccache [0]? It doesn’t always choose to cache a dependency, but it helps about 70% of the time. Anecdotally, it hastened a release build of a pretty standard CLI tool (with incremental compilation) by almost 4x. In the context of resource-constrained machines, one can always host it remotely on S3. (or mount an NFS share as the CARGO_TARGET_DIR, if you’re feeling adventurous or want fast CI) [0] https://github.com/mozilla/sccache https://github.com/mozilla/sccache
- sitkack 7y ago
  Can I use this across all my projects and machines and have shared, cached compilation of Rust crates?
  - _nhynes 7y ago
    Yes! Even using the local server on your own machine. It’s not as good as incremental, but it really does help when working with multiple projects that use the same deps in the same ways. NB: it works a bit better on not-Mac because staticlibs are deterministic.

by pornel 7y ago

holy_city 7y ago

You can kind of hack a shared cache using cargo workspaces (put all your rust projects in one directory as part of a workspace) but that is far from ideal. All your projects will share a target directory and Cargo.lock. I agree though. I don't mind binary sizes in the single megabytes for a release build, and I don't mind clean/rebuild build times that are a few dozen seconds. I do mind the 500MB of build artifacts in a single target directory, multiplied by all the projects I have that share the same version of serde/winapi/syn/quote/insert-common-dependency-here.

leggomylibro 7y ago

One of the biggest issues that I face with Rust is that its builds are enormous, and I often work on machines with limited disc space. The actual binary sizes are fine - even with embedded devices that have <1MB of program space, there doesn't seem to be much bloat. But I need to remember to clean every project when I finish working on it, because otherwise the build folders will eat 100s-1000s of megabytes each on a 16-32GB partition. I really appreciate Rust's inclusive approach to learning and teaching, but I can't justify using it for education on ultra-affordable machines for that reason. People often scatter one-off projects all over the place as they learn, and when they run out of space it takes a long time to clean everything up. So I would also be very in favor of some sort of simple shared package cache.

asgeir 7y ago
Have you considered doing something like this: cargo build --target-dir ~/build-artifacts/$(basename $(pwd)) Then you can have something like this in your ~/.config/user-tmpfiles.d/clean_build.conf d /home/user/build-artifacts - - - 1d - You can then do cleanup by calling systemd-tmpfiles --user Or you can use the tmpfiles timer systemctl enable --user --now systemd-tmpfiles-clean.timer
- leggomylibro 7y ago
  That sounds like an interesting idea, thanks! Can that '--target-dir' argument be added in the build.rs script?
  - asgeir 7y ago
    It looks like you can set it using a CARGO_TARGET_DIR environment variable or build.target-dir config value. So I guess you could create a $HOME/.cargo/config file with the following in it (letting you skip the --target-dir part): [build] target-dir = "/home/user/build-artifacts" But I don't know if the different projects will trample each other that way. If that's the case you could just go with a .cargo/config per-project targeting a subdirectory of the build-artifacts directory. https://doc.rust-lang.org/cargo/commands/cargo-build.html https://doc.rust-lang.org/cargo/commands/cargo-build.html https://doc.rust-lang.org/cargo/reference/config.html https://doc.rust-lang.org/cargo/reference/config.html

maxmcd 7y ago

Can you not do this by setting CARGO_TARGET_DIR to something universal? Maybe there is a downside here that I haven't considered.

est31 7y ago
One downside is shared locks. You can't compile multiple projects the same time. Don't know of any other downsides.

Scuds 7y ago

Cargo compiles dependencies in parallel so if you have lots of cores you'll hear your fans spin up. I see a lot of difference between my i5 laptop and i9 desktop.

Macha 7y ago
As a case in point, I have a project that uses amethyst and nalgebra (and their 200 transitive dependencies). On upgrading from an OC i5-4670k to a r9-3900x, my compile times for a clean release build went from 20 minutes to 2.
- lasagnaphil 7y ago
  But two minutes is still a lot of time to wait for a build, especially if you’re doing gamedev and want to prototype something fast. It seems nalgebra is the culprit here: because Rust doesn’t yet support const generics, it has to use some hacky type-level metaprogramming to represent numbers, and that will definitely destroy build times.
  - dpc_pw 7y ago
    2 minutes for a full build from scratch. Incremental builds afterwards take seconds, though unfortunately linking of big projects can still sometimes takes up to around a minute.
    - pjmlp 7y ago
      My toy project that I ported from a Gtkmm article done in the days of "The C/C++ Users Journal" takes around 25 minutes to build from scratch on a Asus 1215B netbook (dual core, 8GB, HDD). The original code, after being migrated to an up to date version of Gtkmm, takes a couple of seconds with GCC 7, not more than one minute if at all. The big difference? I don't need to compile from scratch all the 3rd party dependencies. With every release from Rust I do a clean build to assert how much it has improved. It was much worse, so congrats on the work achieved thus far, but it is still a pain to set up a project from scratch.

_nhynes 7y ago

Have you tried sccache [0]? It doesn’t always choose to cache a dependency, but it helps about 70% of the time. Anecdotally, it hastened a release build of a pretty standard CLI tool (with incremental compilation) by almost 4x. In the context of resource-constrained machines, one can always host it remotely on S3. (or mount an NFS share as the CARGO_TARGET_DIR, if you’re feeling adventurous or want fast CI) [0] https://github.com/mozilla/sccache https://github.com/mozilla/sccache

sitkack 7y ago
Can I use this across all my projects and machines and have shared, cached compilation of Rust crates?
- _nhynes 7y ago
  Yes! Even using the local server on your own machine. It’s not as good as incremental, but it really does help when working with multiple projects that use the same deps in the same ways. NB: it works a bit better on not-Mac because staticlibs are deterministic.