4 ms·

Cargo assumes you want to cache downloaded/built dependencies at the granularity of a unix user account. It's very easy to break things if you try to force curr

by gue5t 10y ago

Cargo assumes you want to cache downloaded/built dependencies at the granularity of a unix user account. It's very easy to break things if you try to force current cargo cache dependencies at the per-project level and there's been no interest in caching things at a per-machine or shared-between-machines level. If duplicating compilation work is desirable to ensure some amount of noninterference, it should be supported to have a package cache per project. If we can treat Cargo as a pure function from inputs to outputs, we should be sharing these build results across the Internet, at least within the set of machines used by one person. Cargo also refuses to follow the XDG directory specification for its per-unix-user configuration; it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME, but instead dumps both configuration and downloaded/compiled stuff in ~/.cargo.

yberreby 10y ago
> If we can treat Cargo as a pure function from inputs to outputs, we should be sharing these build results across the Internet, at least within the set of machines used by one person. AFAIK, all it takes is a single `build.rs` to make rustc, and therefore Cargo, an impure 'function'. I'm not sure about compiler plugins, but I expect them to behave the same way.
dikaiosune 10y ago
There's an ongoing discussion of using build caches with Cargo: https://internals.rust-lang.org/t/is-a-shared-build-cache-a-good-fit-for-cargo https://internals.rust-lang.org/t/is-a-shared-build-cache-a-... TLDR: There's interest, but some wrinkles need to be sorted out.
- cm3 10y ago
  I like Nix-style caches myself, but a gc-cache command is needed in order to clean up only the stuff that isn't referenced anymore. I hope that's a design consideration.
wyldfire 10y ago
> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME I was going to submit an issue but one already exists [1][2] [1] https://github.com/rust-lang/cargo/pull/148 https://github.com/rust-lang/cargo/pull/148 [2] https://github.com/rust-lang/cargo/issues/1734 https://github.com/rust-lang/cargo/issues/1734
steveklabnik 10y ago
You could also change $CARGO_HOME per project, to get a per-project cache.
- BenTheElder 10y ago
  I actually do this with one project, with a very small script that wraps cargo and sets $CARGO_HOME to a project local path before actually calling cargo. I keep a very short cargo.py and cargo.sh with my project. I've experimented with using this to "vendor" dependencies in a local .cargo and so far that seems to work. An early example with discussion of the python version is here: https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_better_way_to_vendor_dependencies/ https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_bet...
- SimonSapin 10y ago
  Servo does this by default. (People who tried Servo but don’t otherwise develop in Rust didn’t like having random files in their home directory.)
masklinn 10y ago
> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME My unix does not follow XDG[0], neither of these are set and the XDG "fallbacks" are utter garbage, now what? [0] hell, only a minority of linux distros do at all
- jhasse 10y ago
  Why are the fallbacks (~/.config and ~/.cache) utter garbage?
- thristian 10y ago
  How can a unix follow or not follow XDG? It's an application level concern, not a distro or kernel level one.
  - steveklabnik 10y ago
    Distros distribute application packages. If those packages follow XDG or not, I would say the distro does or doesn't.
dbaupp 10y ago
> Cargo assumes you want to cache ... built dependencies at the granularity of a unix user account This isn't true: actual build artifacts are cached in a per-project way (in `./target` by default). Cargo does download all packages to a user-shared directory, but this directory is essentially immutable, just a list of the source code. Sharing more broadly than per-project is the form that is unreliable, things can change subtly between projects e.g. different targets, different compiler flags. Of course, cargo has essentially full information about everything involved in a build and so can track this---in the limit caching all the different configurations of each crate version---but I don't think it does currently (I recall an issue about it, but I cannot find it at the moment).
e12e 10y ago
It sounds like there's two obvious work-arounds, and I'm not sure if I'd consider them all that bad: 1) use a caching http proxy for the downloads (in my experience cargo itself is pretty quick), or 2) use a shared user account for building ("sudo -s cargo-build; ... "). I'm not sure concurrently writing to a group-writeable cache-folder is such a hot idea -- even things like apt use a write-lock when doing updates. I suppose a third option is to mount (or probably symlink) ~/.cargo on a filesystem with deduplication -- but that wouldn't buy you caching for free (might work well coupled with a caching http proxy though).

by gue5t 10y ago

yberreby 10y ago

> If we can treat Cargo as a pure function from inputs to outputs, we should be sharing these build results across the Internet, at least within the set of machines used by one person. AFAIK, all it takes is a single `build.rs` to make rustc, and therefore Cargo, an impure 'function'. I'm not sure about compiler plugins, but I expect them to behave the same way.

dikaiosune 10y ago

There's an ongoing discussion of using build caches with Cargo: https://internals.rust-lang.org/t/is-a-shared-build-cache-a-good-fit-for-cargo https://internals.rust-lang.org/t/is-a-shared-build-cache-a-... TLDR: There's interest, but some wrinkles need to be sorted out.

cm3 10y ago
I like Nix-style caches myself, but a gc-cache command is needed in order to clean up only the stuff that isn't referenced anymore. I hope that's a design consideration.

wyldfire 10y ago

> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME I was going to submit an issue but one already exists [1][2] [1] https://github.com/rust-lang/cargo/pull/148 https://github.com/rust-lang/cargo/pull/148 [2] https://github.com/rust-lang/cargo/issues/1734 https://github.com/rust-lang/cargo/issues/1734

steveklabnik 10y ago

You could also change $CARGO_HOME per project, to get a per-project cache.

BenTheElder 10y ago
I actually do this with one project, with a very small script that wraps cargo and sets $CARGO_HOME to a project local path before actually calling cargo. I keep a very short cargo.py and cargo.sh with my project. I've experimented with using this to "vendor" dependencies in a local .cargo and so far that seems to work. An early example with discussion of the python version is here: https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_better_way_to_vendor_dependencies/ https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_bet...
SimonSapin 10y ago
Servo does this by default. (People who tried Servo but don’t otherwise develop in Rust didn’t like having random files in their home directory.)

masklinn 10y ago

> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME My unix does not follow XDG[0], neither of these are set and the XDG "fallbacks" are utter garbage, now what? [0] hell, only a minority of linux distros do at all

jhasse 10y ago
Why are the fallbacks (~/.config and ~/.cache) utter garbage?
thristian 10y ago
How can a unix follow or not follow XDG? It's an application level concern, not a distro or kernel level one.
- steveklabnik 10y ago
  Distros distribute application packages. If those packages follow XDG or not, I would say the distro does or doesn't.

dbaupp 10y ago

> Cargo assumes you want to cache ... built dependencies at the granularity of a unix user account This isn't true: actual build artifacts are cached in a per-project way (in `./target` by default). Cargo does download all packages to a user-shared directory, but this directory is essentially immutable, just a list of the source code. Sharing more broadly than per-project is the form that is unreliable, things can change subtly between projects e.g. different targets, different compiler flags. Of course, cargo has essentially full information about everything involved in a build and so can track this---in the limit caching all the different configurations of each crate version---but I don't think it does currently (I recall an issue about it, but I cannot find it at the moment).

e12e 10y ago

It sounds like there's two obvious work-arounds, and I'm not sure if I'd consider them all that bad: 1) use a caching http proxy for the downloads (in my experience cargo itself is pretty quick), or 2) use a shared user account for building ("sudo -s cargo-build; ... "). I'm not sure concurrently writing to a group-writeable cache-folder is such a hot idea -- even things like apt use a write-lock when doing updates. I suppose a third option is to mount (or probably symlink) ~/.cargo on a filesystem with deduplication -- but that wouldn't buy you caching for free (might work well coupled with a caching http proxy though).