3 ms·

I'm sure we can all agree that Cargo is the best thing since sliced arrays, but I've finally reached the point where we have enough different Rust projects at w

I'm sure we can all agree that Cargo is the best thing since sliced arrays, but I've finally reached the point where we have enough different Rust projects at work that rebuilding the world from scratch all the time and having ~40 GiB `target/` directories scattered around the place is getting old. I'm not meaning to criticize Rust or Cargo here; they're both doing their jobs just fine. But I do find myself craving a different compromise for much of what I/we use Rust for today. And I'm really hoping that Wasm Components (and WASI) will be that different compromise — e.g.: - Don't rebuild an entire HTTP stack from scratch for every tool that happens to use HTTP. - Therefore most projects won't have enormous `target/` dirs. - Reuse components built in CI for Linux directly on a Mac dev machine (cross platform). - Mix and match new processor architectures available in AWS without building two versions of everything. - Also reuse some components in the web browser (yes, we actually have real, boring use cases for this). - Wait, I'll be able to do all of this and still be writing Rust? Shut up and take my money.

rtpg 3y ago
I do wonder how much sharing could be done at a project level for libraries. Would it be possible to have at least debug builds with matching compilation settings be compiled down into a shared place?
- jeffparsons 3y ago
  Definitely. Much of this can be mitigated by using a shared target directory, which is at least somewhat supported in Cargo. I should probably start doing that and promote it at work if I don't run into any problems. Some clever garbage collection would help, too, but I imagine different people would have different and very strong opinions about how that should work. The fact that two projects that use the same library may enable different features also complicates things. Again, a of this can be mitigated. I'm looking for a step change for application development, though — i.e. not 5 minute build reduced to 2 minutes, but 5 minutes reduced to 5 seconds (and a correspondingly tiny target directory). That's what excites me about WASI in this context at least. Leaving WASI aside for a minute, I do wonder how much more could be saved in local disk space and compilation time across projects (and hosts, a la sccache) if this was a high priority goal for the Rust project. E.g. even if the MIR for a crate with two different sets of feature flags enabled ends up substantially different, would they still compress well against each other if a lot remained common?
  - regularfry 3y ago
    Once upon a time I briefly looked at symlinking rubygem install dirs from global to project-specific directories (because a project-specific $GEM_HOME avoided functionally all the problems with just about any other approach, and is still what I use today). Functionally it worked, it just needed some tooling to make it easy. It sounds like what you might want is a shared global dir that uses hashes of feature flags to separate crate installs in the same target directory, then some after-the-fact GC to hardlink matching files across different builds of the same crate. Then you can symlink from there into your local project target.

rtpg 3y ago

I do wonder how much sharing could be done at a project level for libraries. Would it be possible to have at least debug builds with matching compilation settings be compiled down into a shared place?

jeffparsons 3y ago
Definitely. Much of this can be mitigated by using a shared target directory, which is at least somewhat supported in Cargo. I should probably start doing that and promote it at work if I don't run into any problems. Some clever garbage collection would help, too, but I imagine different people would have different and very strong opinions about how that should work. The fact that two projects that use the same library may enable different features also complicates things. Again, a of this can be mitigated. I'm looking for a step change for application development, though — i.e. not 5 minute build reduced to 2 minutes, but 5 minutes reduced to 5 seconds (and a correspondingly tiny target directory). That's what excites me about WASI in this context at least. Leaving WASI aside for a minute, I do wonder how much more could be saved in local disk space and compilation time across projects (and hosts, a la sccache) if this was a high priority goal for the Rust project. E.g. even if the MIR for a crate with two different sets of feature flags enabled ends up substantially different, would they still compress well against each other if a lot remained common?
- regularfry 3y ago
  Once upon a time I briefly looked at symlinking rubygem install dirs from global to project-specific directories (because a project-specific $GEM_HOME avoided functionally all the problems with just about any other approach, and is still what I use today). Functionally it worked, it just needed some tooling to make it easy. It sounds like what you might want is a shared global dir that uses hashes of feature flags to separate crate installs in the same target directory, then some after-the-fact GC to hardlink matching files across different builds of the same crate. Then you can symlink from there into your local project target.