8 ms·
Mozilla sccache: ccache with cloud storage
- Sytten 3y agoCan you combine that with cross rs? I positively hate the official GHA cache action so anything to replace that would be nice. But we cross compile with cross rs.
- Scarjit 3y agoIt should work if you modify the cross docker image, so that it uses the sccache executable as wrapper
- sgammon 3y agoAnother alternative is Buildless with the Actions setup step. This sets up a remote and local endpoint for sccache inside actions, with a connection up to remote caching by Dragonfly https://github.com/buildless/setup https://github.com/buildless/setup
- fifteen1506 3y agoIDK what happened at Mozilla but keep going!
- CaptainOfCoit 3y agoWorth noting that the first commit in sccache git repository was in 2014 (https://github.com/mozilla/sccache/commit/115016e0a83b290dc2dbb366127499fb09168707 https://github.com/mozilla/sccache/commit/115016e0a83b290dc2...). So I suppose that what "happened" happened waay back. Then in 2016 it seems like sccache was re-implemented in Rust (https://github.com/mozilla/sccache/commit/3da89195ce91a576cc58617d76bbdf7be20bced2 https://github.com/mozilla/sccache/commit/3da89195ce91a576cc...), from the initial Python implementation.
- altairprime 3y agoAlso around then, taskcluster happened. https://github.com/taskcluster/taskcluster/commit/54ffef79db5730146d73344cab184e17c5b81910 https://github.com/taskcluster/taskcluster/commit/54ffef79db...
- jandeboevrie 3y agoRegular ccache supports remote storage as well: https://ccache.dev/manual/latest.html#_remote_storage_backends https://ccache.dev/manual/latest.html#_remote_storage_backen...
- CaptainOfCoit 3y agoI guess the difference is that sccache supports cloud storage (S3, R2, Google Cloud Storage, el al) out of the box while cache doesn't, as far as I can tell.
- slavik81 3y agoThe sccache description was written before remote storage was added to ccache. Remote storage is a relatively new feature for ccache, and it didn't exist when sccache was created.
- kinkaid 3y agoThe ability to use the local and remote caches in tandem is the most important feature to me. sccache will use one or the other exclusively, so successive builds will have to pay the latency and bandwidth costs every time an artifact is built rather than just unpacking from the local cache. ccache stages remote artifacts locally by default, so it only pays the network costs once per artifact. In CI builds they are more or less the same, but the local build experience for ccache is much nicer imo.
- sgammon 3y agoboth are supported by buildless as well https://less.build https://less.build
- sgammon 3y ago(ccache is only about caching, not distributing builds, also...)
- wongarsu 3y agosccache is also delightfully simple to set up if you just want local storage. It's my go-to solution for sharing build artifacts between rust projects
- MuffinFlavored 3y ago> It's my go-to solution for sharing build artifacts between rust projects Could you expand a bit please? As somebody who is too cheap to get more than the 256gb MacBook, those heavy ~1gb random target/ directories add up over time. Didn't know there was an easy/cheap way around this?
- wongarsu 3y agosccache mostly helps to cut down build times. For saving space, the simplest trick is to use the same target/ for all your projects, by setting export CARGO_TARGET_DIR=/absolute/path/to/target Then you can just periodically nuke it once it grows too big. You can combine that with sccache (two-line setup as elsewhere in this thread) so nuking your target dir isn't as big of an impact on your build times. Though beware that projects that share a target dir can't build simultaneously. If you have multiple projects open that can get annoying, because it can also effect the language server. I'm that case you can use gentler measures like cargo-sweep.
- IshKebab 3y agoIn my experience the time this saves is generally outweighed by the effort of setting it up and the many hours lost when it goes wrong and you don't think to try a clean build. It's certainly better than ccache in that regard but you really need something like Bazel if you're going to be aggressively caching C++ builds like this (or impure Rust builds). (And if you're using Bazel or one of its brethren then they generally have native remote caching and execution support.)
- goku12 3y agoSetting it up took hardly 5 minutes for me (for Rust). And it hasn't caused any issues so far. Forcing a fresh build of artifacts is also very easy. Meanwhile, the only legitimate problem you mentioned is if it causes a build error and we don't immediately consider it as the source of the issue. But I use the check command so often that it is easy to suspect the cache if check succeeds and build fails.
- IshKebab 3y agoWhen these things go wrong it often doesn't cause a compilation error; it can just cause inexplicable runtime behaviour. You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade (though it's still better than with an actual C/C++ program because you aren't actually editing the C/C++ code generally). And are you actually using remote caching/compilation? Because there's no way you can set that up in 5 minutes.
- goku12 3y ago> And are you actually using remote caching/compilation? Because there's no way you can set that up in 5 minutes. Who said anything about remote caching? You don't need it to benefit from it. It's useful if you build a lot of Rust code. A lot of packages turn up repeatedly as dependencies among several projects. > You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade You're making assumptions again. What is the issue with pure Rust code? Rust isn't like Python needing C or C++ support for process-intensive parts. And much of the C/C++ dependencies are dynamically-linked, with Rust wrappers. I haven't seen many projects that require Rust and C/C++ code to be built together and statically linked. Besides, I haven't heard anyone complain about sccache that much. How prevalent is the degradation anyway?
- sgammon 3y agoIf you’re interested in a drop in remote cache for sccache, check out Buildless We just released S3 and Redis support. https://less.build https://less.build Buildless also supports Gradle, Maven, Bazel, CCache and Turbo
- sgammon 3y agoOur beta is open, just shoot me an email at sam@less.build if you want to try it out!
- phamilton 3y agoI sent an email and it was blocked: "Recipient address rejected: Access denied."
- xjia 3y agoWhat is the benefit of using a remote cache instead of a local ~/.cache directory? Is it only for sharing build results among team members? How do you make sure the build results are not spoofed?
- sgammon 3y agoSharing with team members, sharing with CI, and the ability to pull from more than just what's on your machine (i.e. a larger addressable cache than you are willing to keep on disk). Cache objects also compound across projects, so it's nice to ship them up somewhere and have them nearby when you need them. Re/spoofing, obviously it's all protected with API keys and tokens, and we're working on mechanisms to perform end-to-end encryption. In general, build cache objects are usually addressed by a content-addressable-hash, so that also helps because your build typically knows the content it's looking for and can verify. That isn't true for all tools, though, so we're working to understand where the gaps are and fix them.
- satvikpendem 3y agoI use this with Rust, works great. Simply add a ~/.cargo/config.toml with [build] rustc-wrapper = "/path/to/sccache" And it will work everywhere with cargo. I also like to combine it with the mold linker.
- throwup238 3y agoYou can also set the RUSTC_WRAPPER environment variable to make it system wide.
- pie_flavor 3y agoThe parent comment is also system wide.
- sodality2 3y agoIt should be system wide already because it’s in .home/cargo
- Already__Taken 3y agono you'd have to make that in every users /home to be system wide. Else it's just user-wide
- goodpoint 3y agoPity they are not using GPL or at least MPL.
- heads 3y agoWe used this for a while at Speechmatics but our LM researchers have a well established workflow based on git working copies on NFS /home and we had a lot of instability between sccache and NFS. Is sccache susceptible to cache misses when using full paths as cache keys? It would be very helpful if compiling “/home/heads/project/foo.c” could use the cached result of compiling “/home/thunderbong/project/foo.c”.
- phlip9 3y agosccache only caches if builds are run from the same absolute path, so indeed different home dirs won't work
- tentacleuno 3y agoAs a bystander, what would the reasoning be for doing this? I would have assumed that they'd hash each file and use that as a key in a lookup table.
- MereInterest 3y agoIt sounds like somebody assuming a docker build, where everybody’s build will use the same file path. It’s still a very silly restriction, because not everything occurs within docker.
- sgammon 3y agoIt’s an unfortunate safety tradeoff to guarantee consistency. Better visibility into program behavior could fix it.
- sgammon 3y agoIn some languages, symbols are provided which evaluate to a file’s path or directory parent, so program behavior can vary even for the same content hash. That’s just one way paths can bleed in to violate hermeticity/correctness.
- 3y ago
- szundi 3y agoMy first thought is how to prevent bad guys injecting rootkit binaries into these systems.
- throwawaaarrgh 3y agoIt would be nice if every application in the world didn't have to hack on support for X number of different almost-identical storage vendors who all decided it would be better to have completely incompatible interfaces. NFS isn't horrible, though it's limited. OTOH object storage is limited but has some other advantages. So there needs to either be the latter as a standard that can be adopted by all vendors, apps and OSes, or a new standard that fills the gaps of what apps want.
- sgammon 3y agosccache uses OpenDAL… https://opendal.apache.org/ https://opendal.apache.org/
- throwawaaarrgh 3y agoYeah that's not a storage standard
- sgammon 3y agoYeah it’s also not a problem anymore
- tedunangst 3y agoNext step: store the object files in the blockchain to make a global cache so everyone can compile the browser at the speed of, uh...
- sgammon 3y agoI know you’re joking, but Unison built this without the blockchain cruft on top. Very cool project. When any unique piece of Unison is compiled by anyone, it no longer needs to be compiled by everyone. https://www.unison-lang.org/ https://www.unison-lang.org/
- deleted 3y ago[deleted]
- tdiff 3y agoIs Redis/memcached actually a good fit for C++ object cache? C++ object files may have relatively large size (megabytes).
- sgammon 3y agoOur experience with this at Buildless is that remote caching loses profitability past 100mb or so. As a result, we impose reasonable caps which force a recompilation for the largest outputs, and it ends up being faster. Just our experience and of course mileage varies by project. Redis can do 512mb values but the cap needed for efficiency’s sake is much smaller than that.
- tdiff 3y agoInteresting, I thought the threshold would be smaller (did not check that).
- dataangel 3y agoDoes it still have the issue where it can't be shared between users on different hosts unless they all store their source code at the same path?
- tdiff 3y agoDoes not ccache handle it with some parameter like "base dir"?
- oarfish 3y agoSo does this support native compilation flags or will it break with -march=native like e.g. distcc does?
- hulitu 3y ago> Mozilla sccache: ccache with cloud storage RCE to a new level. Trust us, it is secure.