5 ms·
If you’re interested in a drop in remote cache for sccache, check out Buildless We just released S3 and Redis support. https://less.build https://less.build B
by sgammon 3y ago
If you’re interested in a drop in remote cache for sccache, check out Buildless
We just released S3 and Redis support. https://less.build https://less.build
Buildless also supports Gradle, Maven, Bazel, CCache and Turbo
- sgammon 3y agoOur beta is open, just shoot me an email at sam@less.build if you want to try it out!
- phamilton 3y agoI sent an email and it was blocked: "Recipient address rejected: Access denied."
- xjia 3y agoWhat is the benefit of using a remote cache instead of a local ~/.cache directory? Is it only for sharing build results among team members? How do you make sure the build results are not spoofed?
- sgammon 3y agoSharing with team members, sharing with CI, and the ability to pull from more than just what's on your machine (i.e. a larger addressable cache than you are willing to keep on disk). Cache objects also compound across projects, so it's nice to ship them up somewhere and have them nearby when you need them. Re/spoofing, obviously it's all protected with API keys and tokens, and we're working on mechanisms to perform end-to-end encryption. In general, build cache objects are usually addressed by a content-addressable-hash, so that also helps because your build typically knows the content it's looking for and can verify. That isn't true for all tools, though, so we're working to understand where the gaps are and fix them.
- sgammon 3y ago(Fwiw, group conversation encryption tech like MLS is somewhat applicable, and that's the sort of pattern we're looking at, but it would be cool to know if that's moving to you on the problem of safety w.r.t. builds.)
- Thorrez 3y ago>In general, build cache objects are usually addressed by a content-addressable-hash How does that work? I would think the simplest case of a build object that needs to be cached is a .o file created from a .c file. The compiler sees the .c file and can determine its hash, but how can the compiler determine the hash of the .o file to know what to look up in the cache? I think the compiler would need to perform the lookup using the hash of the .c file, which isn't a hash of the data in the cache.
- sgammon 3y agoIn Bazel’s case and other cases, build cache objects are held in CAS and then referenced from other keys. I believe BuildXL from Microsoft also works this way. Of course one other advantage to build caches is they are verifiable: the intent is to produce the exact same output as a normal call, and that’s easily checked on the client side. No question that build caching poses inherent supply chain risks though and that’s part of what we want to solve. I think people are hesitant to trust build caching for good reason until there are safer mechanisms and better cryptographic patterns applied.
- krupan 3y agoWhen a .o is stored in the cache it is associated with the hash of the .c file
- aseipp 3y agoIn the case of the Remote Execution/Cache API used by Bazel among others[1] at least, it's a bit more detailed. There's an "ActionCache" and an actual content-addressed cache that just stores blobs ("ContentAddressableStorage"). When you run a `gcc -O2 foo.c -o foo.o` command (locally or remotely; doesn't matter), you upload an "Action" into the action cache, which basically said "This command was run. As a result it had this stderr, stdout, error code, and these input files read and output files written." The input and output files are referenced by the hash of their contents, in this case, and they get uploaded into the CAS system. Most importantly you can look up an action in the ActionCache without actually running it, provided you have the inputs at hand. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o, so you can just instantly download it from the CAS. So there are a few more moving parts to make it all work. But the system really is ultimately content-addressed, for the most part. [1] https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto#L555 https://github.com/bazelbuild/remote-apis/blob/main/build/ba...
- aseipp 3y agoNot just team members; if you make your cache publicly readable, contributors to e.g. your GitHub/GitLab/Whatever project can also use them and get really fast builds, the first time they try to contribute. So a remote cache is nice to have, if it's seamless. Nix works this way by default (and much of the community operates caches like this) and it can be a massive, massive time saver. > How do you make sure the build results are not spoofed? What do you mean "spoofed?" As in, someone put an evil artifact in the cache? Or overwrote an existing artifact with a new one? Or someone just stole your developers access and started shoving shit in there? There's a whole bunch of small details here that really matter to understand what security/integrity properties you want the cache to uphold. FWIW, I've been looking into this in Buck2/Bazel land, and my understanding is that most large orgs just use some kind of terminating auth proxy that the underlying connection/flow/build artifacts can be correlated back to. So you know this cache artifact was first inserted by build B, done by user X, who authenticated with their key K, etc etc.
- sgammon 3y agoExactly — just like Git, everything is ultimately identified with a key which can tie back to a stable identity thru OIDC or similar mechanisms. At least that’s how we did it.
- yjftsjthsd-h 3y agoNix only caches at the package level, doesn't it?
- sgammon 3y agoNix is different, yeah, and it won’t wire together a build cache for you. Nix is great for many things of course, it’s just not a replacement for sccache per se Nix + sccache would probably be pretty great for preserving paths and environment, which is really healthy for build caching in general.
- mikepurvis 3y agoProperly handling cargo and bazel builds in nix is very much not a solved problem, and Nix's resistance to allowing ccache into the sandbox for purity reasons definitely exacerbates the problem.
- mgaunard 3y agoI built my own build system that does something similar. I've set it up at work with two S3 buckets: trusted and untrusted. CI/CD read/write from trusted only. Developers read/write from untrusted, and read-only from trusted.
- sgammon 3y agoWe decided to back our main cache with in-memory storage for spicier performance. I’m curious how well S3 has worked for you here? Is it fast enough? Or, maybe the blobs you’re dealing with are on the bigger end? That would also make sense
- mgaunard 3y agoEach object file (.o) has a unique hash and is stored as thehash.o. It's certainly much faster to download the .o than it is to build it. Once it's downloaded it stays on the local filesystem until it's garbage-collected.
- sgammon 3y agoHm, interesting. Our free tier is planned to be this plus R2, so I’m happy to hear S3-style data exchange is working for people. Thanks for sharing
- mgaunard 3y agoThe whole point of S3 is that it is inexpensive. You don't want to pay premium money for terabytes of data that are usually invalidated everyone someone makes a significant change.
- sgammon 3y agoOur blobs end up quite a bit smaller than that, and I can see using S3 for many build cache problems. In our case in-memory worked better with overflow to disk for large or infrequently used objects. R2 is S3 to me, btw, or at least it’s the same API. Object storage as a model is really what I’m asking about. I’m genuinely glad to hear S3 was enough to be impactful.
- throwawaaarrgh 3y agoIt's for sharing and aggregating. Ccache is useful locally, but really shines when combined with Distcc, a distributed compiler. Every host contributes a cache object that other hosts can use, and every host can use the cache object contributed by other hosts. So you don't even have to built it once yourself to benefit from the cache of everyone else. It therefore speeds up multiple hosts/users builds, distributed builds and the dev experience of individuals.
- __float 3y agoIs this only a remote cache for Bazel, but it does not support the remote execution API at all? It's a little worrisome to trust all user outputs when you do not also control the execution of them. (In the "best" case this could mean caching non-reproducible ("works on my machine") build results, in the worst case this could be actively dangerous if a malicious user poisons the build cache.)
- sgammon 3y agoIt’s only a remote cache and that’s deliberate. We see it as much safer to only offer a cache that the user can control and use however they want We would see taking over execution of your build as much more dangerous. No question though that build caching in shared form, in SaaS form, needs extra special attention paid to security. Our product doesn’t introspect cache blobs and in fact doesn’t really want to. Once we figure out how to make the crypto work, we shouldn’t be able to see any of that data at all. Access can be made public for reads (OSS) but is always identified for writes.
- sgammon 3y ago(Also, speaking as a Bazel user now, the Remote Execution APIs have always been a bit brittle and hard to setup, use, and maintain; certainly harder than just setting a cache endpoint. I’ve found that remote execution ends up returning much less benefit than remote caching, but that’s just me and it’s entirely possible I Did It Wrong the whole time)
- sgammon 3y agoYes!! Glad people are thinking about this. We just added Cache Projects which will be launching soon, it should allow this style of public cache sharing. The intent with Buildless is to release a free-first toolchain that helps with build caching in earnest and makes the whole problem much less error prone. Then the Cloud stuff on top is for groups who need more gas. Cloudflare is generously supporting our upcoming free tier.
- wilted-iris 3y agoI remember reading that sccache requires builds to be done in the same absolute directory by each user to share cache. Do you know if that's correct?
- sgammon 3y agoStill the case, yes, but stay tuned as we’re working to fix that ;)