3 ms·
Content-defined chunking added to Bazel
- londons_explore 5mo agoDoesn't this mean that malicious inputs can deliberately cause super tiny or super huge chunks?
- a_t48 5mo agoThis is something I'm very interested in implementing for Docker builds. I've tested out CDC for the final image outputs, it results in smaller outputs but requires tuning between saved space versus request count when pulling. For build cache it might be even more advantageous.
- stabbles 5mo agoIsn't that rather difficult given the `.tar.gz` layers?
- tracnar 5mo agoIt also supports .tar but that's probably not very commonly used.
- auscompgeek 5mo agoIn theory eStargz layers should be amenable to CDC.
- a_t48 5mo agoIt feels that way, but eStargz is still only addressable as a single layer, or range of one.
- a_t48 5mo agoI have a custom pull client/registry/builder that uses a different format, but can output standard OCI if needed.
- dotwaffle 5mo agoI have done extensive research on CDC and it almost never works out because most utilities don't create compressed archives in an "rsyncable" (rsync does CDC) format, I actually saved a lot of storage using restic when I switched my backups of certain things so that files were stored in archives uncompressed, and sorted in a stable order. I know syncthing eventually removed CDC and just went with constant-size block sizes. Bazel, on the other hand, is completely in control of this, and it makes perfect sense to do this at that point -- and it seems to be a relatively efficient implementation too, really nice to see!