14 ms·

Great post, but: > How Do We Get It? > I’m afraid to find that out, you’ll need to wait until the next instalment. I hope to get it complete in a few weeks (I

Great post, but: > How Do We Get It? > I’m afraid to find that out, you’ll need to wait until the next instalment. I hope to get it complete in a few weeks (I was hoping to have a PoC available with the next instalment but that’s just a silly goal at this point). > If you want a taste though, the general idea is that we can resolve most of the issues I’ve listed and gain most of the properties we want by creating our own format that is built on top of the OCI content-addressable store, and is a Merkle tree with content-defined chunking of file contents. The basic idea is very similar to backup tools like restic or borgbackup. Since OCI has smart pointers, we can define a few new media-types, and then our new format would be completely transparent to OCI image tools (as opposed to opaque tar archives). > But you’ll learn all about that next time. Thanks for reading, and happy hacking! Any follow-up on this 3 year old post? This appears to be the final entry on cyphar's blog. Did this ever happen? I've done some hunting but can't find any stated plans for OCIv2 from Cyphar. There was a hack-session in march 2020 on this topic: https://hackmd.io/@cyphar/ociv2-brainstorm https://hackmd.io/@cyphar/ociv2-brainstorm . Really fascinating document. It has a section for each problem, & does a great job citing & discussing prior work in the area. Was a delight to stumble into this. But still seems very formative. It felt like cyphar was already onto a plan, well before this date. I don't see any evidence of v2 forming in the official repo, https://github.com/opencontainers/image-spec/tree/main/specs-go https://github.com/opencontainers/image-spec/tree/main/specs... . Cyphar does still seem active in the project. Seems there was a come-together call on the mailing list last June, calling for trying to extend-their-way-forwards rather than do a big revamp/rewrite: https://groups.google.com/a/opencontainers.org/g/dev/c/6DGtH48fGOg/m/1kUGGhRPBAAJ https://groups.google.com/a/opencontainers.org/g/dev/c/6DGtH...

cyphar 5y ago
Hello. I was surprised to see someone post this so long after I wrote it. We had several design discussions about this at the time but we struggled to come to an agreement about how we might migrate and various other issues. Tycho and I were the only people pushing for this and it wasn't clear how much other interest there is -- stargz is being pushed as a (pretty clever) attempt to hack around the tar issue by continuing to use tar but structuring it such that you can do random-access (among other things). I haven't had much time for blogging recently. I did have a draft part 2 written but I decided it might make more sense to have some consensus on where we might go before publishing it -- at this point though, neither have happened so maybe I should've published the blog post regardless (though in fairness I was still writing prototypes so publishing something might've been a bit premature). The general thesis would've been that we can basically design everything in a similar way to restic (content-defined-chunking and a snapshot-style system where the granularity of the filesystem data is much smaller than the layer level -- allowing for massively increased deduplication and eliminating the vast majority of issues with the layer-based model we have now). However after looking into it more, there are several other things we should rethink (device inodes for instance -- which would require rethinking how systems like Kubernetes express their requirements from an image). Also there's a fair argument that we should move compression entirely to distribution (and use decompressed data in the image format), but such a change would possibly require even more discussions with them to make sure that wouldn't be too burdensome. Also feel free to call me Aleksa lol.

cyphar 5y ago

Hello. I was surprised to see someone post this so long after I wrote it. We had several design discussions about this at the time but we struggled to come to an agreement about how we might migrate and various other issues. Tycho and I were the only people pushing for this and it wasn't clear how much other interest there is -- stargz is being pushed as a (pretty clever) attempt to hack around the tar issue by continuing to use tar but structuring it such that you can do random-access (among other things). I haven't had much time for blogging recently. I did have a draft part 2 written but I decided it might make more sense to have some consensus on where we might go before publishing it -- at this point though, neither have happened so maybe I should've published the blog post regardless (though in fairness I was still writing prototypes so publishing something might've been a bit premature). The general thesis would've been that we can basically design everything in a similar way to restic (content-defined-chunking and a snapshot-style system where the granularity of the filesystem data is much smaller than the layer level -- allowing for massively increased deduplication and eliminating the vast majority of issues with the layer-based model we have now). However after looking into it more, there are several other things we should rethink (device inodes for instance -- which would require rethinking how systems like Kubernetes express their requirements from an image). Also there's a fair argument that we should move compression entirely to distribution (and use decompressed data in the image format), but such a change would possibly require even more discussions with them to make sure that wouldn't be too burdensome. Also feel free to call me Aleksa lol.