4 ms·
This is remarkably off-beat for the GNU project. Tar files are far from the most ideal tool for container images because they are sequential archives and thus e
by cyphar 8y ago
This is remarkably off-beat for the GNU project. Tar files are far from the most ideal tool for container images because they are sequential archives and thus extraction cannot be done using any parallelism (without adding an index and being in a seekable medium, see the rest of this comment). I should really write a blog post about this.
Another problem is that there is no way to just get the latest entry in a multi-layered image without scanning every layer sequentially (this can be made faster with a top-level index but I don't think anyone has implemented this yet -- I am working on it for umoci but nobody else will probably use it even if I implement it). This means you have to extract all of the archives.
Yet another problem is that if you have a layer which just includes a metadata change (like the mode of a file), then you have to include a full copy of the file into the archive (same goes for a single bit change in the file contents -- even if the file is 10GB in size). This balloons up the archive size needlessly due to restrictions in the tar format (no way of representing a metadata entry in a standard-complying way), and increases the effect of the previous problem I mentioned.
And all of the above ignores the fact that tar archives are not actually standardised (you have at least 3 "extension" formats -- GNU, PAX, and libarchive), and different implementations produce vastly different archive outputs and structures (causing problems with making them content-addressable). To be fair, this is a fairly solved problem at this point (though sparse archives are sort of unsolved) but it requires storing the metadata of the archive structure in addition to the archive.
Despite all of this Docker and OCI (and AppC) all use tar archives, so this isn't really a revolutionary blog post (it's sort of what everyone does, but nobody is really happy about it). In the OCI we are working on switching to a format that solves the above problems by having a history for each file (so the layering is implemented in the archiving layer rather than on top) and having an index where we store all of the files in the content-addressable storage layer. I believe we also will implement content-based-chunking for deduplication to allow us to handle minor changes in files without blowing up image sizes. These are things you cannot do in tar archives and are fundamentally limited.
I appreciate that tar is a very good tool (and we shouldn't reinvent good tools), but not wanting to improve the state-of-the-art over literal tape archives seems a bit too nostalgic to me. Especially when there are clear problems with the current format, with obvious ways of improving them.
- Rapzid 8y agoIt sounds there is a lot of use cases that overlap with services provided by general file systems. I'd be curious to here your thoughts on that.
- cyphar 8y agoYou're right that general-purpose filesystems have solved quite a few of the indexing problems already, unfortunately there are a few things stopping general filesystems on a loopback device from being practical (or safe, or the best idea): * The container (file) for the filesystem must necessarily be larger than the metadata+data for the filesystem because filesystems really don't like almost-full disks. And unless I'm mistaken sparse files are not usable for loopback devices (so you can't hack your way out of it). * Most filesystems don't have a snapshot-style history so you would have to pick a specific filesystem from that list (otherwise you'd be forced to make CoW duplicates of the filesystem to create snapshots -- which is interestingly how Docker does layered storage with devicemapper) which has slightly similar problems to layered tar archives. * The kernel's filesystem parsers are not really considered to be safe against an adversary, from what I've been told by filesystem engineers. So mounting random loopback files with filesystems on them might end badly. * There is no way of looking at the archive using a userspace tool (without mounting), unless you re-implement the kernel parser for the filesystem. To be fair, this is true for any format, but filesystems are far more complicated and harder-to-parse than most other formats. * Having a single blob as your entire image history and so on will mean that you can no longer have content-addressable storage for your images without adding something like content-defined chunking on top (which is then another layer of storage on top of your underlying storage). * Using a Linux filesystem would mean you couldn't use the filesystem on different operating systems very easily. Even if it was compatible on whatever other filesystem you are using, userspace has no way of being sure there isn't a bug in either side's parser -- and what happens if one side changes the on-disk format. If the protocol is in userspace then it can be handled there. * Most filesystems don't let you remap users, so if you wanted to run a container in a user namespace you would need to either rewrite the filesystem structure or mount the filesystem and copy it to another filesystem. To be fair, tar archives require you to do the mapping on extraction which is a similar problem, but far less complicated. * Everyone would be opinionated about what filesystem to use, which means that you'd have to deal with every filesystem people throw at you, making it harder to be interoperable and adding choices where they aren't necessary. It should be up to the user what filesystem they use for storage, not the image distributor. Now, this hasn't stopped people from trying to use this. Singularity's internal format is a loopback file with a filesystem inside, and they have privileged suid binaries that mount it. And it does have genuine performance benefits, and if you don't want things like content-addressability then it can work for some usecases.