16 ms·

Tarballs, the ultimate container image format

342 points by severus_snape 8y ago

AdmiralAsshat 8y ago
Do tarballs still have that unfixed/unfixable bug where the extracted files will have the permissions of the person who untarr'd the file?
- delinka 8y ago
  I think you mean owner rather than permissions. In most cases, you want to maintain permissions/file mode (read/write/execute) but not the original owner.
  - AdmiralAsshat 8y ago
    Except it doesn't do either. I've had files that had 666 user:group permissions/owner that I tar into a backup file, then untar, only to find that the file is now 664 with me:me ownership. It's brought production to a halt on more than one occasion if I try to "restore" from a backup by extracting the files and moving into production without manually fixing them first.
    - noja 8y ago
      What was your umask?
    - delinka 8y ago
      Your umask affects mode during extraction. You can pass -p to tar asking tar to attempt to restore exactly the modes in the archive. If you extract as root, it'll preserve the owner and group. Otherwise, the default is to assign the owner and group of the user running tar.
    - dozzie 8y ago
      > I've had files that had 666 user:group permissions/owner that I tar into a backup file, then untar, only to find that the file is now 664 with me:me ownership. It was PEBKAC, not tar's fault (GNU tar, anyway). Tar does store the original owner and permissions. But the ownership of the unpacked files -- do you really expect your process to set ownership of the files to another user? The permissions would also be restored to 666 if you ran the tar as root; there are several options whose defaults depend on whether EUID is 0 or not.
- master-litty 8y ago
  That seems sensible to me, what else would you expect?
  - AdmiralAsshat 8y ago
    I expect that they should preserve the ownership and permissions of the original file if I tell it to.
    - rakoo 8y ago
      How can it have the same owner if it's a different machine, and users aren't the same ?
      - spookthesunset 8y ago
        Do tarballs store the user/group names as strings or do they store the uid/gid instead? It is one of the goofy things about Unix systems is most tools speak uid/gid and woah is you if two machines on the network have “bob” only as different uid’s. Not entirely sure if windows has the same problem as to be honest if you use active directory most of that stuff is auto-magic. My hunch is that going with the ID vs. the “friendly name” has a bunch of trade offs and whichever you pick will come with serious drawbacks.
        saulrh 8y ago
        I feel you're on the right track with that hunch - Zooko's triangle should apply in some way.
        justincormack 8y ago
        They can do either - traditional tar formats have uid as a number, but the newer pax format has both numeric and named values.
        Hello71 8y ago
        in fact, names are the default. "--numeric-owner" must be passed to use numeric values.
- oconnor663 8y ago
  Or where the paths in the tarball can start with `..`?
- vinceguidry 8y ago
  You can use the --same-owner flag and extract the tarball as root in order to preserve ownership. The -p flag ensures that the permissions umask will match the archive's as well.
- tannhaeuser 8y ago
  It's a feature: you must be running tar as root or equivalently to restore to uids/gids other than the effective process uid. Otherwise you could happily overwrite any host system file including parts of the O/S. It's a restriction shared by all archivers.
- Grue3 8y ago
  I like the amazing "feature" where the act of extracting a tar file into a directory can change permissions on this directory. You have to pass --no-overwrite-dir flag to disable this.
- cyphar 8y ago
  That's a detail of the extraction tool. In umoci (which extracts tar archives as part of an OCI image)[1] you can remap the users or even extract as yourself and then add an xattr which represents the original owner in the archive (which is then read back when creating a new tar archive from the delta of the rootfs). [1]: https://github.com/openSUSE/umoci https://github.com/openSUSE/umoci
kuwze 8y ago
Does anyone know how this would apply, for example, to sharing a Guile 2.2 application with Debian/Red Hat based distributions? I want to use Guile 2.2 for development, but I am worried because it was only recently was released for major distros (at least with Ubuntu I know it was released with 18.04) and it doesn't seem to support the creation of executables.
- sitkack 8y ago
  See this older discussion on statically linking guile [0], one should be able to bake your source into a C program that statically links Guile 2.2 to create a self contained executable. If that is too cumbersome, I would use a container. [0] https://lists.gnu.org/archive/html/bug-guile/2013-03/msg00002.html https://lists.gnu.org/archive/html/bug-guile/2013-03/msg0000...
RX14 8y ago
I really love the work the guix folk are doing. I'd love to run guixsd on my laptop if it was easy and supported to run plain upstream linux instead of linux-libre. It just seems like such a lovely easy to use project from the little time I've spent playing with it, it's actually a small shame they're part of the "unsexy" GNU project and subject to GNU politics.
- jolmg 8y ago
  What GNU politics are you referring to that makes you reconsider using guixsd? EDIT: Also what's unsexy about GNU? I'm really curious.
  - jcoffland 8y ago
    It's not currently cool to like Richard Stallman because he has opinions that run contrary to Silicon Valley.
    - ronsor 8y ago
      Stallman is generally not a very... tactful person.
      - seba_dos1 8y ago
        That's right, it might even hurt his mission, but it doesn't make him less right.
        deleted 8y ago
        [deleted]
        astrodust 8y ago
        Was he right about abortion jokes? David Bowie made predictions far more profound than Stallman, and they came from a place of genuine concern, not tin-foil hattery of the GNU variety: https://www.theverge.com/2016/1/11/10753158/david-bowie-internet-future-interview https://www.theverge.com/2016/1/11/10753158/david-bowie-inte... I'd rather have people that cared and were on the right path, picking the right battles, than assholes who are technically correct but their observations are ultimately irrelevant to the larger fight.
stuaxo 8y ago
Please, can we move to an archive format that isn't so sprawlingly massive ?
- spookthesunset 8y ago
  What do you mean by massive?
- cpburns2009 8y ago
  Are you complaining about the complexity of file format itself? My understanding is it's pretty simple: a linked list of headers with the contents of each file after each header. Or are you complaining that it doesn't do compression itself like ZIPs do?
  - GrayShade 8y ago
    One think I dislike about tarballs is the lack of random access support.
    - cpuguy83 8y ago
      You can build random access around tarballs, just need to index the header data. Shameless plug, this is what https://github.com/cpuguy83/tarfs https://github.com/cpuguy83/tarfs does. Granted, you do have to traverse the entire tarball.
      - masklinn 8y ago
        > You can build random access around tarballs, just need to index the header data. > Granted, you do have to traverse the entire tarball. So you can't randomly access a tarball, you can cache the linear access you've already done.
        Vendan 8y ago
        You can read all the headers without reading the whole file by just seeking over the file data....
geofft 8y ago
I realize the title is just a hook for the (very cool!) work in the article, but a couple things that tarballs don't/can't specify that Docker containers can: - environment variables like locales. If your software expects to run with English sorting rules and UTF-8 character decoding, it shouldn't run with ASCII-value sorting and reject input bytes over 127. - Entrypoints. If your application expects all commands to run within a wrapper, you can't enforce that from a tarball. You can make conventions for both of these like "if /etc/default/locales exists, parse it for environment variables" and "if /entrypoint is executable, prepend it to all command lines", but then you have a convention on top of tarballs. (Which, to be fair, might be easier than OCI—I have no particular love for the OCI format—but the problem is harder than just "here are a bunch of files.")
- catern 8y ago
  It's not necessarily a good thing for the container to be able to specify locale. Locale should be picked up from the surrounding system; it's just that unfortunately the surrounding system is usually not configured correctly. And entrypoints/wrappers are definitely possible from a tarball. Just wrap the executables in bin/, replacing them with shell script (or whatever) wrappers pointing to the real executables. That's what Nix/Guix do for languages like Python which require dependencies to be provided by environment variables (as they don't have a way to "close over" the locations of their dependencies).
  - oconnore 8y ago
    > Locale should be picked up from the surrounding system; it's just that unfortunately the surrounding system is usually not configured correctly. And around and around we go
- sleepybrett 8y ago
  Also docker containers are just tarballs of tarballs (one per layer)
tannhaeuser 8y ago
That article made me warm up to guix and its practical side. Are guix app bundles just bare tar archives with /usr/local prefix semantics or do they need special metadata files? How are compiled binaries with hardcoded and/or autoconf'd prefixes handled for relocation (I guess using Linux namespaces somehow)?
- rekado 8y ago
  In Guix every package ends up in its own directory, which may have references to other packages in /gnu/store. An application bundle is really just a package closure, i.e. the directory for the package and all directories it references, recursively. One way to bundle up things is with `tar` (the default of `guix pack`), but Guix also supports other bundling targets, such as Docker. No special metadata files are required. Relocation currently requires a little C wrapper, which uses Linux namespaces, as the blog post indicates. If you want something more advanced, such as a bundle that includes an init and services, it's best to use `guix system`, which builds VM images among others.
chx 8y ago
For relocatable ELF binaries, there's also https://github.com/intoli/exodus https://github.com/intoli/exodus
- foob 8y ago
  The packages that Exodus produces are actually quite similar to those introduced in this announcement. Both tools generate simple tarballs that can be extracted anywhere to relocate programs along with their dependencies, and both tools bootstrap the program execution using small statically compiled launchers written in C. They contrast guix pack against Snap, Flatpak, and Docker, but Exodus would probably make a more apt comparison in many ways.
- civodul 8y ago
  Interesting! The trick that Exodus uses (invoking ld-linux.so directly) is very smart. Perhaps an option to add to 'guix pack' in the future. :-)
digi_owl 8y ago
A quick FYI, Gobolinux operates much the same way. 1. Binary packages are simply compressed archives (tarballs) of the relevant branch in the /Programs tree. 2. branches do not have to actually live inside the /Programs tree. There are tools available to move the branches in and out of /Programs. All this because Gobolinux leverages symbolic links as much as possible.
- matthewbauer 8y ago
  Gobolinux sort of does this. The main difference is GoboLinux uses “version numbers” while Nix & Guix use hashes. It makes a lot of difference for more complicated stuff.
  - digi_owl 8y ago
    True. I suspect there are ways to introduce hashes to Gobo, if one were so inclined. But so far nobody has.
- TylerE 8y ago
  Nitpick: A vanilla tarball is a concatenation, not a compression.
justinsaccount 8y ago
Articles like this are pointless. I get that guix and nix are neat, and I think that every single time something about one of them is posted, but I don't have the slightest clue how to use either one of them. Do you want to convince people that something like guix is better than docker? Then take something that is currently distributed using docker and actually show how the guix approach is simpler. i.e. I have a random app I recently worked on where the dockerfile was something like FROM python:2.7 WORKDIR /app ADD requirements.txt /app RUN pip install -r requirements.txt ADD . /app RUN groupadd -r notifier && useradd --no-log-init -r -g notifier notifier USER notifier EXPOSE 8080/tcp CMD ./notify.py How do I actually take a random application like that and build a guix package of it? Another project I work on is built on top of zeromq, and it would be great to use something like guix to define all the libsodium+zeromq+czmq+zyre dependancies and be able to spit out an 'ultimate container image' of all of that, but all this post shows me how to do is install an existing guile package.
- t0nt0n 8y ago
  Packaged zyre and czmq. I'll send the patch for their inclusion, but until then here is the code: https://notabug.org/thomassgn/guixsd-configuration/src/master/modules/ton-tull.scm#L210 https://notabug.org/thomassgn/guixsd-configuration/src/maste...
- t0nt0n 8y ago
  With Guix you get full introspection of your entire package dependency graph, you can check and manipulate every aspect - and it is still simple and easy to work with. With GuixSD you get this same introspection and overview, but of your entire system. creating a container, vm or even a docker image is a simple '$ guix system <container|vm> config.scm' away. And your config.scm is as complex as you like it to. The simplest way would be to package the app for guix and you could just run '$ guix environment <name-of-package>' and you would be dropped into an environment with all your dependencies and whatever else the application requires in your path ready for hacking, get your sources and editor and start working. If you need a vm or similar though I'd translate your example above into a system config where: - packages include python-2.7 and whatever is in requirements.txt (this may mean you have to package a few things, but again this is usually super easy) - users and groups are added to the config, as they always are, no extra step necessary. - exposing ports and networking is available as options for qemu script guix produces to launch the vm. - CMD ./notify.py: create a "simple" service that can be autostarted by the system on boot. - filesystem access is also handled by arguments to the qemu script. As always though there are several paths to Rome, and these are just two of them. Zeromq and libsodium are already packaged on guix, czmq and zyre looks like they would be simple to package, guix is really quite simple to work with, which I think is the reason so many of the users and devs are running it as our daily drivers, even though it is strictly beta (0.14. I think is the last release). And pointless, come on - what does that even mean? Does it mean you don't value them? I was quite happy to read about a neat new thing I can use my favorite tool for.
matthewbauer 8y ago
Nix has a very similar tool called nix-bundle[1]. [1]: https://github.com/matthewbauer/nix-bundle https://github.com/matthewbauer/nix-bundle
nerpderp83 8y ago
Tarballs don't have a TOC and can't easily index into individual entities. One could create a utility to make tarballs with a TOC and the ability to index while still remaining compatible with tar and gzip. Pigz is one step in the direction.
- matthewbauer 8y ago
  I think AppImage does this with SquashFS images currently.
- tejtm 8y ago
  to list what is in a tar ball `tar -vtf tarball.tar` to extract a particular entity 'tar -vxf tarball.tar path_in tarball_to_entity` edit: good points on it not being efficient for large archives, just demonstrating it is possible.
  - discreditable 8y ago
    The problem is that to know what files are in the tarball you have to read the whole thing. If the archive is large that's a lot of reading just to get a file list.
  - nerpderp83 8y ago
    A tar is a linked list of file paths and contents, it cannot be indexed to a particular file. A compressed tar has to first be decompressed and then the chain of links traversed. Accessing a file in compressed tar is o(n) with where the file is placed within the compressed tar stream. It isn't that it is possible, it is that is horribly inefficient. Zips on other hand unify storage and compression such that one has random access to particular file, hence most modern file formats are zips with xml or json inside.
AnIdiotOnTheNet 8y ago
Reinventing Application Bundles only 30 years after NeXTStep, poorly.
- matthewbauer 8y ago
  Why poorly? I don’t see anything worse about this.
  - AnIdiotOnTheNet 8y ago
    Really? Seems like an awful lot of tooling for what is essentially "Put binary and dependencies in folder. Move folder around at will" in sane environments.
    - pxc 8y ago
      The tooling already existed because it's part of a stack that goes from build tool to package manager to operating system configuration manager, with all kinds of features for developers floating around along the periphery. It handles all of these things uniformly, reliably, reproducibly, and in a way that deduplicates shared dependencies. This article is just showcasing a relatively small bit of tooling on top all that which makes it possible to reuse that work to produce containers out of the very same stuff, in a whole range of formats. `guix pack` and `nix-bundle` are illustrations of how a novel solution (functional package management) to the very problem to which app bundling constitutes utter capitulation (dependency management) can not only retain the virtues the app bundle approach throws away in the hopes of making deployment simple, but even match it in ease of deployment when _none_ of the infrastructure of the package management system is expected to be present on the deployment target. From where I stand, that's damn impressive. All of this was achieved without the kind of ‘standardization from above’ that Apple gets to do on its platform. It's true that app bundling could have been a lot simpler if the Linux community lived in a locked box at the mercy of a Vampire King bearing the power to upgrade users' kernels in the dead of night without bothering to ask them, who preempted any diversity or choice in operating system components with a uniform common runtime, and gleefully ripped unseemly APIs out from under developers with every OS release. But instead— thank God!— we have such a wide range of environments under the name ‘Linux’ that I'm ready to agree with you and call it insane. Yet here we see that hackers made it work anyway, without bossing anyone around or compromising on the strengths of proper package management. And that's fucking awesome.
theamk 8y ago
I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad: Either you extract it from scratch every time you run an app, taking a long time penalty... ... or you extract once to cache, and assume that nothing changes the cache. This is pretty bad from both operational and security perspective: - backups have to walk through tens of thousands of files, thus becoming much slower - a damaged disk or a malicious actor can change one file in the cache, making damage which is very hard to detect. There are plenty of mountable container formats -- ISO, squashfs, even zip files -- which all provide much faster initial access, and much better security/reliability guarantees, especially with things like dm-verity.
- 2ion 8y ago
  Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.
  - textmode 8y ago
    "... there are some metadata extensions that allow this)." Where to find these extensions? Are they portable between Linux and BSD? The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files. Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives? Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.
    - Hello71 8y ago
      afaik, only pixz does this; they store the index in xz.
paulfitz 8y ago
How about sqlar as a container format? https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html A regular sqlite database file, with anything you like in it. Mountable as a file system with sqlarfs. Written by the sqlite guy.
- infogulch 8y ago
  Interesting I didn't know this existed. Is there a way to layer sqlar like docker images? (Besides just tarring them up I guess.) I wonder if this could be implemented with the WAL/journal system. Make each layer immutably append to the previous layers to make restarting at any layer trivial. I'm not sure if there's such a way to hook into the journal directly like that though.
  - zaarn 8y ago
    Should be doable with overlayfs (or similar) or alternatively some extensions to sqlar. sqlar is after all only a table definition, if you don't need FUSE access or are willing to write your own, SQLite3 can go a long way of providing arbitrary neat functionality.
cyphar 8y ago
This is remarkably off-beat for the GNU project. Tar files are far from the most ideal tool for container images because they are sequential archives and thus extraction cannot be done using any parallelism (without adding an index and being in a seekable medium, see the rest of this comment). I should really write a blog post about this. Another problem is that there is no way to just get the latest entry in a multi-layered image without scanning every layer sequentially (this can be made faster with a top-level index but I don't think anyone has implemented this yet -- I am working on it for umoci but nobody else will probably use it even if I implement it). This means you have to extract all of the archives. Yet another problem is that if you have a layer which just includes a metadata change (like the mode of a file), then you have to include a full copy of the file into the archive (same goes for a single bit change in the file contents -- even if the file is 10GB in size). This balloons up the archive size needlessly due to restrictions in the tar format (no way of representing a metadata entry in a standard-complying way), and increases the effect of the previous problem I mentioned. And all of the above ignores the fact that tar archives are not actually standardised (you have at least 3 "extension" formats -- GNU, PAX, and libarchive), and different implementations produce vastly different archive outputs and structures (causing problems with making them content-addressable). To be fair, this is a fairly solved problem at this point (though sparse archives are sort of unsolved) but it requires storing the metadata of the archive structure in addition to the archive. Despite all of this Docker and OCI (and AppC) all use tar archives, so this isn't really a revolutionary blog post (it's sort of what everyone does, but nobody is really happy about it). In the OCI we are working on switching to a format that solves the above problems by having a history for each file (so the layering is implemented in the archiving layer rather than on top) and having an index where we store all of the files in the content-addressable storage layer. I believe we also will implement content-based-chunking for deduplication to allow us to handle minor changes in files without blowing up image sizes. These are things you cannot do in tar archives and are fundamentally limited. I appreciate that tar is a very good tool (and we shouldn't reinvent good tools), but not wanting to improve the state-of-the-art over literal tape archives seems a bit too nostalgic to me. Especially when there are clear problems with the current format, with obvious ways of improving them.
- Rapzid 8y ago

342 points by severus_snape 8y ago

Do tarballs still have that unfixed/unfixable bug where the extracted files will have the permissions of the person who untarr'd the file?

delinka 8y ago
I think you mean owner rather than permissions. In most cases, you want to maintain permissions/file mode (read/write/execute) but not the original owner.
- AdmiralAsshat 8y ago
  Except it doesn't do either. I've had files that had 666 user:group permissions/owner that I tar into a backup file, then untar, only to find that the file is now 664 with me:me ownership. It's brought production to a halt on more than one occasion if I try to "restore" from a backup by extracting the files and moving into production without manually fixing them first.
  - noja 8y ago
    What was your umask?
  - delinka 8y ago
    Your umask affects mode during extraction. You can pass -p to tar asking tar to attempt to restore exactly the modes in the archive. If you extract as root, it'll preserve the owner and group. Otherwise, the default is to assign the owner and group of the user running tar.
  - dozzie 8y ago
    > I've had files that had 666 user:group permissions/owner that I tar into a backup file, then untar, only to find that the file is now 664 with me:me ownership. It was PEBKAC, not tar's fault (GNU tar, anyway). Tar does store the original owner and permissions. But the ownership of the unpacked files -- do you really expect your process to set ownership of the files to another user? The permissions would also be restored to 666 if you ran the tar as root; there are several options whose defaults depend on whether EUID is 0 or not.
master-litty 8y ago
That seems sensible to me, what else would you expect?
- AdmiralAsshat 8y ago
  I expect that they should preserve the ownership and permissions of the original file if I tell it to.
  - rakoo 8y ago
    How can it have the same owner if it's a different machine, and users aren't the same ?
    - spookthesunset 8y ago
      Do tarballs store the user/group names as strings or do they store the uid/gid instead? It is one of the goofy things about Unix systems is most tools speak uid/gid and woah is you if two machines on the network have “bob” only as different uid’s. Not entirely sure if windows has the same problem as to be honest if you use active directory most of that stuff is auto-magic. My hunch is that going with the ID vs. the “friendly name” has a bunch of trade offs and whichever you pick will come with serious drawbacks.
      - saulrh 8y ago
        I feel you're on the right track with that hunch - Zooko's triangle should apply in some way.
      - justincormack 8y ago
        They can do either - traditional tar formats have uid as a number, but the newer pax format has both numeric and named values.
        Hello71 8y ago
        in fact, names are the default. "--numeric-owner" must be passed to use numeric values.
oconnor663 8y ago
Or where the paths in the tarball can start with `..`?
vinceguidry 8y ago
You can use the --same-owner flag and extract the tarball as root in order to preserve ownership. The -p flag ensures that the permissions umask will match the archive's as well.
tannhaeuser 8y ago
It's a feature: you must be running tar as root or equivalently to restore to uids/gids other than the effective process uid. Otherwise you could happily overwrite any host system file including parts of the O/S. It's a restriction shared by all archivers.
Grue3 8y ago
I like the amazing "feature" where the act of extracting a tar file into a directory can change permissions on this directory. You have to pass --no-overwrite-dir flag to disable this.
cyphar 8y ago
That's a detail of the extraction tool. In umoci (which extracts tar archives as part of an OCI image)[1] you can remap the users or even extract as yourself and then add an xattr which represents the original owner in the archive (which is then read back when creating a new tar archive from the delta of the rootfs). [1]: https://github.com/openSUSE/umoci https://github.com/openSUSE/umoci

kuwze 8y ago

Does anyone know how this would apply, for example, to sharing a Guile 2.2 application with Debian/Red Hat based distributions? I want to use Guile 2.2 for development, but I am worried because it was only recently was released for major distros (at least with Ubuntu I know it was released with 18.04) and it doesn't seem to support the creation of executables.

sitkack 8y ago
See this older discussion on statically linking guile [0], one should be able to bake your source into a C program that statically links Guile 2.2 to create a self contained executable. If that is too cumbersome, I would use a container. [0] https://lists.gnu.org/archive/html/bug-guile/2013-03/msg00002.html https://lists.gnu.org/archive/html/bug-guile/2013-03/msg0000...

RX14 8y ago

I really love the work the guix folk are doing. I'd love to run guixsd on my laptop if it was easy and supported to run plain upstream linux instead of linux-libre. It just seems like such a lovely easy to use project from the little time I've spent playing with it, it's actually a small shame they're part of the "unsexy" GNU project and subject to GNU politics.

jolmg 8y ago
What GNU politics are you referring to that makes you reconsider using guixsd? EDIT: Also what's unsexy about GNU? I'm really curious.
- jcoffland 8y ago
  It's not currently cool to like Richard Stallman because he has opinions that run contrary to Silicon Valley.
  - ronsor 8y ago
    Stallman is generally not a very... tactful person.
    - seba_dos1 8y ago
      That's right, it might even hurt his mission, but it doesn't make him less right.
      - deleted 8y ago
        [deleted]
      - astrodust 8y ago
        Was he right about abortion jokes? David Bowie made predictions far more profound than Stallman, and they came from a place of genuine concern, not tin-foil hattery of the GNU variety: https://www.theverge.com/2016/1/11/10753158/david-bowie-internet-future-interview https://www.theverge.com/2016/1/11/10753158/david-bowie-inte... I'd rather have people that cared and were on the right path, picking the right battles, than assholes who are technically correct but their observations are ultimately irrelevant to the larger fight.

stuaxo 8y ago

Please, can we move to an archive format that isn't so sprawlingly massive ?

spookthesunset 8y ago
What do you mean by massive?
cpburns2009 8y ago
Are you complaining about the complexity of file format itself? My understanding is it's pretty simple: a linked list of headers with the contents of each file after each header. Or are you complaining that it doesn't do compression itself like ZIPs do?
- GrayShade 8y ago
  One think I dislike about tarballs is the lack of random access support.
  - cpuguy83 8y ago
    You can build random access around tarballs, just need to index the header data. Shameless plug, this is what https://github.com/cpuguy83/tarfs https://github.com/cpuguy83/tarfs does. Granted, you do have to traverse the entire tarball.
    - masklinn 8y ago
      > You can build random access around tarballs, just need to index the header data. > Granted, you do have to traverse the entire tarball. So you can't randomly access a tarball, you can cache the linear access you've already done.
      - Vendan 8y ago
        You can read all the headers without reading the whole file by just seeking over the file data....

geofft 8y ago

I realize the title is just a hook for the (very cool!) work in the article, but a couple things that tarballs don't/can't specify that Docker containers can: - environment variables like locales. If your software expects to run with English sorting rules and UTF-8 character decoding, it shouldn't run with ASCII-value sorting and reject input bytes over 127. - Entrypoints. If your application expects all commands to run within a wrapper, you can't enforce that from a tarball. You can make conventions for both of these like "if /etc/default/locales exists, parse it for environment variables" and "if /entrypoint is executable, prepend it to all command lines", but then you have a convention on top of tarballs. (Which, to be fair, might be easier than OCI—I have no particular love for the OCI format—but the problem is harder than just "here are a bunch of files.")

catern 8y ago
It's not necessarily a good thing for the container to be able to specify locale. Locale should be picked up from the surrounding system; it's just that unfortunately the surrounding system is usually not configured correctly. And entrypoints/wrappers are definitely possible from a tarball. Just wrap the executables in bin/, replacing them with shell script (or whatever) wrappers pointing to the real executables. That's what Nix/Guix do for languages like Python which require dependencies to be provided by environment variables (as they don't have a way to "close over" the locations of their dependencies).
- oconnore 8y ago
  > Locale should be picked up from the surrounding system; it's just that unfortunately the surrounding system is usually not configured correctly. And around and around we go
sleepybrett 8y ago
Also docker containers are just tarballs of tarballs (one per layer)

tannhaeuser 8y ago

That article made me warm up to guix and its practical side. Are guix app bundles just bare tar archives with /usr/local prefix semantics or do they need special metadata files? How are compiled binaries with hardcoded and/or autoconf'd prefixes handled for relocation (I guess using Linux namespaces somehow)?

rekado 8y ago
In Guix every package ends up in its own directory, which may have references to other packages in /gnu/store. An application bundle is really just a package closure, i.e. the directory for the package and all directories it references, recursively. One way to bundle up things is with `tar` (the default of `guix pack`), but Guix also supports other bundling targets, such as Docker. No special metadata files are required. Relocation currently requires a little C wrapper, which uses Linux namespaces, as the blog post indicates. If you want something more advanced, such as a bundle that includes an init and services, it's best to use `guix system`, which builds VM images among others.

chx 8y ago

For relocatable ELF binaries, there's also https://github.com/intoli/exodus https://github.com/intoli/exodus

foob 8y ago
The packages that Exodus produces are actually quite similar to those introduced in this announcement. Both tools generate simple tarballs that can be extracted anywhere to relocate programs along with their dependencies, and both tools bootstrap the program execution using small statically compiled launchers written in C. They contrast guix pack against Snap, Flatpak, and Docker, but Exodus would probably make a more apt comparison in many ways.
civodul 8y ago
Interesting! The trick that Exodus uses (invoking ld-linux.so directly) is very smart. Perhaps an option to add to 'guix pack' in the future. :-)

digi_owl 8y ago

A quick FYI, Gobolinux operates much the same way. 1. Binary packages are simply compressed archives (tarballs) of the relevant branch in the /Programs tree. 2. branches do not have to actually live inside the /Programs tree. There are tools available to move the branches in and out of /Programs. All this because Gobolinux leverages symbolic links as much as possible.

matthewbauer 8y ago
Gobolinux sort of does this. The main difference is GoboLinux uses “version numbers” while Nix & Guix use hashes. It makes a lot of difference for more complicated stuff.
- digi_owl 8y ago
  True. I suspect there are ways to introduce hashes to Gobo, if one were so inclined. But so far nobody has.
TylerE 8y ago
Nitpick: A vanilla tarball is a concatenation, not a compression.

justinsaccount 8y ago

Articles like this are pointless. I get that guix and nix are neat, and I think that every single time something about one of them is posted, but I don't have the slightest clue how to use either one of them. Do you want to convince people that something like guix is better than docker? Then take something that is currently distributed using docker and actually show how the guix approach is simpler. i.e. I have a random app I recently worked on where the dockerfile was something like FROM python:2.7 WORKDIR /app ADD requirements.txt /app RUN pip install -r requirements.txt ADD . /app RUN groupadd -r notifier && useradd --no-log-init -r -g notifier notifier USER notifier EXPOSE 8080/tcp CMD ./notify.py How do I actually take a random application like that and build a guix package of it? Another project I work on is built on top of zeromq, and it would be great to use something like guix to define all the libsodium+zeromq+czmq+zyre dependancies and be able to spit out an 'ultimate container image' of all of that, but all this post shows me how to do is install an existing guile package.

t0nt0n 8y ago
Packaged zyre and czmq. I'll send the patch for their inclusion, but until then here is the code: https://notabug.org/thomassgn/guixsd-configuration/src/master/modules/ton-tull.scm#L210 https://notabug.org/thomassgn/guixsd-configuration/src/maste...
t0nt0n 8y ago
With Guix you get full introspection of your entire package dependency graph, you can check and manipulate every aspect - and it is still simple and easy to work with. With GuixSD you get this same introspection and overview, but of your entire system. creating a container, vm or even a docker image is a simple '$ guix system <container|vm> config.scm' away. And your config.scm is as complex as you like it to. The simplest way would be to package the app for guix and you could just run '$ guix environment <name-of-package>' and you would be dropped into an environment with all your dependencies and whatever else the application requires in your path ready for hacking, get your sources and editor and start working. If you need a vm or similar though I'd translate your example above into a system config where: - packages include python-2.7 and whatever is in requirements.txt (this may mean you have to package a few things, but again this is usually super easy) - users and groups are added to the config, as they always are, no extra step necessary. - exposing ports and networking is available as options for qemu script guix produces to launch the vm. - CMD ./notify.py: create a "simple" service that can be autostarted by the system on boot. - filesystem access is also handled by arguments to the qemu script. As always though there are several paths to Rome, and these are just two of them. Zeromq and libsodium are already packaged on guix, czmq and zyre looks like they would be simple to package, guix is really quite simple to work with, which I think is the reason so many of the users and devs are running it as our daily drivers, even though it is strictly beta (0.14. I think is the last release). And pointless, come on - what does that even mean? Does it mean you don't value them? I was quite happy to read about a neat new thing I can use my favorite tool for.

matthewbauer 8y ago

Nix has a very similar tool called nix-bundle[1]. [1]: https://github.com/matthewbauer/nix-bundle https://github.com/matthewbauer/nix-bundle

nerpderp83 8y ago

Tarballs don't have a TOC and can't easily index into individual entities. One could create a utility to make tarballs with a TOC and the ability to index while still remaining compatible with tar and gzip. Pigz is one step in the direction.

matthewbauer 8y ago
I think AppImage does this with SquashFS images currently.
tejtm 8y ago
to list what is in a tar ball `tar -vtf tarball.tar` to extract a particular entity 'tar -vxf tarball.tar path_in tarball_to_entity` edit: good points on it not being efficient for large archives, just demonstrating it is possible.
- discreditable 8y ago
  The problem is that to know what files are in the tarball you have to read the whole thing. If the archive is large that's a lot of reading just to get a file list.
- nerpderp83 8y ago
  A tar is a linked list of file paths and contents, it cannot be indexed to a particular file. A compressed tar has to first be decompressed and then the chain of links traversed. Accessing a file in compressed tar is o(n) with where the file is placed within the compressed tar stream. It isn't that it is possible, it is that is horribly inefficient. Zips on other hand unify storage and compression such that one has random access to particular file, hence most modern file formats are zips with xml or json inside.

AnIdiotOnTheNet 8y ago

Reinventing Application Bundles only 30 years after NeXTStep, poorly.

matthewbauer 8y ago
Why poorly? I don’t see anything worse about this.
- AnIdiotOnTheNet 8y ago
  Really? Seems like an awful lot of tooling for what is essentially "Put binary and dependencies in folder. Move folder around at will" in sane environments.
  - pxc 8y ago
    The tooling already existed because it's part of a stack that goes from build tool to package manager to operating system configuration manager, with all kinds of features for developers floating around along the periphery. It handles all of these things uniformly, reliably, reproducibly, and in a way that deduplicates shared dependencies. This article is just showcasing a relatively small bit of tooling on top all that which makes it possible to reuse that work to produce containers out of the very same stuff, in a whole range of formats. `guix pack` and `nix-bundle` are illustrations of how a novel solution (functional package management) to the very problem to which app bundling constitutes utter capitulation (dependency management) can not only retain the virtues the app bundle approach throws away in the hopes of making deployment simple, but even match it in ease of deployment when _none_ of the infrastructure of the package management system is expected to be present on the deployment target. From where I stand, that's damn impressive. All of this was achieved without the kind of ‘standardization from above’ that Apple gets to do on its platform. It's true that app bundling could have been a lot simpler if the Linux community lived in a locked box at the mercy of a Vampire King bearing the power to upgrade users' kernels in the dead of night without bothering to ask them, who preempted any diversity or choice in operating system components with a uniform common runtime, and gleefully ripped unseemly APIs out from under developers with every OS release. But instead— thank God!— we have such a wide range of environments under the name ‘Linux’ that I'm ready to agree with you and call it insane. Yet here we see that hackers made it work anyway, without bossing anyone around or compromising on the strengths of proper package management. And that's fucking awesome.

theamk 8y ago

I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad: Either you extract it from scratch every time you run an app, taking a long time penalty... ... or you extract once to cache, and assume that nothing changes the cache. This is pretty bad from both operational and security perspective: - backups have to walk through tens of thousands of files, thus becoming much slower - a damaged disk or a malicious actor can change one file in the cache, making damage which is very hard to detect. There are plenty of mountable container formats -- ISO, squashfs, even zip files -- which all provide much faster initial access, and much better security/reliability guarantees, especially with things like dm-verity.

2ion 8y ago
Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.
- textmode 8y ago
  "... there are some metadata extensions that allow this)." Where to find these extensions? Are they portable between Linux and BSD? The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files. Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives? Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.
  - Hello71 8y ago
    afaik, only pixz does this; they store the index in xz.

paulfitz 8y ago

How about sqlar as a container format? https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html A regular sqlite database file, with anything you like in it. Mountable as a file system with sqlarfs. Written by the sqlite guy.

infogulch 8y ago
Interesting I didn't know this existed. Is there a way to layer sqlar like docker images? (Besides just tarring them up I guess.) I wonder if this could be implemented with the WAL/journal system. Make each layer immutably append to the previous layers to make restarting at any layer trivial. I'm not sure if there's such a way to hook into the journal directly like that though.
- zaarn 8y ago
  Should be doable with overlayfs (or similar) or alternatively some extensions to sqlar. sqlar is after all only a table definition, if you don't need FUSE access or are willing to write your own, SQLite3 can go a long way of providing arbitrary neat functionality.

cyphar 8y ago

This is remarkably off-beat for the GNU project. Tar files are far from the most ideal tool for container images because they are sequential archives and thus extraction cannot be done using any parallelism (without adding an index and being in a seekable medium, see the rest of this comment). I should really write a blog post about this. Another problem is that there is no way to just get the latest entry in a multi-layered image without scanning every layer sequentially (this can be made faster with a top-level index but I don't think anyone has implemented this yet -- I am working on it for umoci but nobody else will probably use it even if I implement it). This means you have to extract all of the archives. Yet another problem is that if you have a layer which just includes a metadata change (like the mode of a file), then you have to include a full copy of the file into the archive (same goes for a single bit change in the file contents -- even if the file is 10GB in size). This balloons up the archive size needlessly due to restrictions in the tar format (no way of representing a metadata entry in a standard-complying way), and increases the effect of the previous problem I mentioned. And all of the above ignores the fact that tar archives are not actually standardised (you have at least 3 "extension" formats -- GNU, PAX, and libarchive), and different implementations produce vastly different archive outputs and structures (causing problems with making them content-addressable). To be fair, this is a fairly solved problem at this point (though sparse archives are sort of unsolved) but it requires storing the metadata of the archive structure in addition to the archive. Despite all of this Docker and OCI (and AppC) all use tar archives, so this isn't really a revolutionary blog post (it's sort of what everyone does, but nobody is really happy about it). In the OCI we are working on switching to a format that solves the above problems by having a history for each file (so the layering is implemented in the archiving layer rather than on top) and having an index where we store all of the files in the content-addressable storage layer. I believe we also will implement content-based-chunking for deduplication to allow us to handle minor changes in files without blowing up image sizes. These are things you cannot do in tar archives and are fundamentally limited. I appreciate that tar is a very good tool (and we shouldn't reinvent good tools), but not wanting to improve the state-of-the-art over literal tape archives seems a bit too nostalgic to me. Especially when there are clear problems with the current format, with obvious ways of improving them.

Rapzid 8y ago