> This is mostly a good post, but he doesn’t mention one huge security disadvantage of containers: When application artifacts all include their own versions of

> This is mostly a good post, but he doesn’t mention one huge security disadvantage of containers: When application artifacts all include their own versions of dependencies locked at build time, it becomes very hard to update a component that has a security fix. I absolutely agree. So how do we fix this? Well, first we need to identify what's broken, and then we can talk about how to fix it. We could look at container images and see that, in the pursuit of reproducibility/determinism, they make it infeasible to make critical updates; from this we could infer that determinism is the root of our security woes, and that we have to choose between having a run-time environment that you can depend on, or being able to promptly make critical security updates. Fortunately, that a false dichotomy. Actually, it's even better than that: it turns out there's a third option that gives you all of the above: 1. The ability to quickly make security updates 2. A deterministic and serializable run-time environment 3. A deterministic and serializable build-time environment Note the addition of bullet point #3. In something like Docker, you can successfully build an image from a Dockerfile today, and yet it could fail to build tomorrow. You can build a Dockerfile on your machine, but it might not build successfully on my machine. What I am claiming is that there is another approach to containers that would additionally provide determinism when you're building the container's root file system, whereas e.g. Docker only gives you determinism in the sense that a previously built image can be passed around. So what system could provide all of the above? How could we make that work? (If you get bored with the details, feel free to skip down to the spoiler after TL;DR) We'd first need a deterministic build mechanism; broadly speaking, one in which we could build e.g. executables, fetch tarballs, construct file-systems, etc. To be deterministic, we our build system must have certain properties: * "normal" builds must be cut off from network access, whereas * A special "fixed output" build type could e.g. fetch tarballs, provided that e.g. a SHA-512 hash is given beforehand * Each build specification must specify what other build specs it depends on. Each build environment is constructed such that processes therein are only permitted to see/access files that are the resulting artifacts of those explicitly listed dependencies. Arbitrary read/write access to the host file-system is not permitted. * The resulting build artifacts go in their own immutable prefix, something like, let's say, /store/<hash>-<name>, where <hash> is a unique key derived by hashing the build specification, and <name> is a friendly name for the build artifact. To provide a concrete example, if you were to build `gcc`, you'd have something like /store/4r5kszyy0iirc5agfah45lvz7mnnsrb4-gcc-7.3.0/{bin,include,lib,lib64,libexec,share}, which would all contain the usual suspects. Note that when we hash the build specification, that spec refers to its dependencies, which in turn have their own unique hash, and so on. As a result of this design decision, we have: support for multiple versions of the same software; updates can't break existing executables (due to, let's say, breaking ABI changes in openssl or libc or whatever); and a final point that's important enough to split out as the following bullet point: * We can trivially construct deterministic environments: write a build spec that takes as an input all of the dependencies you wish to have in the environment, and (as the single resulting build artifact) produce a symlink tree that is the union of all the build artifacts of those aforementioned inputs. For example, you might want an environment with, I don't know, let's say gcc, python and ruby; you would end up with something like /store/m6n3y1rlz9rf06rj4spmvlkmv51angbm-container-env, which would contain bin/gcc, bin/python, bin/ruby, etc. Of course, having this environment wouldn't be much good unless you could actually run something inside of it, so that leads to the next point: * We can take an environment (as described in the previous point) and use it as the file system for a container: just do all of the usual container/namespace stuff, and e.g. symlink the env to /usr. A byproduct of this is that rollback/forward is easy: just bump a single symlink to the appropriate env path. * Due to our design, it's a trivial matter to serialize an environment and (re)distribute it. * Also important: the host system can build and retain all of the container environments in the host's /store. We could then bind-mount paths into the container as necessary. As a result, when two containers depend on precisely the same software, you only pay the price of a single copy. This is unlike Docker in that Docker requires that your image's layers directly inherit from each other; if you have, say for argument, an Ubuntu based image and a CentOS based image, it doesn't matter if there are components that are exactly the same in both - you have to pay the price of two copies. TL;DR -- TL;DR -- TL;DR --TL;DR We now have a system in which we can construct recipes to build container environments, and those recipes can be deterministically built, serialized, distributed, and then deserialized and used as a reproducible environment. Thanks to this, it's easy to tweak an existing recipe to bump up, say, openssl when a critical CVE is announced. We've beaten the typical image-based approach to containers (e.g. as used by Docker) in terms of security risk mitigation, disk foot print, developer time (from otherwise fighting with broken build specs / Dockerfiles), etc. Okay, so we don't actually have this system, we've just fantasized about one... ...unless something like this already exists! Indeed, this is precisely what you'd get by using the Nix package manager (and you get extra goodness on top if you use the NixOS distribution): you write what amounts to a Dockerfile in a slightly different syntax, but then you get all of the above benefits. https://nixos.org/nix/ https://nixos.org/nix/ https://nixos.org/ https://nixos.org/