5 ms·

This is mostly a good post, but he doesn’t mention one huge security disadvantage of containers: When application artifacts all include their own versions of de

• aeorgnoieang 9y ago
  Why's that a disadvantage of containers versus, e.g. third-party libraries used by an application?
  • Retric 9y ago
    Mostly scale, 3rd party libraries are a subset of the dependencies you see in a container.
  • wrs 9y ago
    You can update a shared library without regard to the applications using it. You just "apt update" the vulnerable library and all applications get it. (In theory--but the other side of this argument is that occasionally that will trigger a bug.) If it's a static library, see my zlib comment. Those have the same problem.
    • deathanatos 9y ago
      > You just "apt update" the vulnerable library and all applications get it. And restart every effected service. You can't "just" run the apt update; the binaries using any shared libraries must be restarted so as to load the newer version of the library. I find this is often the harder problem to solve. (If you can just reboot the machine, do that, it's easier; otherwise you have to track down who was using the old binary and restart each one of them…) (sorry, this is a rather critical step that I've seen missed before.)
      • maccam94 9y ago
        Yeah, I wouldn't say this is common knowledge. It's not terribly hard though, on debian-based systems you can just use checkrestart.
      • merb 9y ago
        as far as I know as long as a shared library is loaded you can't just restart every effected service. the os would first need to unload the shared library, thus if two services need the same library you need to stop both first and start them again. thus making the "positive" side of shared libraries basically look very bad on paper. I think that a lot of systems are affected by that problem. I like shared libraries for a desktop os and applications where you often reboot. But for a server? Hell it's easier to upgrade golang applications/java applications (if the jvm has no bugs and since j9 even bundled) is "saner" to upgrade. also what a lot of people just don't get is that a sane system mostly only has one service per os/container. thus docker pull vs apt update is basically the same.
        • teraflop 9y ago
          > the os would first need to unload the shared library, thus if two services need the same library you need to stop both first and start them again. No, that's not how it works, at least on Unix systems. Shared libraries are mapped into each process's address space separately; there's no system-wide mechanism that prevents different processes from running different versions of the same library at the same time.
        • blattimwind 9y ago
          If you try to write a file backing an executable mapping you get ETXTBSY (or not, if it's NFS/some other broken FS). So what everyone is doing is unlink the old file and create a new one. Unlinking preserves open file descriptors, extending to mappings. Note that often the actual backing files are disjunct anyway and only symlinks get replaced, due to versioned .so names (as in libfoo.so.2.4.5, libfoo.so.2.4@, libfoo.so.2@, libfoo.so@, you've all seen it).
      • spencercw 9y ago
        CentOS has a great utility called needs-restarting which finds processes which have been updated and need to be restarted. I just stick it in a cron job and I have an email waiting for me in the morning if I need to do anything.
• subway 9y ago
  Good luck with the fork-and-rebuild strategy. It's a maddening process to track down then vendor all the upstream Dockerfiles when you want to use a DockerHub image, but don't trust DockerHub binary images. I seriously don't get how folks are comfortable with using all these binary layers built and rebuilt from unspecified versions of other binary layers, all shipped with no cryptographic verification by default. (Transport security with SSL is good enough, yeah? sob)
  • wrs 9y ago
    True, but really the verification problem isn't nearly as bad as the fact that people are running perfectly valid year-old images with year-old vulnerabilities permanently ensconced.
  • im_down_w_otp 9y ago
    We're in a situation of needing to create a very stringent SDLC + supporting tooling for use in safety-critical software development, and it's sometimes surprising how much of the traditional ecosystem (and thus convenience) we have to opt-out of because of how many layers of weak trust + verification we'd have to be willing to take on the liability for using without meaningful ways to measure how exposed to risk we'd be. It's thankfully a very interesting problem to have to address, but it does mean that our time-to-market is inherently atypical compared to others.
    • ljm 9y ago
      It’d be a great strategy for Docker, considering what looks like an industry convergence on K8S as an orchestration platform. Double down on the Dockerfile format and abstract things that would otherwise be frozen in time, taking ownership of some aspects of DockerHub that are currently managed by contributors. They do this already with official language images but there has to be more to it outside of that than expecting people to figure out the intimate details of each base image while keeping them up to date across OS versions and library versions.
      • pm90 9y ago
• convolvatron 9y ago
  this is really about static vs dynamic linkage, regardless of the mechanism you use to effect run or build time binding of dependencies. there are as always tradeoffs, but a few more arguments against dynamic: by opening up a path (automatic updates) to a third party, you're providing a wide vector for those updates to cause new* problems (correctness, security) for your deployment, as well as fix old ones transitive api compatibility across these upgrades is more of a statement of faith and effort of will than any real guarentee. in reality the compatibilities between the various components are piecewise, and if the graph gets big enough it can be really hard to find a compatible set of libraries. runtime binary integration is always strictly worse than build time source integration, since there is tooling in the latter promoting consistency, and basically none in the former since you'd really like to be able to deploy changes in your own* software with very little effort - dependency updates can ride on the same train for free. presumably you have to have some intervention to restart your service against the new dependencies regardless. at that point its a question of the relative complexity of a dynamic restart of an existing service vs a build and deploy. in the limit these are both buttons. from that perspective, dynamic updates are a helpful crutch for people who don't have a good handle on their own build, test and deploy. thats pretty far from being an arguable fundamental good.
  • wrs 9y ago
    >runtime binary integration is always strictly worse than build time source integration That is true, except in the case where you aren't empowered to do the building. Then dynamic update is better than no update.
  • pjc50 9y ago
    > runtime binary integration is always strictly worse than build time source integration This is rather harsh on "plugin" architectures, COM, and OS libraries.
    • pm90 9y ago
      It may be harsh but its true. Don't get me wrong: plugin architectures, OS libraries etc. do make the problem much easier to solve and 99% of the time you will be OK. Its that 1% outlier, when a dependency you thought was rock-solid-stable that needs an immediate security patch but fails in weird ways when you apply it that it really shows up. And because its so infrequent you may not even have any mechanism in place to fix it quickly.
• cstrahan 9y ago
  > This is mostly a good post, but he doesn’t mention one huge security disadvantage of containers: When application artifacts all include their own versions of dependencies locked at build time, it becomes very hard to update a component that has a security fix. I absolutely agree. So how do we fix this? Well, first we need to identify what's broken, and then we can talk about how to fix it. We could look at container images and see that, in the pursuit of reproducibility/determinism, they make it infeasible to make critical updates; from this we could infer that determinism is the root of our security woes, and that we have to choose between having a run-time environment that you can depend on, or being able to promptly make critical security updates. Fortunately, that a false dichotomy. Actually, it's even better than that: it turns out there's a third option that gives you all of the above: 1. The ability to quickly make security updates 2. A deterministic and serializable run-time environment 3. A deterministic and serializable build-time environment Note the addition of bullet point #3. In something like Docker, you can successfully build an image from a Dockerfile today, and yet it could fail to build tomorrow. You can build a Dockerfile on your machine, but it might not build successfully on my machine. What I am claiming is that there is another approach to containers that would additionally provide determinism when you're building the container's root file system, whereas e.g. Docker only gives you determinism in the sense that a previously built image can be passed around. So what system could provide all of the above? How could we make that work? (If you get bored with the details, feel free to skip down to the spoiler after TL;DR) We'd first need a deterministic build mechanism; broadly speaking, one in which we could build e.g. executables, fetch tarballs, construct file-systems, etc. To be deterministic, we our build system must have certain properties: * "normal" builds must be cut off from network access, whereas * A special "fixed output" build type could e.g. fetch tarballs, provided that e.g. a SHA-512 hash is given beforehand * Each build specification must specify what other build specs it depends on. Each build environment is constructed such that processes therein are only permitted to see/access files that are the resulting artifacts of those explicitly listed dependencies. Arbitrary read/write access to the host file-system is not permitted. * The resulting build artifacts go in their own immutable prefix, something like, let's say, /store/<hash>-<name>, where <hash> is a unique key derived by hashing the build specification, and <name> is a friendly name for the build artifact. To provide a concrete example, if you were to build `gcc`, you'd have something like /store/4r5kszyy0iirc5agfah45lvz7mnnsrb4-gcc-7.3.0/{bin,include,lib,lib64,libexec,share}, which would all contain the usual suspects. Note that when we hash the build specification, that spec refers to its dependencies, which in turn have their own unique hash, and so on. As a result of this design decision, we have: support for multiple versions of the same software; updates can't break existing executables (due to, let's say, breaking ABI changes in openssl or libc or whatever); and a final point that's important enough to split out as the following bullet point: * We can trivially construct deterministic environments: write a build spec that takes as an input all of the dependencies you wish to have in the environment, and (as the single resulting build artifact) produce a symlink tree that is the union of all the build artifacts of those aforementioned inputs. For example, you might want an environment with, I don't know, let's say gcc, python and ruby; you would end up with something like /store/m6n3y1rlz9rf06rj4spmvlkmv51angbm-container-env, which would contain bin/gcc, bin/python, bin/ruby, etc. Of course, having this environment wouldn't be much good unless you could actually run something inside of it, so that leads to the next point: * We can take an environment (as described in the previous point) and use it as the file system for a container: just do all of the usual container/namespace stuff, and e.g. symlink the env to /usr. A byproduct of this is that rollback/forward is easy: just bump a single symlink to the appropriate env path. * Due to our design, it's a trivial matter to serialize an environment and (re)distribute it. * Also important: the host system can build and retain all of the container environments in the host's /store. We could then bind-mount paths into the container as necessary. As a result, when two containers depend on precisely the same software, you only pay the price of a single copy. This is unlike Docker in that Docker requires that your image's layers directly inherit from each other; if you have, say for argument, an Ubuntu based image and a CentOS based image, it doesn't matter if there are components that are exactly the same in both - you have to pay the price of two copies. TL;DR -- TL;DR -- TL;DR --TL;DR We now have a system in which we can construct recipes to build container environments, and those recipes can be deterministically built, serialized, distributed, and then deserialized and used as a reproducible environment. Thanks to this, it's easy to tweak an existing recipe to bump up, say, openssl when a critical CVE is announced. We've beaten the typical image-based approach to containers (e.g. as used by Docker) in terms of security risk mitigation, disk foot print, developer time (from otherwise fighting with broken build specs / Dockerfiles), etc. Okay, so we don't actually have this system, we've just fantasized about one... ...unless something like this already exists! Indeed, this is precisely what you'd get by using the Nix package manager (and you get extra goodness on top if you use the NixOS distribution): you write what amounts to a Dockerfile in a slightly different syntax, but then you get all of the above benefits. https://nixos.org/nix/ https://nixos.org/nix/ https://nixos.org/ https://nixos.org/
• tylfin 9y ago
  After using https://github.com/coreos/clair https://github.com/coreos/clair to scan some official images, I was blown away by the amount of exploits in every one. Seems like the only path forward if security is a priority is to build the base layers from scratch...
  • AnIdiotOnTheNet 9y ago
    Good thing hardly anyone actually cares anywhere near as much about security as they claim to.
• snarfy 9y ago
  You could solve the problem with containers, but then you've re-implemented the same problem. You could have an authentication microservice that other services use that way authentication is handled in one place. If there is a security issue with the code in authentication, updating the one container fixes the whole system. But doing this has the same problem. You've just re-implemented DLL hell as container hell.