4 ms·

The main reason why containers are (a lot) better than the former status quo is shockingly simple: Dockerfiles. They list the steps that you need to build a wor

by jchw 3y ago

The main reason why containers are (a lot) better than the former status quo is shockingly simple: Dockerfiles. They list the steps that you need to build a working image. And yep, there are some caveats, but most of the time they do not cause problems (example: I have containers where :latest has been fine for over 5 years.) I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways. But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.

Corsome 3y ago
Agreed with most of the points raised here although reproducible images are currently hard to achieve due to technical reasons on how docker builder operates. See https://medium.com/nttlabs/bit-for-bit-reproducible-builds-with-dockerfile-7cc2b9faed9f https://medium.com/nttlabs/bit-for-bit-reproducible-builds-w...
- jchw 3y ago
  Yeah, I understand what you mean; bit-for-bit reproducibility is a great idea, and to be clear, I was not talking about that sort of reproducibility, though we should certainly strive for it.
pmontra 3y ago
Yes I remember the problems with Vagrant. I'm unsure about what's making Docker more predictable across machines than Vagrant. Possible reasons - it's usually headless - it comes with a layer that mounts the host file system, instead of installing extensions - better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac) - it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine. Anything else?
- simmerup 3y ago
  Similar to your last point, Dockerfiles are a lot easier to learn/use than Vagrant, making people more likely to put in the effort to doing them right
- seba_dos1 3y ago
  Not sure I get the last point. Spawning a shell in a container, persisting the changes made and distributing such edited container is trivial. Of course once you want to rebuild it reproducibly from scratch you need to edit the Dockerfile, but then it's exactly the same in Vagrant, isn't it? Perhaps the difference is that storing your changes is an explicit operation that needs to happen consciously, and that data is usually persisted outside of the container.
  - pmontra 3y ago
    The scenario is: write a Vagrantfile and create the VM on Linux, then manually fix it or evolve it and never destroy and rebuild the VM. Then give that Vagrantfile to somebody on Windows a couple of years later. Chanches are that it won't work as intended. That's more difficult to happen with a container and its image.
friendzis 3y ago
On one hand, layered docker builds mean that with some care you can only care about the top layer and treat base layers as immutable. As long as they are not repulled. On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away. What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.
greiskul 3y ago
Even if I wasn't using containers for production, the ability to make a repeatable build allows you to make software with complex dependencies extremely easy to develop in. Making it possible for a new developer in a new environment to get a build working on their own laptop, in the first day on the job, didn't use to be simple. And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.
- verdverm 3y ago
  You have been able to do that with VMs for a long time
  - jchw 3y ago
    As a long time user of VMs for this purpose, this is definitely missing the point. Containers are sort of a technology, or perhaps some combination of technologies, but the actual value of containers for development has nothing to do with cgroups, seccomp, etc. In fact, using firecracker/ignite/whatever would probably be just as effective as using Docker or Podman. It really does come down to the concepts that underlie OCI images and Dockerfiles. Why? - Immutable root design promotes hygienic containers that keep state only in mounts. This makes upgrading, backup, etc. of containers very easy. Containerized software can be written around the assumption that this is how things work, by having a way to e.g. run database migrations at startup. - Composable images: Rather than having every environment ever made try to perfect making a reproducible Go development setup, you can just use FROM statements to use other images. With multi-stage builds you can even compose multiple images together to an extent. - Single-process design: Prior to containers, most VMs operated like full machines that could standalone, with just emulated or paravirtual hardware. Containers are designed around running a single process. In practice, this dramatically lowers the complexity of containers and their lifecycle. Of course, you Can and sometimes even will need to have multiple processes in containers sometimes, but the general mantra holds well. - Composable containers: Because of the previous point, it's natural to compose containers too, using features like Podman or Kubernetes pods, or Docker Compose. Containers in the same pod can share resources that would otherwise be isolated across typical containers, such as a network namespace. This makes for concepts like sidecars that allow composable functionality, and the ability to e.g. re-use PostgreSQL and Redis containers. - As a bonus, these approaches avoid plenty of typical pitfalls of configuration management systems often used with e.g. Vagrant such as Ansible. Because the environment is built up from scratch to your specifications, rather than working backwards from a mutable Linux system and trying to bring it to match a certain specification. Some of the problems that can lead to configuration drift still exist, but they are somewhat contained, especially if you use tags and/or image digests to pin your images. Virtual machines are technically fine, but containers elegantly offer solutions for both development and production deployment issues that VM setups never really did prior. Indeed, you could make something like testcontainers but for VMs... but then what do you use in place of OCI images and Dockerfiles for the base images? And you need to handle networking between VMs and the host, probably across multiple different VM runtimes. Eventually you realize you're just making worse containers!
trabant00 3y ago
I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't. I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath. Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.
- rahoulb 3y ago
  (I'm no expert on sysadmin stuff) They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup. When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.
- renegade-otter 3y ago
  Every new generation of devs is determined to learn the hard way that "complexity kills". No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.
- raffraffraff 3y ago
  The problem with config management is that it turns into a sprawling unmaintainable mess when you include everything in it. Back in 2014 I was working for a growing startup that was heavy into puppet. Everything was done through puppet. The puppet codebase was a gigantic mess. It has to care about a hundred different types of node. We containerized all of the internal apps and many of the other services (except for things like DNS, database and a few other bits and pieces). So yes, of course we "still had config management", but it shrunk from being a huge mess that we struggled to release weekly (and it always broke shit) to a simple codebase that was only concerned configuring the OS. Every developer was responsible for understanding their own container, and how it ran in production. And the infra team was responsible for providing a fleet of machines for that shit to run on. And a devops team took charge of building a system that scheduled containers on the fleet (pre-kubernetes-and-nomad). Simple, daily releases, and a nice separation of concerns.
matthewcroughan 3y ago
What you suggested about the listed steps is a bad suggestion. Docker should crash if you don't use a sha256 in a Dockerfile, or at least make some sort of lock file. But it instead allows you to make mistakes. I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology. If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes). > They list the steps that you need to build a working image. No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions. Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox". Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis. TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.

by jchw 3y ago

Corsome 3y ago

Agreed with most of the points raised here although reproducible images are currently hard to achieve due to technical reasons on how docker builder operates. See https://medium.com/nttlabs/bit-for-bit-reproducible-builds-with-dockerfile-7cc2b9faed9f https://medium.com/nttlabs/bit-for-bit-reproducible-builds-w...

jchw 3y ago
Yeah, I understand what you mean; bit-for-bit reproducibility is a great idea, and to be clear, I was not talking about that sort of reproducibility, though we should certainly strive for it.

pmontra 3y ago

Yes I remember the problems with Vagrant. I'm unsure about what's making Docker more predictable across machines than Vagrant. Possible reasons - it's usually headless - it comes with a layer that mounts the host file system, instead of installing extensions - better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac) - it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine. Anything else?

simmerup 3y ago
Similar to your last point, Dockerfiles are a lot easier to learn/use than Vagrant, making people more likely to put in the effort to doing them right
seba_dos1 3y ago
Not sure I get the last point. Spawning a shell in a container, persisting the changes made and distributing such edited container is trivial. Of course once you want to rebuild it reproducibly from scratch you need to edit the Dockerfile, but then it's exactly the same in Vagrant, isn't it? Perhaps the difference is that storing your changes is an explicit operation that needs to happen consciously, and that data is usually persisted outside of the container.
- pmontra 3y ago
  The scenario is: write a Vagrantfile and create the VM on Linux, then manually fix it or evolve it and never destroy and rebuild the VM. Then give that Vagrantfile to somebody on Windows a couple of years later. Chanches are that it won't work as intended. That's more difficult to happen with a container and its image.

friendzis 3y ago

On one hand, layered docker builds mean that with some care you can only care about the top layer and treat base layers as immutable. As long as they are not repulled. On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away. What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.

greiskul 3y ago

Even if I wasn't using containers for production, the ability to make a repeatable build allows you to make software with complex dependencies extremely easy to develop in. Making it possible for a new developer in a new environment to get a build working on their own laptop, in the first day on the job, didn't use to be simple. And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.

verdverm 3y ago
You have been able to do that with VMs for a long time
- jchw 3y ago
  As a long time user of VMs for this purpose, this is definitely missing the point. Containers are sort of a technology, or perhaps some combination of technologies, but the actual value of containers for development has nothing to do with cgroups, seccomp, etc. In fact, using firecracker/ignite/whatever would probably be just as effective as using Docker or Podman. It really does come down to the concepts that underlie OCI images and Dockerfiles. Why? - Immutable root design promotes hygienic containers that keep state only in mounts. This makes upgrading, backup, etc. of containers very easy. Containerized software can be written around the assumption that this is how things work, by having a way to e.g. run database migrations at startup. - Composable images: Rather than having every environment ever made try to perfect making a reproducible Go development setup, you can just use FROM statements to use other images. With multi-stage builds you can even compose multiple images together to an extent. - Single-process design: Prior to containers, most VMs operated like full machines that could standalone, with just emulated or paravirtual hardware. Containers are designed around running a single process. In practice, this dramatically lowers the complexity of containers and their lifecycle. Of course, you Can and sometimes even will need to have multiple processes in containers sometimes, but the general mantra holds well. - Composable containers: Because of the previous point, it's natural to compose containers too, using features like Podman or Kubernetes pods, or Docker Compose. Containers in the same pod can share resources that would otherwise be isolated across typical containers, such as a network namespace. This makes for concepts like sidecars that allow composable functionality, and the ability to e.g. re-use PostgreSQL and Redis containers. - As a bonus, these approaches avoid plenty of typical pitfalls of configuration management systems often used with e.g. Vagrant such as Ansible. Because the environment is built up from scratch to your specifications, rather than working backwards from a mutable Linux system and trying to bring it to match a certain specification. Some of the problems that can lead to configuration drift still exist, but they are somewhat contained, especially if you use tags and/or image digests to pin your images. Virtual machines are technically fine, but containers elegantly offer solutions for both development and production deployment issues that VM setups never really did prior. Indeed, you could make something like testcontainers but for VMs... but then what do you use in place of OCI images and Dockerfiles for the base images? And you need to handle networking between VMs and the host, probably across multiple different VM runtimes. Eventually you realize you're just making worse containers!

trabant00 3y ago

I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't. I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath. Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.

rahoulb 3y ago
(I'm no expert on sysadmin stuff) They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup. When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.
renegade-otter 3y ago
Every new generation of devs is determined to learn the hard way that "complexity kills". No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.
raffraffraff 3y ago
The problem with config management is that it turns into a sprawling unmaintainable mess when you include everything in it. Back in 2014 I was working for a growing startup that was heavy into puppet. Everything was done through puppet. The puppet codebase was a gigantic mess. It has to care about a hundred different types of node. We containerized all of the internal apps and many of the other services (except for things like DNS, database and a few other bits and pieces). So yes, of course we "still had config management", but it shrunk from being a huge mess that we struggled to release weekly (and it always broke shit) to a simple codebase that was only concerned configuring the OS. Every developer was responsible for understanding their own container, and how it ran in production. And the infra team was responsible for providing a fleet of machines for that shit to run on. And a devops team took charge of building a system that scheduled containers on the fleet (pre-kubernetes-and-nomad). Simple, daily releases, and a nice separation of concerns.

matthewcroughan 3y ago

What you suggested about the listed steps is a bad suggestion. Docker should crash if you don't use a sha256 in a Dockerfile, or at least make some sort of lock file. But it instead allows you to make mistakes. I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology. If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes). > They list the steps that you need to build a working image. No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions. Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox". Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis. TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.