14 ms·

Containers are tents

192 points by vmarsy 5y ago

encryptluks2 5y ago
Starts off saying VMs are like brick and mortar houses and containers are like tents. I agree somewhat but there has been significant progress to sandbox containers with the same security we'd expect from a VM. It isn't a ridiculous idea that VMs will one day be antiquated, but probably won't happen for a few more years.
- kasey_junk 5y ago
  Do you have any links to secure container runtimes that don’t either virtualize or replace all the system calls of the container such that it might as well be virtual?
  - adolph 5y ago
    Maybe Singularity? https://sylabs.io/guides/3.5/admin-guide/configfiles.html https://sylabs.io/guides/3.5/admin-guide/configfiles.html
    - kasey_junk 5y ago
      Singularity is likely* less secure than default container runtimes. *not a security person or an expert on singularity but it advertises that it doesn’t do file system or user isolation by default
  - deleted 5y ago
    [deleted]
  - encryptluks2 5y ago
    First, saying it might as well be virtual is a bit of a misnomer. There are various options, and although they may act like a VM they are significantly faster than machine-based VMs like QEMU: https://kubernetes.io/docs/concepts/policy/pod-security-policy/ https://kubernetes.io/docs/concepts/policy/pod-security-poli... > As of Kubernetes v1.19, you can use the seccompProfile field in the securityContext of Pods or containers to control use of seccomp profiles. If you're looking for a more general abstraction, there is gVisor and others as well.
    - kasey_junk 5y ago
      Again, not an expert but security policies aren't immune from container breakouts right? Which leaves you to either use something like firecracker or gvisor which are either virtualization solutions or the next closest thing in that they intermediate all of your syscalls?
      - encryptluks2 5y ago
        Almost all container breakout concerns rely on running containers as a privileged user: https://stackoverflow.com/questions/53024790/kubernetes-support-for-docker-user-namespace-remapping https://stackoverflow.com/questions/53024790/kubernetes-supp... There is an issue that I've been tracking, and there has been a new PR that will hopefully land soon to implement this in Kubernetes in a simplified manner: https://github.com/kubernetes/enhancements/issues/127 https://github.com/kubernetes/enhancements/issues/127 As for whether security policies prevent breakouts, it really depends on what the exploit is but they can significantly help. The idea of user namespace remapping solves a secondary issue though... if there is a breakout, what user privileges will they have.
  - viraptor 5y ago
    We can't answer that question because "secure container runtime" is not a well defined idea. Secure from what, in what way, with what guarantees? Docker is both secure and not depending how you draw the lines.
    - kasey_junk 5y ago
      Sure. I mean as secure as a traditional virtualization environment.
  - jb_gericke 5y ago
    Pod security policies and seccomp for call filtering at an OCI level
- effie 5y ago
  You can't make the Linux kernel isolation of processes as secure as Xen or Firecracker or SEL4 can. Yes, processes can be restricted to subset of syscalls and system resources but Linux is just too big and its attack surface is too big to put it on the same level of confidence as above hypervisors.
- mercora 5y ago
  i don't think that is necessarily the case but instead i believe in the near future the differences between container sandboxes and virtual machines might be less clear. I imagine CPU and memory namespaces coming implemented on hardware isolation features like VT-d io-mmus and alike thus making virtual machines integrated into some sandboxing feature.
tptacek 5y ago
That's a valid way to look at it, but there are other ways. Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics. That sense of the term isn't loaded with any specific notion of how attack surfaces should work. I think modern "Docker"'s security properties are underrated†. But you still can't run multitenant workloads from arbitrary untrusted tenants on shared-kernel isolation. It turns out to be pretty damned useful to be able to ship application components as containers, but have them run as VMs. † https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/
- RcouF1uZ4gsC 5y ago
  > Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics. What I find interesting, is that many uses of containers are just reinventing statically linked binaries in a more complicated form.
  - brian_cunnie 5y ago
    > many uses of containers are ... statically linked binaries in a more complicated form I have found that to be true in at least one case—I had built a custom DNS server in Go (statically linked), and originally planned to run it in a container, but on further reflection realized the container brought no added value, and it was much simpler to write a systemd service control script than to bring in the extra baggage of a container ecosystem to run the DNS server.
    - ric2b 5y ago
      That's a very basic example. Let's say your program also depends on ffmpeg to convert some images, psql to interact with a database and a few other non-library dependencies. With containers you can trivially ensure those are always present and with the correct versions. Plus containers do give you some security benefits when compared to running natively.
unethical_ban 5y ago
Container tech can be used for small scale "pet" deployment, but my understanding is that the true benefit of containers come with seeing them as "cattle". You should never login to the shell of a container for config. All application state lives elsewhere, and any new commit to your app triggers a new build. If that's not for you, then while containers like proxmox/LXC can still be handy, you're just doing VM at a different layer. The article was a bit hand wavey about how "they" complain about containers, and then uses the analogy more than explaining the problems and solutions.
- deleted 5y ago
  [deleted]
- Sparkyte 5y ago
  Only time it should be utilized as a small scale "pet" is when you externalize the storage mounts and it is an on-demand non-persistent virtual environment not directly connected to a data sensitive environment. That's mainly my take on it. I will often use docker locally to test out some kubernetes service connectivity, but never bring my Frankenstein stuff into an environment any higher than local.
- foobar33333 5y ago
  >You should never login to the shell of a container for config I absolutely do this and think it works great. Fedora has built a tool called "toolbox" which is basically a wrapper on podman which can create and enter containers where you can install development tools without touching your actual OS. I basically do all of my development inside a container where the source code is volume mounted in but git/ruby/etc only exist in the container. This has the benefit of letting me very quickly create a fresh env to test something out. Recently I wanted to try doing a git pull using project access tokens on gitlab and containers let me have a copy of git which does not have my ssh keys in it. This is somewhat of an edge case though, for a server deployment, yes you shouldn't rely on anything changed inside the container and should use volume mounts or modify the image.
smitty1e 5y ago
I explain that if the an Amazon Virtual Private Cloud (VPC) is a datacenter "cloud", then a container implementation is a "puff". Virtualizing the kernel like the Amazon Machine Image (AMI) virtualizes a chip core sounds great. But now, in the "puff", all of those networking details that AWS keeps below the hypervizor line confront us. Storage, load balancing, name services, firewalls. . . Containers can solve packaging issues, but wind up only relocating a vast swath of other problems.
- wmf 5y ago
  If you have few enough containers you can give each one an ENI.
- zaphirplane 5y ago
  I have to say inventing a new metaphor for container made it harder to follow the point you were trying to make
lamontcg 5y ago
containers are fat RPMs that construct a chroot jail.
eVeechu7 5y ago
>Finally, there’s the whole business of resource isolation. While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions. I don't think virtualization really offers hardware-level guarantees against noisy neighbours either.
- habeebtc 5y ago
  It offers the opportunity to throttle noisy neighbors in hopes the party isn't too big.
  - dilyevsky 5y ago
    Cgroups can do the same via cfs
- amarshall 5y ago
  VMs provide stronger guarantees for maximum CPU, network, and disk usage, as well as memory size consumption. This because the abstraction boundaries are fairly clear (e.g. threads and virtual devices).
Helmut10001 5y ago
I place all my tents in a house (Docker VMs inside unprivileged LXC containers on Proxmox - yes, unprivileged = not a brick house, more like wood). The only reason I use Docker is that I can access the system design knowledge that is available with docker-compose.yml's. Last example: Gitlab. Could not get it running on unrivileged LXC using the official installation instructions, with Docker it was simply editing the `.env` and then `docker-compose up -d`. All of this in a local, non-public (ipsec-distributed) network. I often find myself creating a single, separate unprivileged LXC container->Docker nesting for each new container, because I do not need to follow the complicated and error prone installation instructions for native installs.
Sparkyte 5y ago
I was totally expecting this to go in the direction about tech debt with a homeless analogy, but it was about destructability. Yes we know this already and if you catch people treating it as a persistent host, slap their hands and say no.
rossmohax 5y ago
I believe, that success of containers is not because of lightweightness or other isolation properties of them. Containers won dev mindshare because of ease packaging and distribution of the artifacts. Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.
- enw 5y ago
  > glorified tarballs Calling container images glorified tarballs is like calling cars glorified lawnmowers.
  - kjeetgill 5y ago
    Maybe closer to calling a 16-wheeler a glorified minivan. I think there's something meaningful in that comparison.
  - brazzy 5y ago
    Um...a container image is very literally and exactly a collection of tarballs and some JSON files with metadata.
- aequitas 5y ago
  > Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up. Packaged VM's existed for a while already with thing like Vagrant on top, there was also already LXC which leaned more into the VM concept. Where Docker made the difference imho is with Dockerfiles and the layered/cached build steps.
- throwaway894345 5y ago
  > Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up. IMO the important, catalyzing difference is that Docker containers have a standard interface for logging, monitoring, process management, etc which allow us to just think in terms of “the app” rather than the app plus the SSH daemon, log exfiltration, host metrics daemon, etc. In other words, Docker got the abstraction right: I only care about the app, not all of the ceremony required to run my app in a VM. These common interfaces allow orchestration tools to provide more value: they aren’t just scheduling VMs, they’re also managing your log exfiltration, your process management, your SSH connection, your metrics, etc, and all of those things are configurable in the same declarative format rather than configuring them with some fragile Ansible playbook that requires you to understand each of the daemons it is configuring, possibly including their unique configuration file/filesystem conventions and syntaxes.
jb_gericke 5y ago
Enjoyed the article but having watched containerization and kubernetes maturing over the last 5 years (especially at an enterprise level), I'd say a huge part of the value proposition is (and this applies more to K8) it really catalyses prototyping/experimenting and (depending on the org I suppose) promotes a lot of autonomy for app teams who'd historically have to log calls to infrastructure to get compute, network/lb/dns, databases et al. built up before kicking the tyres on something. I've seen those types of things take months in large orgs. And then there's the inevitable drift between tiered environments that happens over time in richer operating environments (I've seen VMs so laden with superfluous monitoring and agentware they fall over all the time, while simultaneously being on completely different OS and patch versions from dev to prod). Containers provide immutability at the service layer, so I have confidence in at least having that level of parity between dev and prod (albeit hardly ever at a data or network layer).
znpy 5y ago
> We don’t expect tents to serve the same purpose as brick-and-mortar houses—so why do we expect containers to function like VMs? Marketing. Because of Marketing.
deleted 5y ago
[deleted]
jsiepkes 5y ago
Should be noted that a portion of this (valid) criticism applies specifically to the most prominent "container" implementation; Docker. Not containers as a whole. For example resources isolation with the Solaris / Illumos container implementation (zones) works just as well as full blown virtualization. You are just as well equipped to handle noisy neighbors with zones as you are with hardware VM's. > Much as you’d likely choose to live in a two-bedroom townhouse over a tent, if what you need is a lightweight operating system, containers aren’t your best option. So I think this is true for Docker but doesn't really do justice to other container implementations such as FreeBSD jails and Solaris / Illumos zones. Because those containers are really just lightweight operating systems. In the end Docker started out and was designed to be a deployment tool. Not necessarily an isolation tool in all aspects. And yeah, it shows.
- belter 5y ago
  I can not agree more. It is the saddest thing the appalling implementation of Docker, and the whole lack of security around the ecosystem, made people think Containers equal to Docker. Docker is what happens when you put your security implementation in the hands of your Developer team and not in the hands of your DevSecOps people.
- nyx__ 5y ago
  Criticism applies to all Linux containers, not just Docker, which is one implementation of Linux containers. One could argue that zones are distinct from containers (a Linux implementation), with both being OS specific versions of jails.
eyeyeyerg 5y ago
containers are cattle, VMs were pets. If one does not get the operational differences nor understands that these are completely two different usescases then probably should not work in IT industry
- detaro 5y ago
  VMs can be cattle. Physical machines can be cattle. This is not tied to the runtime technology, but to how you design and manage your machines and applications.
  - cratermoon 5y ago
    The original pet v cattle metaphor was indeed inspired by servers, not containers. http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-... What makes a given component – server, vm, container, whatever – is not the runtime, but how you deal with it when it gets seriously ill. Pets are taken to the vet or hospital to get treatment. Cattle are.. well, read the article I linked.
johbjo 5y ago
I'd be curious to see services designed to run as PID 1 inside containers, and contain or run nothing else other than the required binaries. Maybe someone is doing this.
- onei 5y ago
  Pretty sure this is what tini[1] is for, and there's supervisord[2] for more complex use cases, e.g. running multiple processes. Neither are a full replacement for traditional systemd, which I heard you can technically run in a container these days, but I've never seen anyone try. [1] https://github.com/krallin/tini https://github.com/krallin/tini [2] https://github.com/Supervisor/supervisor https://github.com/Supervisor/supervisor
- antonvs 5y ago
  It's pretty common for languages that compile to binaries. Golang, for example. Just inherit your container "FROM scratch". I have a number of service containers that are around 8 - 12MB, which contain compiled Haskell binaries.
- nijave 5y ago
  Related, https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
nickjj 5y ago
IMO comparing containers to an apartment is more accurate than a tent. Because with an apartment each tenant gets to share certain infrastructure like heating and plumbing from the apartment building, just like containers get to the share things from the Linux host they run on. In the end both houses and apartments protect you from outside guests, just in their own way. I went into this analogy in my Dive into Docker course. Here's a video link to this exact point: https://youtu.be/TvnZTi_gaNc?t=427 https://youtu.be/TvnZTi_gaNc?t=427, that video was recorded back in 2017 but it still applies today.
leephillips 5y ago
I’ve found systemd-nspawn useful. Use debootstrap to install a minimal Debian system inside your system, then boot it with this command. It isolates the filesystem while sharing the network interface, and is convenient for most things that I guess people use Docker for. I wonder why it’s not mentioned more often.
- dijit 5y ago
  I know it sounds like I want to be spoonfed, but do you have a walkthrough of this flow? I'd be interested in trying it out but I don't want to spend some hours reading documentation trying to get it working.
  - leephillips 5y ago
    Well, that’s OK, because it did take me a while to track down the pieces of the documentation and find a procedure that worked for me. There is some less-than-optimal advice out there about this. Become root. Install debootstrap, which is in the Debian and Ubuntu repositories, at least. Make a directory to contain your embedded system. It can be anywhere. Let's use /var/lib/machine/machinename. This command will install a new, minimal Debian system in that directory: debootstrap --include=systemd-container stable /var/lib/machines/machinename http://deb.debian.org/debian http://deb.debian.org/debian It will download everything and, if I recall correctly, works unattended (doesn’t ask questions). Enter the container with systemd-nspawn -D /var/lib/machines/machinename/ and set the root container password with passwd. Then do echo 'pts/0' >> /etc/securetty so the guest OS will let you log in after it's booted up. You may have to add other pts/x entries. I'm not sure about this part; it may be that if there is no /etc/securetty file that there is no problem. Now log out of the container. To boot up the guest OS, use systemd-nspawn -b -D /var/lib/machines/machinename You will see the familiar console messages. You will find advice on the web to include the -U flag here, which causes files in the guest OS to only use UIDs known to the guest OS when determining ownership and permissions. This leads to headaches, because you have to set the ownership of any file you copy in from the host system. Leave it out, and you can have parallel users on the host and guest OSs, which is more convenient. But you may have to change the UIDs of the users on the guest OS so that they match. Now, on the host OS, you can use the `machinectl` command to control all your guest OSs. `machinectl list` shows you what’s running, `machinectl login` lets you log in to them, and there are several commands for killing them with various levels of violence. If you want your machine to be a long-running service, just `nohup` the spawn command, and direct output as desired. If you want to be able to communicate with your machine from the internet, opening sockets from within the guest OS works, as they share the network interface. For a public-facing web service, you can install (for example) Apache and pick a port number to listen on, then set up a reverse proxy on the host OS, using a dedicated domain or a subdomain, so the users don’t have to use the custom port number. I’ve found that certificates for HTTPS need to be installed on both the host and guest OSs. Good luck! More information: https://wiki.debian.org/nspawn https://wiki.debian.org/nspawn
1MachineElf 5y ago
You had me at containers
luord 5y ago
I've seen the problems of treating containers as houses, primarily during development: Multiple different processes inside a single container, with a wrapper around them (inside the container) that makes it even more difficult to debug. So, assuming I understood correctly, treating them like tents is infinitely the better choice.
0xcmoney 5y ago
Am I the only one getting tired of people stating confidently that containers don't improve safety _at all_ because they run on the same kernel? It's just not true.
nijave 5y ago
Modern containers do provide lots of security features with namespaces, seccomp, cgroups (to some extent) The author seems to largely ignore this. I would consider that a bit stronger than a "tent wall". Comparing it to a tent seems more akin to a plain chroot. If I have my tent right next to someone else, I can trivially "IPC" just speaking out loud which would be prevented by an IPC namespace (which is Docker's current default container setup) Also worth mentioning, turning a container into a VM (for enhanced security) is generally easier than trying to do the opposite. AWS Lambda basically does that as do a lot of the minimal "cloud" Linux distributions that just run Docker with a stripped down userland (like Container Linux and whatever its successors are)
- throwaway894345 5y ago
  I’m a big proponent of containers, but in fairness to TFA, I don’t know how to configure namespaces, second, or cgroups and I don’t know what settings my orchestrator uses by default. If containers can be secure but we don’t enable those security features properly, then it’s a bit of a moot point. That said, I think (but am not sure) most of us understand enough not to trust containers for isolation between untrusted processes, so I don’t regard containers as lightweight VMs, but rather collocated processes with their own namespaces. When I run untrusted code, like jupyterhub, I make sure those untrusted containers get scheduled onto their own dedicated mode pool with single tenancy (at which point the container is more of a tooling/orchestration convenience than a resource optimization tool).
fsociety 5y ago
This is oversimplifying containers and VM by using the house vs tent analogy. Just talking about Docker weakens this too, because Docker is not the only way to setup containers. > Tents, after all, aren’t a particularly secure place to store your valuables. Your valuables in a tent in your living room, however? Pretty secure. Containers do provide strong security features, and sometimes the compromises you have to make hosting something on a VM vs. a container will make the container more secure. > While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions. Cgroups are more than a neat resource isolation mechanism, they work. That's really all there is to it. Paranoia around trusting the Linux kernel is unnecessary if at the end of the day you end up running Linux in production. If anything breaks, security patches will come quick and the general security attitude of the Linux community is improving everyday. If you are really paranoid, perhaps run BSD, use grsec, or the best choice is to use SELinux IMO. If anything, you will be pwned because you have a service open to the world, not because cgroups or containers let you down.
- deleted 5y ago
  [deleted]
deleted 5y ago
[deleted]
srg0 5y ago
If VM is like a nuclear war bunker, containers are like brick and mortar houses. They are not air tight and have glass windows which can be easily broken, but that's where most people live. They're cheaper to build, comfortable enough, and can last a human lifetime most of the time. An analogy can go a long way. Both ways.

192 points by vmarsy 5y ago

Starts off saying VMs are like brick and mortar houses and containers are like tents. I agree somewhat but there has been significant progress to sandbox containers with the same security we'd expect from a VM. It isn't a ridiculous idea that VMs will one day be antiquated, but probably won't happen for a few more years.

kasey_junk 5y ago
Do you have any links to secure container runtimes that don’t either virtualize or replace all the system calls of the container such that it might as well be virtual?
- adolph 5y ago
  Maybe Singularity? https://sylabs.io/guides/3.5/admin-guide/configfiles.html https://sylabs.io/guides/3.5/admin-guide/configfiles.html
  - kasey_junk 5y ago
    Singularity is likely* less secure than default container runtimes. *not a security person or an expert on singularity but it advertises that it doesn’t do file system or user isolation by default
- deleted 5y ago
  [deleted]
- encryptluks2 5y ago
  First, saying it might as well be virtual is a bit of a misnomer. There are various options, and although they may act like a VM they are significantly faster than machine-based VMs like QEMU: https://kubernetes.io/docs/concepts/policy/pod-security-policy/ https://kubernetes.io/docs/concepts/policy/pod-security-poli... > As of Kubernetes v1.19, you can use the seccompProfile field in the securityContext of Pods or containers to control use of seccomp profiles. If you're looking for a more general abstraction, there is gVisor and others as well.
  - kasey_junk 5y ago
    Again, not an expert but security policies aren't immune from container breakouts right? Which leaves you to either use something like firecracker or gvisor which are either virtualization solutions or the next closest thing in that they intermediate all of your syscalls?
    - encryptluks2 5y ago
      Almost all container breakout concerns rely on running containers as a privileged user: https://stackoverflow.com/questions/53024790/kubernetes-support-for-docker-user-namespace-remapping https://stackoverflow.com/questions/53024790/kubernetes-supp... There is an issue that I've been tracking, and there has been a new PR that will hopefully land soon to implement this in Kubernetes in a simplified manner: https://github.com/kubernetes/enhancements/issues/127 https://github.com/kubernetes/enhancements/issues/127 As for whether security policies prevent breakouts, it really depends on what the exploit is but they can significantly help. The idea of user namespace remapping solves a secondary issue though... if there is a breakout, what user privileges will they have.
- viraptor 5y ago
  We can't answer that question because "secure container runtime" is not a well defined idea. Secure from what, in what way, with what guarantees? Docker is both secure and not depending how you draw the lines.
  - kasey_junk 5y ago
    Sure. I mean as secure as a traditional virtualization environment.
- jb_gericke 5y ago
  Pod security policies and seccomp for call filtering at an OCI level
effie 5y ago
You can't make the Linux kernel isolation of processes as secure as Xen or Firecracker or SEL4 can. Yes, processes can be restricted to subset of syscalls and system resources but Linux is just too big and its attack surface is too big to put it on the same level of confidence as above hypervisors.
mercora 5y ago
i don't think that is necessarily the case but instead i believe in the near future the differences between container sandboxes and virtual machines might be less clear. I imagine CPU and memory namespaces coming implemented on hardware isolation features like VT-d io-mmus and alike thus making virtual machines integrated into some sandboxing feature.

tptacek 5y ago

That's a valid way to look at it, but there are other ways. Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics. That sense of the term isn't loaded with any specific notion of how attack surfaces should work. I think modern "Docker"'s security properties are underrated†. But you still can't run multitenant workloads from arbitrary untrusted tenants on shared-kernel isolation. It turns out to be pretty damned useful to be able to ship application components as containers, but have them run as VMs. † https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/

RcouF1uZ4gsC 5y ago
> Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics. What I find interesting, is that many uses of containers are just reinventing statically linked binaries in a more complicated form.
- brian_cunnie 5y ago
  > many uses of containers are ... statically linked binaries in a more complicated form I have found that to be true in at least one case—I had built a custom DNS server in Go (statically linked), and originally planned to run it in a container, but on further reflection realized the container brought no added value, and it was much simpler to write a systemd service control script than to bring in the extra baggage of a container ecosystem to run the DNS server.
  - ric2b 5y ago
    That's a very basic example. Let's say your program also depends on ffmpeg to convert some images, psql to interact with a database and a few other non-library dependencies. With containers you can trivially ensure those are always present and with the correct versions. Plus containers do give you some security benefits when compared to running natively.

unethical_ban 5y ago

Container tech can be used for small scale "pet" deployment, but my understanding is that the true benefit of containers come with seeing them as "cattle". You should never login to the shell of a container for config. All application state lives elsewhere, and any new commit to your app triggers a new build. If that's not for you, then while containers like proxmox/LXC can still be handy, you're just doing VM at a different layer. The article was a bit hand wavey about how "they" complain about containers, and then uses the analogy more than explaining the problems and solutions.

deleted 5y ago
[deleted]
Sparkyte 5y ago
Only time it should be utilized as a small scale "pet" is when you externalize the storage mounts and it is an on-demand non-persistent virtual environment not directly connected to a data sensitive environment. That's mainly my take on it. I will often use docker locally to test out some kubernetes service connectivity, but never bring my Frankenstein stuff into an environment any higher than local.
foobar33333 5y ago
>You should never login to the shell of a container for config I absolutely do this and think it works great. Fedora has built a tool called "toolbox" which is basically a wrapper on podman which can create and enter containers where you can install development tools without touching your actual OS. I basically do all of my development inside a container where the source code is volume mounted in but git/ruby/etc only exist in the container. This has the benefit of letting me very quickly create a fresh env to test something out. Recently I wanted to try doing a git pull using project access tokens on gitlab and containers let me have a copy of git which does not have my ssh keys in it. This is somewhat of an edge case though, for a server deployment, yes you shouldn't rely on anything changed inside the container and should use volume mounts or modify the image.

smitty1e 5y ago

I explain that if the an Amazon Virtual Private Cloud (VPC) is a datacenter "cloud", then a container implementation is a "puff". Virtualizing the kernel like the Amazon Machine Image (AMI) virtualizes a chip core sounds great. But now, in the "puff", all of those networking details that AWS keeps below the hypervizor line confront us. Storage, load balancing, name services, firewalls. . . Containers can solve packaging issues, but wind up only relocating a vast swath of other problems.

wmf 5y ago
If you have few enough containers you can give each one an ENI.
zaphirplane 5y ago
I have to say inventing a new metaphor for container made it harder to follow the point you were trying to make

lamontcg 5y ago

containers are fat RPMs that construct a chroot jail.

eVeechu7 5y ago

>Finally, there’s the whole business of resource isolation. While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions. I don't think virtualization really offers hardware-level guarantees against noisy neighbours either.

habeebtc 5y ago
It offers the opportunity to throttle noisy neighbors in hopes the party isn't too big.
- dilyevsky 5y ago
  Cgroups can do the same via cfs
amarshall 5y ago
VMs provide stronger guarantees for maximum CPU, network, and disk usage, as well as memory size consumption. This because the abstraction boundaries are fairly clear (e.g. threads and virtual devices).

Helmut10001 5y ago

I place all my tents in a house (Docker VMs inside unprivileged LXC containers on Proxmox - yes, unprivileged = not a brick house, more like wood). The only reason I use Docker is that I can access the system design knowledge that is available with docker-compose.yml's. Last example: Gitlab. Could not get it running on unrivileged LXC using the official installation instructions, with Docker it was simply editing the `.env` and then `docker-compose up -d`. All of this in a local, non-public (ipsec-distributed) network. I often find myself creating a single, separate unprivileged LXC container->Docker nesting for each new container, because I do not need to follow the complicated and error prone installation instructions for native installs.

Sparkyte 5y ago

I was totally expecting this to go in the direction about tech debt with a homeless analogy, but it was about destructability. Yes we know this already and if you catch people treating it as a persistent host, slap their hands and say no.

rossmohax 5y ago

I believe, that success of containers is not because of lightweightness or other isolation properties of them. Containers won dev mindshare because of ease packaging and distribution of the artifacts. Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.

enw 5y ago
> glorified tarballs Calling container images glorified tarballs is like calling cars glorified lawnmowers.
- kjeetgill 5y ago
  Maybe closer to calling a 16-wheeler a glorified minivan. I think there's something meaningful in that comparison.
- brazzy 5y ago
  Um...a container image is very literally and exactly a collection of tarballs and some JSON files with metadata.
aequitas 5y ago
> Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up. Packaged VM's existed for a while already with thing like Vagrant on top, there was also already LXC which leaned more into the VM concept. Where Docker made the difference imho is with Dockerfiles and the layered/cached build steps.
throwaway894345 5y ago
> Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up. IMO the important, catalyzing difference is that Docker containers have a standard interface for logging, monitoring, process management, etc which allow us to just think in terms of “the app” rather than the app plus the SSH daemon, log exfiltration, host metrics daemon, etc. In other words, Docker got the abstraction right: I only care about the app, not all of the ceremony required to run my app in a VM. These common interfaces allow orchestration tools to provide more value: they aren’t just scheduling VMs, they’re also managing your log exfiltration, your process management, your SSH connection, your metrics, etc, and all of those things are configurable in the same declarative format rather than configuring them with some fragile Ansible playbook that requires you to understand each of the daemons it is configuring, possibly including their unique configuration file/filesystem conventions and syntaxes.

jb_gericke 5y ago

Enjoyed the article but having watched containerization and kubernetes maturing over the last 5 years (especially at an enterprise level), I'd say a huge part of the value proposition is (and this applies more to K8) it really catalyses prototyping/experimenting and (depending on the org I suppose) promotes a lot of autonomy for app teams who'd historically have to log calls to infrastructure to get compute, network/lb/dns, databases et al. built up before kicking the tyres on something. I've seen those types of things take months in large orgs. And then there's the inevitable drift between tiered environments that happens over time in richer operating environments (I've seen VMs so laden with superfluous monitoring and agentware they fall over all the time, while simultaneously being on completely different OS and patch versions from dev to prod). Containers provide immutability at the service layer, so I have confidence in at least having that level of parity between dev and prod (albeit hardly ever at a data or network layer).

znpy 5y ago

> We don’t expect tents to serve the same purpose as brick-and-mortar houses—so why do we expect containers to function like VMs? Marketing. Because of Marketing.

deleted 5y ago

[deleted]

jsiepkes 5y ago

Should be noted that a portion of this (valid) criticism applies specifically to the most prominent "container" implementation; Docker. Not containers as a whole. For example resources isolation with the Solaris / Illumos container implementation (zones) works just as well as full blown virtualization. You are just as well equipped to handle noisy neighbors with zones as you are with hardware VM's. > Much as you’d likely choose to live in a two-bedroom townhouse over a tent, if what you need is a lightweight operating system, containers aren’t your best option. So I think this is true for Docker but doesn't really do justice to other container implementations such as FreeBSD jails and Solaris / Illumos zones. Because those containers are really just lightweight operating systems. In the end Docker started out and was designed to be a deployment tool. Not necessarily an isolation tool in all aspects. And yeah, it shows.

belter 5y ago
I can not agree more. It is the saddest thing the appalling implementation of Docker, and the whole lack of security around the ecosystem, made people think Containers equal to Docker. Docker is what happens when you put your security implementation in the hands of your Developer team and not in the hands of your DevSecOps people.
nyx__ 5y ago
Criticism applies to all Linux containers, not just Docker, which is one implementation of Linux containers. One could argue that zones are distinct from containers (a Linux implementation), with both being OS specific versions of jails.

eyeyeyerg 5y ago

containers are cattle, VMs were pets. If one does not get the operational differences nor understands that these are completely two different usescases then probably should not work in IT industry

detaro 5y ago
VMs can be cattle. Physical machines can be cattle. This is not tied to the runtime technology, but to how you design and manage your machines and applications.
- cratermoon 5y ago
  The original pet v cattle metaphor was indeed inspired by servers, not containers. http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-... What makes a given component – server, vm, container, whatever – is not the runtime, but how you deal with it when it gets seriously ill. Pets are taken to the vet or hospital to get treatment. Cattle are.. well, read the article I linked.

johbjo 5y ago

I'd be curious to see services designed to run as PID 1 inside containers, and contain or run nothing else other than the required binaries. Maybe someone is doing this.

onei 5y ago
Pretty sure this is what tini[1] is for, and there's supervisord[2] for more complex use cases, e.g. running multiple processes. Neither are a full replacement for traditional systemd, which I heard you can technically run in a container these days, but I've never seen anyone try. [1] https://github.com/krallin/tini https://github.com/krallin/tini [2] https://github.com/Supervisor/supervisor https://github.com/Supervisor/supervisor
antonvs 5y ago
It's pretty common for languages that compile to binaries. Golang, for example. Just inherit your container "FROM scratch". I have a number of service containers that are around 8 - 12MB, which contain compiled Haskell binaries.
nijave 5y ago
Related, https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless

nickjj 5y ago

IMO comparing containers to an apartment is more accurate than a tent. Because with an apartment each tenant gets to share certain infrastructure like heating and plumbing from the apartment building, just like containers get to the share things from the Linux host they run on. In the end both houses and apartments protect you from outside guests, just in their own way. I went into this analogy in my Dive into Docker course. Here's a video link to this exact point: https://youtu.be/TvnZTi_gaNc?t=427 https://youtu.be/TvnZTi_gaNc?t=427, that video was recorded back in 2017 but it still applies today.

leephillips 5y ago

I’ve found systemd-nspawn useful. Use debootstrap to install a minimal Debian system inside your system, then boot it with this command. It isolates the filesystem while sharing the network interface, and is convenient for most things that I guess people use Docker for. I wonder why it’s not mentioned more often.

dijit 5y ago
I know it sounds like I want to be spoonfed, but do you have a walkthrough of this flow? I'd be interested in trying it out but I don't want to spend some hours reading documentation trying to get it working.
- leephillips 5y ago
  Well, that’s OK, because it did take me a while to track down the pieces of the documentation and find a procedure that worked for me. There is some less-than-optimal advice out there about this. Become root. Install debootstrap, which is in the Debian and Ubuntu repositories, at least. Make a directory to contain your embedded system. It can be anywhere. Let's use /var/lib/machine/machinename. This command will install a new, minimal Debian system in that directory: debootstrap --include=systemd-container stable /var/lib/machines/machinename http://deb.debian.org/debian http://deb.debian.org/debian It will download everything and, if I recall correctly, works unattended (doesn’t ask questions). Enter the container with systemd-nspawn -D /var/lib/machines/machinename/ and set the root container password with passwd. Then do echo 'pts/0' >> /etc/securetty so the guest OS will let you log in after it's booted up. You may have to add other pts/x entries. I'm not sure about this part; it may be that if there is no /etc/securetty file that there is no problem. Now log out of the container. To boot up the guest OS, use systemd-nspawn -b -D /var/lib/machines/machinename You will see the familiar console messages. You will find advice on the web to include the -U flag here, which causes files in the guest OS to only use UIDs known to the guest OS when determining ownership and permissions. This leads to headaches, because you have to set the ownership of any file you copy in from the host system. Leave it out, and you can have parallel users on the host and guest OSs, which is more convenient. But you may have to change the UIDs of the users on the guest OS so that they match. Now, on the host OS, you can use the `machinectl` command to control all your guest OSs. `machinectl list` shows you what’s running, `machinectl login` lets you log in to them, and there are several commands for killing them with various levels of violence. If you want your machine to be a long-running service, just `nohup` the spawn command, and direct output as desired. If you want to be able to communicate with your machine from the internet, opening sockets from within the guest OS works, as they share the network interface. For a public-facing web service, you can install (for example) Apache and pick a port number to listen on, then set up a reverse proxy on the host OS, using a dedicated domain or a subdomain, so the users don’t have to use the custom port number. I’ve found that certificates for HTTPS need to be installed on both the host and guest OSs. Good luck! More information: https://wiki.debian.org/nspawn https://wiki.debian.org/nspawn

1MachineElf 5y ago

You had me at containers

luord 5y ago

I've seen the problems of treating containers as houses, primarily during development: Multiple different processes inside a single container, with a wrapper around them (inside the container) that makes it even more difficult to debug. So, assuming I understood correctly, treating them like tents is infinitely the better choice.

0xcmoney 5y ago

Am I the only one getting tired of people stating confidently that containers don't improve safety _at all_ because they run on the same kernel? It's just not true.

nijave 5y ago

Modern containers do provide lots of security features with namespaces, seccomp, cgroups (to some extent) The author seems to largely ignore this. I would consider that a bit stronger than a "tent wall". Comparing it to a tent seems more akin to a plain chroot. If I have my tent right next to someone else, I can trivially "IPC" just speaking out loud which would be prevented by an IPC namespace (which is Docker's current default container setup) Also worth mentioning, turning a container into a VM (for enhanced security) is generally easier than trying to do the opposite. AWS Lambda basically does that as do a lot of the minimal "cloud" Linux distributions that just run Docker with a stripped down userland (like Container Linux and whatever its successors are)

throwaway894345 5y ago
I’m a big proponent of containers, but in fairness to TFA, I don’t know how to configure namespaces, second, or cgroups and I don’t know what settings my orchestrator uses by default. If containers can be secure but we don’t enable those security features properly, then it’s a bit of a moot point. That said, I think (but am not sure) most of us understand enough not to trust containers for isolation between untrusted processes, so I don’t regard containers as lightweight VMs, but rather collocated processes with their own namespaces. When I run untrusted code, like jupyterhub, I make sure those untrusted containers get scheduled onto their own dedicated mode pool with single tenancy (at which point the container is more of a tooling/orchestration convenience than a resource optimization tool).

fsociety 5y ago

This is oversimplifying containers and VM by using the house vs tent analogy. Just talking about Docker weakens this too, because Docker is not the only way to setup containers. > Tents, after all, aren’t a particularly secure place to store your valuables. Your valuables in a tent in your living room, however? Pretty secure. Containers do provide strong security features, and sometimes the compromises you have to make hosting something on a VM vs. a container will make the container more secure. > While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions. Cgroups are more than a neat resource isolation mechanism, they work. That's really all there is to it. Paranoia around trusting the Linux kernel is unnecessary if at the end of the day you end up running Linux in production. If anything breaks, security patches will come quick and the general security attitude of the Linux community is improving everyday. If you are really paranoid, perhaps run BSD, use grsec, or the best choice is to use SELinux IMO. If anything, you will be pwned because you have a service open to the world, not because cgroups or containers let you down.

deleted 5y ago
[deleted]

deleted 5y ago

[deleted]

srg0 5y ago

If VM is like a nuclear war bunker, containers are like brick and mortar houses. They are not air tight and have glass windows which can be easily broken, but that's where most people live. They're cheaper to build, comfortable enough, and can last a human lifetime most of the time. An analogy can go a long way. Both ways.