4 ms·

> turns Docker images into running VMs I honestly don't understand what's going on here. I thought we turned to Docker/containers because VMs were too heavy? N

> turns Docker images into running VMs I honestly don't understand what's going on here. I thought we turned to Docker/containers because VMs were too heavy? Now we've got VMs that run Docker? (Not trying to be dense - what is the advantage?)

wmf 4y ago
Everybody hypnotized themselves into believing that containers are not secure and can never be made secure so they run one container per VM. Instead of investing in making containers secure, the industry decided to invest in making VMs ligher, so VMs are now efficient enough that you can run one container per VM. Why run Docker in VMs instead of using VM images? Because Docker's build tools are more popular than Packer-style tooling.
- viraptor 4y ago
  > hypnotized themselves into believing that containers are not secure They provide any extra layer of indirection which helps with usual exploit attempts, but also introduce new scope. We've had exploits specifically targeting the namespaces API already.
  - solarkraft 4y ago
    > We've had exploits specifically targeting the namespaces API already Well, isn't that what happens when you put a shield into place? Someone tries to break it. Why have people concluded that it can never be made properly secure?
    - tptacek 4y ago
      Because the broad kernel attack surface is huge, and the shield has to reliably protect all of it, or all you've done is create a jungle gym for vulnerability researchers. The win with virtualization is that it drastically scopes down the amount of kernel code exposed to untrusted code.
- tptacek 4y ago
  I don't know if hypnosis works or not, but quitting smoking is good whether or not you hypnotize yourself to do it, and so is avoiding multi-tenant Docker. The broad kernel attack surface is much too scary to expose directly to multi-tenant workloads, and there have been fairly recent kernel LPEs that would have avoided any sane system call filter you could come up with. It's a moot point, because this is a solved problem. Use containers for single-tenant workloads; use micro-VMs, whichever flavor you like best, for multi-tenant.
  - zachthewf 4y ago
    What is the status of hardware acceleration for micro-VMs? I know that Firecracker doesn't have GPU support yet, are there any other options that handle this?
    - tptacek 4y ago
      It's a good question. GPUs are in some ways a difficult case for the Firecracker model, which prizes a minimal, mostly memory-safe attack surface that (critically) is easy to reason about. We'd very much like to get an instance or machine type that supports GPUs, but we perceive it as a Big Project. We might not even use Firecracker to do it when we finally get it rolling. If you're reading this and have big thoughts on how we might do GPUs at Fly.io without keeping us up at night about security, you should reach out; we're hiring.
- foobiekr 4y ago
  The issue with container security is that the linux kernel boundary is porous and probably not possible to secure in depth.
  - freedomben 4y ago
    I don't necessarily disagree with "the linux kernel boundary is porous and probably not possible to secure in depth", but IIUC the hypervisor is part of the kernel too, so wouldn't it have the same problem? The win would be in the attack surface area. For hypervisors there's a good layer of abstraction to pivot over, whereas with containers it's a much thinner wall.
    - tptacek 4y ago
      KVM is a much smaller chunk of the kernel than the entire system call interface, along with every exposed device and its suite of ioctls.
    - foobiekr 4y ago
      KVM presents a very narrow, well understood surface that is much much less of a moving target and changes at a much slower rate. Qemu has traditionally been a problem which is why Amazon and google have moved away from it. The container kernel surface is just insane by comparison.
- zokier 4y ago
  Not everybody. Google notably uses gvisor instead of vms
  - ithkuil 4y ago
    Which is an alternative approach to providing a dedicated kernel to the VM/container. (Because that's what basically a hypervisor does). Gvisor effectively implements a Linux kernel in user space, written in a memory safe language. A kernel that insulates all system calls from the host kernel, by literally implementing wrappers around host system calls.
  - yencabulator 4y ago
    As far as I can read between the lines, Google has a non-open-source backend for gVisor, and it really wouldn't surprise me if that was KVM too.
- tremon 4y ago
  Everybody hypnotized themselves into believing that daemons are not secure and can never be made secure so they run one daemon per container. Instead of investing in making daemons secure, the industry decided to invest in making daemons heavier, so each daemon can now be provided with its own hardware/OS abstraction layer. Why run daemons in containers instead of using proper process isolation? Because containers absolve the system administrator from understanding their systems.
rstupek 4y ago
Isolation would be the biggest advantage so they can host multiple clients on the same machine. Fly uses a lightweight vm (someone chime in if they have better details).
- tptacek 4y ago
  https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/
zrail 4y ago
Firecracker is a thin layer on top of KVM. It essentially implements just a handful of devices and it boots in milliseconds. Fly bakes a Docker image into the format that Firecracker expects and then boots it, alongside a bunch of anycast networking magic. You get the security guarantees of KVM with the developer experience of docker.
ignoramous 4y ago
> thought we turned to Docker/containers because VMs were too heavy? VMs are lightweight now, though containers are lighter still; ref: https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ Btw, Docker wasn't about security as much as it was about "package once, run anywhere". > Now we've got VMs that run Docker? Fly.io doesn't run Docker as-is, but rather unpacks it and runs it in a guest through containerd; ref: https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/ > ...what is the advantage? This has been discussed numerous times, and here's a link to one such discussion: https://news.ycombinator.com/item?id=26747701 https://news.ycombinator.com/item?id=26747701 Read also: https://gruchalski.com/posts/2021-03-03-thoughts-on-creating-vmms-from-docker-images/ https://gruchalski.com/posts/2021-03-03-thoughts-on-creating...
viraptor 4y ago
Not quite mentioned in other answers, but historically, the VMs we used to run were heavy. A typical qemu-kvm machine needs to actually boot up, initialise, start a number of services, etc. Firecracker is not that - it essentially gives you a kernel that already knows the environment and can do bare minimum before executing the provided image. It's like a halfway point between unikernels and independent VMs. The VM technology itself is not necessarily heavy - it just depends how you use it.
qbasic_forever 4y ago
IMHO with fly.io their use of containers is more for the dev experience. It's incredibly easy and popular to whip up a Dockerfile, test it locally, ship it to a registry, etc. Anyone can learn the Dockerfile syntax and be productive with it in an afternoon. The tooling for proper VM creation on the other hand is in the stone-age comparatively--there are just a few tools like packer or a frankenstein of ansible scripts and neither are as nice or easy as Dockerfile creation.
- regularfry 4y ago
  Before Docker there was Vagrant. Still works, last I looked.
  - qbasic_forever 4y ago
    It's a frankenstein system of provisioning though. You have to write a config file in Ruby and embed bash scripts, which may or may not do things like invoke ansible scripts (written in YAML). The layers of complexity are immense, and that's before you have to start janitoring virtualbox's always half broken state. The inner loop of change vagrant config -> see result in VM is painfully slow too, waiting for the VM to tear down and setup again. Docker is much, much simpler. Write a Dockerfile that's mostly just bash or shell code. Build and run in seconds to immediately see the results. Once it's working push the container image to a public registry and you can distribute it to anything.
dilyevsky 4y ago
Containers are still more lightweight (tho not by a lot these days) but they are hella insecure for untrusted workloads. Plus people like and depend on docker workflows hence taking docker container (basically just a tarball+json manifest) and making a VM out of it

wmf 4y ago

Everybody hypnotized themselves into believing that containers are not secure and can never be made secure so they run one container per VM. Instead of investing in making containers secure, the industry decided to invest in making VMs ligher, so VMs are now efficient enough that you can run one container per VM. Why run Docker in VMs instead of using VM images? Because Docker's build tools are more popular than Packer-style tooling.

viraptor 4y ago
> hypnotized themselves into believing that containers are not secure They provide any extra layer of indirection which helps with usual exploit attempts, but also introduce new scope. We've had exploits specifically targeting the namespaces API already.
- solarkraft 4y ago
  > We've had exploits specifically targeting the namespaces API already Well, isn't that what happens when you put a shield into place? Someone tries to break it. Why have people concluded that it can never be made properly secure?
  - tptacek 4y ago
    Because the broad kernel attack surface is huge, and the shield has to reliably protect all of it, or all you've done is create a jungle gym for vulnerability researchers. The win with virtualization is that it drastically scopes down the amount of kernel code exposed to untrusted code.
tptacek 4y ago
I don't know if hypnosis works or not, but quitting smoking is good whether or not you hypnotize yourself to do it, and so is avoiding multi-tenant Docker. The broad kernel attack surface is much too scary to expose directly to multi-tenant workloads, and there have been fairly recent kernel LPEs that would have avoided any sane system call filter you could come up with. It's a moot point, because this is a solved problem. Use containers for single-tenant workloads; use micro-VMs, whichever flavor you like best, for multi-tenant.
- zachthewf 4y ago
  What is the status of hardware acceleration for micro-VMs? I know that Firecracker doesn't have GPU support yet, are there any other options that handle this?
  - tptacek 4y ago
    It's a good question. GPUs are in some ways a difficult case for the Firecracker model, which prizes a minimal, mostly memory-safe attack surface that (critically) is easy to reason about. We'd very much like to get an instance or machine type that supports GPUs, but we perceive it as a Big Project. We might not even use Firecracker to do it when we finally get it rolling. If you're reading this and have big thoughts on how we might do GPUs at Fly.io without keeping us up at night about security, you should reach out; we're hiring.
foobiekr 4y ago
The issue with container security is that the linux kernel boundary is porous and probably not possible to secure in depth.
- freedomben 4y ago
  I don't necessarily disagree with "the linux kernel boundary is porous and probably not possible to secure in depth", but IIUC the hypervisor is part of the kernel too, so wouldn't it have the same problem? The win would be in the attack surface area. For hypervisors there's a good layer of abstraction to pivot over, whereas with containers it's a much thinner wall.
  - tptacek 4y ago
    KVM is a much smaller chunk of the kernel than the entire system call interface, along with every exposed device and its suite of ioctls.
  - foobiekr 4y ago
    KVM presents a very narrow, well understood surface that is much much less of a moving target and changes at a much slower rate. Qemu has traditionally been a problem which is why Amazon and google have moved away from it. The container kernel surface is just insane by comparison.
zokier 4y ago
Not everybody. Google notably uses gvisor instead of vms
- ithkuil 4y ago
  Which is an alternative approach to providing a dedicated kernel to the VM/container. (Because that's what basically a hypervisor does). Gvisor effectively implements a Linux kernel in user space, written in a memory safe language. A kernel that insulates all system calls from the host kernel, by literally implementing wrappers around host system calls.
- yencabulator 4y ago
  As far as I can read between the lines, Google has a non-open-source backend for gVisor, and it really wouldn't surprise me if that was KVM too.
tremon 4y ago
Everybody hypnotized themselves into believing that daemons are not secure and can never be made secure so they run one daemon per container. Instead of investing in making daemons secure, the industry decided to invest in making daemons heavier, so each daemon can now be provided with its own hardware/OS abstraction layer. Why run daemons in containers instead of using proper process isolation? Because containers absolve the system administrator from understanding their systems.

rstupek 4y ago

Isolation would be the biggest advantage so they can host multiple clients on the same machine. Fly uses a lightweight vm (someone chime in if they have better details).

tptacek 4y ago
https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/

zrail 4y ago

Firecracker is a thin layer on top of KVM. It essentially implements just a handful of devices and it boots in milliseconds. Fly bakes a Docker image into the format that Firecracker expects and then boots it, alongside a bunch of anycast networking magic. You get the security guarantees of KVM with the developer experience of docker.

ignoramous 4y ago

> thought we turned to Docker/containers because VMs were too heavy? VMs are lightweight now, though containers are lighter still; ref: https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ Btw, Docker wasn't about security as much as it was about "package once, run anywhere". > Now we've got VMs that run Docker? Fly.io doesn't run Docker as-is, but rather unpacks it and runs it in a guest through containerd; ref: https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/ > ...what is the advantage? This has been discussed numerous times, and here's a link to one such discussion: https://news.ycombinator.com/item?id=26747701 https://news.ycombinator.com/item?id=26747701 Read also: https://gruchalski.com/posts/2021-03-03-thoughts-on-creating-vmms-from-docker-images/ https://gruchalski.com/posts/2021-03-03-thoughts-on-creating...

viraptor 4y ago

Not quite mentioned in other answers, but historically, the VMs we used to run were heavy. A typical qemu-kvm machine needs to actually boot up, initialise, start a number of services, etc. Firecracker is not that - it essentially gives you a kernel that already knows the environment and can do bare minimum before executing the provided image. It's like a halfway point between unikernels and independent VMs. The VM technology itself is not necessarily heavy - it just depends how you use it.

qbasic_forever 4y ago

IMHO with fly.io their use of containers is more for the dev experience. It's incredibly easy and popular to whip up a Dockerfile, test it locally, ship it to a registry, etc. Anyone can learn the Dockerfile syntax and be productive with it in an afternoon. The tooling for proper VM creation on the other hand is in the stone-age comparatively--there are just a few tools like packer or a frankenstein of ansible scripts and neither are as nice or easy as Dockerfile creation.

regularfry 4y ago
Before Docker there was Vagrant. Still works, last I looked.
- qbasic_forever 4y ago
  It's a frankenstein system of provisioning though. You have to write a config file in Ruby and embed bash scripts, which may or may not do things like invoke ansible scripts (written in YAML). The layers of complexity are immense, and that's before you have to start janitoring virtualbox's always half broken state. The inner loop of change vagrant config -> see result in VM is painfully slow too, waiting for the VM to tear down and setup again. Docker is much, much simpler. Write a Dockerfile that's mostly just bash or shell code. Build and run in seconds to immediately see the results. Once it's working push the container image to a public registry and you can distribute it to anything.

dilyevsky 4y ago

Containers are still more lightweight (tho not by a lot these days) but they are hella insecure for untrusted workloads. Plus people like and depend on docker workflows hence taking docker container (basically just a tarball+json manifest) and making a VM out of it