4 ms·

Everyone is praising how well written this is, but I personally am confused: what do you actually GAIN by running a docker container without docker? Is it faste

by querez 6y ago

Everyone is praising how well written this is, but I personally am confused: what do you actually GAIN by running a docker container without docker? Is it faster, or more secure? Why?

tptacek 6y ago
It is comparably fast (in practical terms; "of the same general order of startup cost") and much more secure. I didn't want to belabor the security thing, because we wrote extensively about that in a previous (linked) blog post. https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ You can skip to the grafs immediately before and after the string "If you’re running someone else’s applications, you should probably care a lot". I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems (ie: using directly shared kernels), for reasons that post gets into. It's for the same reason that AWS wrote Firecracker to run Fargate, which is also a container hosting service. It's my post and I don't totally get the writing thing either. My M.O. with these posts: write it like it was an HN comment, and then edit the sentences to be shorter. I'm glad people like it, though.
- deepstack 6y ago
  > AWS wrote Firecracker to run Fargate thank you for mention that. Which projects like Firecracker, it really doesn't make sense to have Docker.
  - krageon 6y ago
    Given that this directly precedes the sentence you rewrote: > I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems I'm really not sure what point you're trying to make
    - dsr_ 6y ago
      I interpreted this as: "don't think you can reasonably host" => I encourage our competitors to do this "general-purpose applications on a multi-tenant basis" => any program a black hat who signed up with our automated system wants to run, next to all the other programs random people want to run "on shared hardware" => one lump of iron owned by Fly "using container systems" => containers vs VMs, in this case, and especially a VM manager that pays some attention to security.
    - acdha 6y ago
      I think the important part is immediately after what you quoted: “using directly shared kernels” If you’re running dockerd on one big server, you’re a single bug in either the kernel or dockerd away from someone in account A being able to attack account B. This is also true about Firecracker or another VM approach but look at the relative exposed attack surface: if we’re on the same kernel, there are tons of different modules which I can attack — maybe even obscure ones you don’t or barely use - and if any of them allow me to run code, alter or leak memory, etc. mayhem ensues. Running a VM reduces that risk because you’re sharing much less code & data but they traditionally added more overhead and startup delay, and if you’re deploying a traditional OS there’s a lot of management overhead. You’d see this as a security precaution but many people weren’t willing to pay for it. Firecracker changes that calculation because creating a VM is fast enough and Docker means that you don’t need to support a full Linux distribution, just enough to launch the container image.
      - krageon 6y ago
        > many people weren’t willing to pay When most people don't understand the implications, offering a cheaper option that's almost guaranteed to inconvenience them (or be a catastrophe, depending on what they're doing) is not a fault that should be attributed directly (and certainly not solely) to them.
  - toyg 6y ago
    Docker nowadays is more valuable as a configuration standard "primitive" than as an actual tool.
    - fireflash38 6y ago
      It's super convenient in that it should eliminate the "runs on my box" sort of errors. You should be able to run the same OCI image in Docker, FC, and kata and not worry about that. Really opens you up to freedom of where you dev.
      - toyg 6y ago
        Well, kinda. It’s still “you can use any OS as long as it’s Linux”. If you like the BSDs, forget it.
      - patrick451 6y ago
        Sort of. Docker makes external dependencies management better, but it's an abstraction that leaks like a sieve when it comes to networking. If your /etc/docker/daemon.json file isn't set up just so, or your iptables are a bit off, or your vpn client is a bit too aggressive (e.g., Cisco VPN), the whole thing blows up. Personally, I would rather manage dependencies than iptables rules.
  - imtringued 6y ago
    This is like saying thanks to car company with X security feature, it really doesn't make sense to have competitor without security feature. Why eliminate your options? You don't gain anything.
- larrybud 6y ago
  Just curious.... couldn’t you get the same level of isolation by using a windows server as the host with HyperV isolation for the guest containers?
  - deleted 6y ago
    [deleted]
  - mrkurt 6y ago
    Yes. The advantage of Firecracker is that it's lightweight and boots incredibly fast. It's a pretty good way to run containers. But the virtualization primitives are almost the same.
- jnwatson 6y ago
  That sandboxing post is the best article I've read on isolation in a long time. It is an absolutely fantastic way to get caught up on the last decade of solutions. Well done.
- travisd 6y ago
  Any thoughts on gVisor (kernel reimplementation in Go) vs Firecracker?
  - tptacek 6y ago
    In addition to the article linked above, check this out; the gVisor comes in midway through. We like the codebase. :) https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ https://fly.io/blog/ssh-and-user-mode-ip-wireguard/
ffk 6y ago
If it's using firecracker, it's probably using KVM virtualization while ensuring that the memory the VM consumes is not pinned... that is, that the VM can be swapped out of memory. For reference, firecracker was created by AWS to run and secure AWS Lambda. The hypervisor is written in rust and uses seccomp to eliminate unnecessary system calls. They open sourced it a few years back. What you gain is a stronger security boundary. Just FYI, since 2019, you can also do this in Kubernetes using Kata containers + containerd which will happily shim firecracker. The setup is not simple though. https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support https://github.com/kata-containers/documentation/wiki/Initia... Overall, fly.io building infrastructure on this pattern and making it accessible is fantastic. Looking forward to seeing how this continues to evolve and am happy to see more infra build on top of firecracker. Very exciting!
LdSGSgvupDV 6y ago
Another point of view would be the kernel version. If your service depends on different kernel version. It might be good to have a docker-like VM which could create by Dockerfile <- this is good for CI/CD. But in general case, their target should be the security.
jillesvangurp 6y ago
Exactly what I was wondering. It's a lengthy explanation explaining how they provide a commodity service: i.e. running vanilla docker containers from vanilla docker registries) doing all sorts of clever and seemingly complicated things. But in the end what you buy is the ability to run a docker container that you publish to a docker registry that you develop and test using docker. I'm sure there is some advantage to all that cleverness but its not at all clear what that is (cost?). So, I disagree it the article is that well written. There's a lot of what and how but not a whole lot of why that matters to their customers. It kind of fails to communicate that part. It could be as simple as "you pay less for more because we did X and this is how we did it and why that matters to you". Writing is hard.
- iampims 6y ago
  > But in the end what you buy is the ability to run a docker container That’s not entirely correct. You don’t run a docker container. The filesystem is extracted from an docker image and mounted in a VM (firecracker). Big difference that requires the cleverness described in the blog post.

by querez 6y ago

Everyone is praising how well written this is, but I personally am confused: what do you actually GAIN by running a docker container without docker? Is it faster, or more secure? Why?

tptacek 6y ago

It is comparably fast (in practical terms; "of the same general order of startup cost") and much more secure. I didn't want to belabor the security thing, because we wrote extensively about that in a previous (linked) blog post. https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ You can skip to the grafs immediately before and after the string "If you’re running someone else’s applications, you should probably care a lot". I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems (ie: using directly shared kernels), for reasons that post gets into. It's for the same reason that AWS wrote Firecracker to run Fargate, which is also a container hosting service. It's my post and I don't totally get the writing thing either. My M.O. with these posts: write it like it was an HN comment, and then edit the sentences to be shorter. I'm glad people like it, though.

deepstack 6y ago
> AWS wrote Firecracker to run Fargate thank you for mention that. Which projects like Firecracker, it really doesn't make sense to have Docker.
- krageon 6y ago
  Given that this directly precedes the sentence you rewrote: > I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems I'm really not sure what point you're trying to make
  - dsr_ 6y ago
    I interpreted this as: "don't think you can reasonably host" => I encourage our competitors to do this "general-purpose applications on a multi-tenant basis" => any program a black hat who signed up with our automated system wants to run, next to all the other programs random people want to run "on shared hardware" => one lump of iron owned by Fly "using container systems" => containers vs VMs, in this case, and especially a VM manager that pays some attention to security.
  - acdha 6y ago
    I think the important part is immediately after what you quoted: “using directly shared kernels” If you’re running dockerd on one big server, you’re a single bug in either the kernel or dockerd away from someone in account A being able to attack account B. This is also true about Firecracker or another VM approach but look at the relative exposed attack surface: if we’re on the same kernel, there are tons of different modules which I can attack — maybe even obscure ones you don’t or barely use - and if any of them allow me to run code, alter or leak memory, etc. mayhem ensues. Running a VM reduces that risk because you’re sharing much less code & data but they traditionally added more overhead and startup delay, and if you’re deploying a traditional OS there’s a lot of management overhead. You’d see this as a security precaution but many people weren’t willing to pay for it. Firecracker changes that calculation because creating a VM is fast enough and Docker means that you don’t need to support a full Linux distribution, just enough to launch the container image.
    - krageon 6y ago
      > many people weren’t willing to pay When most people don't understand the implications, offering a cheaper option that's almost guaranteed to inconvenience them (or be a catastrophe, depending on what they're doing) is not a fault that should be attributed directly (and certainly not solely) to them.
- toyg 6y ago
  Docker nowadays is more valuable as a configuration standard "primitive" than as an actual tool.
  - fireflash38 6y ago
    It's super convenient in that it should eliminate the "runs on my box" sort of errors. You should be able to run the same OCI image in Docker, FC, and kata and not worry about that. Really opens you up to freedom of where you dev.
    - toyg 6y ago
      Well, kinda. It’s still “you can use any OS as long as it’s Linux”. If you like the BSDs, forget it.
    - patrick451 6y ago
      Sort of. Docker makes external dependencies management better, but it's an abstraction that leaks like a sieve when it comes to networking. If your /etc/docker/daemon.json file isn't set up just so, or your iptables are a bit off, or your vpn client is a bit too aggressive (e.g., Cisco VPN), the whole thing blows up. Personally, I would rather manage dependencies than iptables rules.
- imtringued 6y ago
  This is like saying thanks to car company with X security feature, it really doesn't make sense to have competitor without security feature. Why eliminate your options? You don't gain anything.
larrybud 6y ago
Just curious.... couldn’t you get the same level of isolation by using a windows server as the host with HyperV isolation for the guest containers?
- deleted 6y ago
  [deleted]
- mrkurt 6y ago
  Yes. The advantage of Firecracker is that it's lightweight and boots incredibly fast. It's a pretty good way to run containers. But the virtualization primitives are almost the same.
jnwatson 6y ago
That sandboxing post is the best article I've read on isolation in a long time. It is an absolutely fantastic way to get caught up on the last decade of solutions. Well done.
travisd 6y ago
Any thoughts on gVisor (kernel reimplementation in Go) vs Firecracker?
- tptacek 6y ago
  In addition to the article linked above, check this out; the gVisor comes in midway through. We like the codebase. :) https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ https://fly.io/blog/ssh-and-user-mode-ip-wireguard/

ffk 6y ago

If it's using firecracker, it's probably using KVM virtualization while ensuring that the memory the VM consumes is not pinned... that is, that the VM can be swapped out of memory. For reference, firecracker was created by AWS to run and secure AWS Lambda. The hypervisor is written in rust and uses seccomp to eliminate unnecessary system calls. They open sourced it a few years back. What you gain is a stronger security boundary. Just FYI, since 2019, you can also do this in Kubernetes using Kata containers + containerd which will happily shim firecracker. The setup is not simple though. https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support https://github.com/kata-containers/documentation/wiki/Initia... Overall, fly.io building infrastructure on this pattern and making it accessible is fantastic. Looking forward to seeing how this continues to evolve and am happy to see more infra build on top of firecracker. Very exciting!

LdSGSgvupDV 6y ago

Another point of view would be the kernel version. If your service depends on different kernel version. It might be good to have a docker-like VM which could create by Dockerfile <- this is good for CI/CD. But in general case, their target should be the security.

jillesvangurp 6y ago

Exactly what I was wondering. It's a lengthy explanation explaining how they provide a commodity service: i.e. running vanilla docker containers from vanilla docker registries) doing all sorts of clever and seemingly complicated things. But in the end what you buy is the ability to run a docker container that you publish to a docker registry that you develop and test using docker. I'm sure there is some advantage to all that cleverness but its not at all clear what that is (cost?). So, I disagree it the article is that well written. There's a lot of what and how but not a whole lot of why that matters to their customers. It kind of fails to communicate that part. It could be as simple as "you pay less for more because we did X and this is how we did it and why that matters to you". Writing is hard.

iampims 6y ago
> But in the end what you buy is the ability to run a docker container That’s not entirely correct. You don’t run a docker container. The filesystem is extracted from an docker image and mounted in a VM (firecracker). Big difference that requires the cleverness described in the blog post.