4 ms·

Why not spin up containers instead of VMs? Seems to me containers would fit much better than VMs.

by BossingAround 4y ago

cogman10 4y ago
Bypassing container security is easier than bypassing VM security.
- tamrix 4y ago
  Then wouldn't that be the ultimate test ;)
spiffytech 4y ago
Containers have a history of escape vulnerabilities, for reasons like sharing a kernel with the host and other containers. VMs are designed from the ground up to isolate guests, rather than focusing on application deployment. Firecracker is the modern container alternative in untrusted compute scenarios, with Fly.io even converting container images into Firecracker VMs.
- NovemberWhiskey 4y ago
  >Containers have a history of escape vulnerabilities Generally agreed, but for this use-case do we care?
  - nneonneo 4y ago
    I haven’t gotten any of the challenges to load, but if you’re going to simulate a sysadmin it would make sense to give you high privileges (or even root) on the box. The more privileged you are inside a container, the more attack surface you expose.
    - BossingAround 4y ago
      Which is why you create a "dummy" host VM that hosts containers. Nobody's saying "host containers on your prod webserver." On the other hand, spinning up a VM for every user seems insane to me.
- encryptluks2 4y ago
  User mapping is now a standard feature in Kubernetes, so escape vulnerabilities aren't so much an issue anymore. Additionally, you can use gVisor.
  - paulfurtado 4y ago
    User namespaces have resulted in multiple new container breakout CVEs in the last year. Some guides actually recommend disabling user namespaces because they are still somewhat new and perilous.
    - encryptluks2 4y ago
      What are the ones in the last year? They provide security benefits as well. I mean, you could say the Linux kernel is also dangerous and the Windows kernel and pretty much anything that has ever had a CVE. You can also limit it to specific users too if that is a major concern.
    - cyphar 4y ago
      You're talking about creating new user namespaces inside a container, not running a container in a user namespace. Running a container in a user namespace is strictly a security improvement over running it in the host user namespace. Also, all container runtimes automatically block unshare(CLONE_NEWUSER) with seccomp already (unless they've disabled seccomp, which I'm not sure if Kubernetes still does).
paulfurtado 4y ago
If the goal of the test is to debug a sad linux server, containers are going to severely limit what ways the server can be sad in, isn't it?
- BossingAround 4y ago
  Can you give me an example of some of the severe limitations you're mentioning?
  - nijave 4y ago
    I don't think the DNS exercise would behave the same although that probably depends on how the container was setup. Docker usually controls /etc/resolv.conf. Another exercise is "try to figure out if you're in a container or VM so that'd definitely be different"
    - BossingAround 4y ago
      The question is not if the exercises would behave identically, but if you can test the objective in a container. For example, you can totally test, screw up, and fix DNS in a container. I would think that "try to figure out if you're in a container or VM" would be exactly the same as it is right now.
  - mmh0000 4y ago
    I can give you a bunch of things that can't be simulated in a container: * Boot problems, such as: GRUB config/install errors, kernel parameters, init startup errors, blocking processes * Many network scenarios, such as: PXE issues, multipath, load-balacing, anything requiring configuring network interface settings, firewall configuration. * Resetting an unknown root password * Booting directly to bash * Filesystem mounts through fstab or systemd mounts There's probably more I could think of, but I think that's a good list.

Why not spin up containers instead of VMs? Seems to me containers would fit much better than VMs.

cogman10 4y ago

Bypassing container security is easier than bypassing VM security.

tamrix 4y ago
Then wouldn't that be the ultimate test ;)

spiffytech 4y ago

Containers have a history of escape vulnerabilities, for reasons like sharing a kernel with the host and other containers. VMs are designed from the ground up to isolate guests, rather than focusing on application deployment. Firecracker is the modern container alternative in untrusted compute scenarios, with Fly.io even converting container images into Firecracker VMs.

NovemberWhiskey 4y ago
>Containers have a history of escape vulnerabilities Generally agreed, but for this use-case do we care?
- nneonneo 4y ago
  I haven’t gotten any of the challenges to load, but if you’re going to simulate a sysadmin it would make sense to give you high privileges (or even root) on the box. The more privileged you are inside a container, the more attack surface you expose.
  - BossingAround 4y ago
    Which is why you create a "dummy" host VM that hosts containers. Nobody's saying "host containers on your prod webserver." On the other hand, spinning up a VM for every user seems insane to me.
encryptluks2 4y ago
User mapping is now a standard feature in Kubernetes, so escape vulnerabilities aren't so much an issue anymore. Additionally, you can use gVisor.
- paulfurtado 4y ago
  User namespaces have resulted in multiple new container breakout CVEs in the last year. Some guides actually recommend disabling user namespaces because they are still somewhat new and perilous.
  - encryptluks2 4y ago
    What are the ones in the last year? They provide security benefits as well. I mean, you could say the Linux kernel is also dangerous and the Windows kernel and pretty much anything that has ever had a CVE. You can also limit it to specific users too if that is a major concern.
  - cyphar 4y ago
    You're talking about creating new user namespaces inside a container, not running a container in a user namespace. Running a container in a user namespace is strictly a security improvement over running it in the host user namespace. Also, all container runtimes automatically block unshare(CLONE_NEWUSER) with seccomp already (unless they've disabled seccomp, which I'm not sure if Kubernetes still does).

paulfurtado 4y ago

If the goal of the test is to debug a sad linux server, containers are going to severely limit what ways the server can be sad in, isn't it?

BossingAround 4y ago
Can you give me an example of some of the severe limitations you're mentioning?
- nijave 4y ago
  I don't think the DNS exercise would behave the same although that probably depends on how the container was setup. Docker usually controls /etc/resolv.conf. Another exercise is "try to figure out if you're in a container or VM so that'd definitely be different"
  - BossingAround 4y ago
    The question is not if the exercises would behave identically, but if you can test the objective in a container. For example, you can totally test, screw up, and fix DNS in a container. I would think that "try to figure out if you're in a container or VM" would be exactly the same as it is right now.
- mmh0000 4y ago
  I can give you a bunch of things that can't be simulated in a container: * Boot problems, such as: GRUB config/install errors, kernel parameters, init startup errors, blocking processes * Many network scenarios, such as: PXE issues, multipath, load-balacing, anything requiring configuring network interface settings, firewall configuration. * Resetting an unknown root password * Booting directly to bash * Filesystem mounts through fstab or systemd mounts There's probably more I could think of, but I think that's a good list.