8 ms·

Kata Containers: Virtual Machines that feel and perform like containers

146 points by flanked-evergl 3y ago

bionsystem 3y ago
I don't understand how can they be as fast as regular containers if they run an entire kernel on top of an hypervisor ?
- Alifatisk 3y ago
  Yeah I'd like to see some numbers on that, like startup time.
  - theaiquestion 3y ago
    Depending on your use case there's potentially negligible startup time. On the scale of single digit seconds to less then half a second depending on how much work you put into optimizing it. For some applications this will be too slow (mainly the type where you boot a container per request, although flyio seems to make it work), I think for a _lot_ of applications this wouldn't be noticed. Kata gives you a few different options for what/how you'd like to boot including firecracker. This isn't exclusive to firecracker but if you stay lightweight you can have vm's booting under a half second if you're using slim images. https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... I honestly think for a lot of people, vm's with the convenience/orchestration tools of containers make more sense for a lot of general use cases simply because of the security benefits. The convenience still needs some work though.
    - insanitybit 3y ago
      Unless you're dealing with a multi-tenant situation I'm not super convinced that a VM is worth the effort. It's not the perf, it's the need to make your kernel, root file system, and other infra needed to make it all work. Compare that to a docker container where there's basically 0 additional work that has to be done to be up and running. For most cases I'd be really tempted to work on hardening the docker container than on setting up a VM. Things like Apparmor and seccomp in particular would likely go a very long way.
      - deleted 3y ago
        [deleted]
- rwmj 3y ago
  I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes). The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers. Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.
  - cogman10 3y ago
    > but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.
    - tracker1 3y ago
      I think in the longer run, WASM might displace a lot of both in practical terms.
      - ehutch79 3y ago
        Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.
- mugsie 3y ago
  there is pretty low overhead if you are opinionated - this is very similar to firecracker (AWS) tooling, so cut down hypervisor with ~ 0 devices, and a cut down guest OS means pretty quick boot times
- kobalsky 3y ago
  there's probably a big asterisk there. the correct term is probably "fast enough" virtualization adds very overhead, a Windows VM running with a dedicated GPU can get 95% of the host's score on 3dmark. the biggest issue on these cases is IO which can be handled in a few ways.
  - lproven 3y ago
    > virtualization adds very overhead Missing word? "Very little"? 5% is enough to turn this year's high-end machine into last year's model. Nested virtualisation is also a thing now, and 5% per layer adds up fast.
- insanitybit 3y ago
  The short version is that kernels support guest/host relationships natively so that guests can pass operations directly to the host without having to go through an additional system call. Everywhere you do this is attack surface where an attacker in the guest can communicate with privileged facilities, so you want to minimize this where you can. There's usually overhead in the places where the communication requires an additional hop. If you want your host filesystem isolated you're going to need a translation layer and it will be slower. If you're willing to open up your host OS's filesystem, you can basically get ~0 overhead.
blastonico 3y ago
[flagged]
jitl 3y ago
Kata used “Linux kernel Direct Access filesystem (DAX)” to directly share access of the host filesystem to the guest kernel. I thought this was pretty interesting, but it sounds like a possible spot to start a jailbreak. I’m guessing these kinds of optimizations along with using super simple virtualized devices is what gives Kata its almost-cgroups-like performance. > Mapping files using DAX provides a number of benefits over more traditional VM file and device mapping mechanisms: > Mapping as a direct access device allows the guest to directly access the host memory pages (such as via Execute In Place (XIP)), bypassing the guest kernel's page cache. This zero copy provides both time and space optimizations. > Mapping as a direct access device inside the VM allows pages from the host to be demand loaded using page faults, rather than having to make requests via a virtualized device (causing expensive VM exits/hypercalls), thus providing a speed optimization. > Utilizing mmap(2)'s MAP_SHARED shared memory option on the host allows the host to efficiently share pages. From https://github.com/kata-containers/kata-containers/tree/main/docs/design/architecture#dax https://github.com/kata-containers/kata-containers/tree/main...
- hinkley 3y ago
  Would it be simple enough to stick a union file system over the top of the host file system?
  - jitl 3y ago
    Do you mean inside the guest? I'm sure they do stuff like that; they operate the normal sort of cgroups + fs mappings inside the guest VM to create inner containers. Eg all the containers in a Kubernetes pod run inside the same VM, but different containers within the VM.
  - rwmj 3y ago
    virtiofs supports DAX.
- insanitybit 3y ago
  Yeah, it's worth understanding the attack surface of DAX - if someone has information I'd be very interested. That said, you could mitigate it in other ways depending on your use case. Having gone through an evaluation of Firecracker's security my main conclusion was that sandboxing the processes in the guest is the highest 'bang for your buck' way to reduce escapes.
grahamgooch 3y ago
From Baidu https://katacontainers.io/collateral/ApplicationOfKataContainersInBaiduAICloud.pdf https://katacontainers.io/collateral/ApplicationOfKataContai...
airocker 3y ago
Is it possible to add kata containers in eks or gke?
- otterley 3y ago
  Not in EKS. EC2 doesn't support nested virtualization today.
  - rwmj 3y ago
    Peer pods are meant to solve this (one day) by running the "containers" (ie VMs) to the side as regular AWS instances and peering the communications: https://www.redhat.com/en/blog/red-hat-openshift-sandboxed-containers-peer-pods-hands https://www.redhat.com/en/blog/red-hat-openshift-sandboxed-c...
    - otterley 3y ago
      AWS already has an implementation of this, namely, Nitro Enclaves. A Nitro Enclave is an isolated VM used for confidential computing purposes. You provide the Enclave an EIF file, which is basically an OCI image with cryptographic measurements. The Nitro hypervisor launches the VM and allows communications with its parent VM only over a Linux vsock connection. Also, AWS has Fargate, which is basically containers-as-VMs. Every ECS task or EKS Pod is launched as a separate EC2 instance under the hood. This obviates a lot of the need for a solution like Kata there.
xyst 3y ago
> Runs in a dedicated kernel, providing isolation of network, I/O and memory and can utilize hardware-enforced isolation with virtualization VT extensions. So if it’s a dedicated kernel, can this fool game anti cheat systems into thinking it’s not in a VM? Or still the same problem?
- jitl 3y ago
  If a game anti-cheat system can detect that a regular VM is a VM, then it will also detect Kata VM is a VM. In both cases you're running a game in a VM with a dedicated kernel. Kata VMs are especially VM-y because they use a lot of VM-only features that wouldn't work with real hardware to enhance performance by sharing work between the guest and the host.
  - xyst 3y ago
    Well it’s not that it “detects” it but most game AC (Valgrind or BattleEye) install at the ring0 layer (kernel). So on VMs, that doesn’t exist (from what I understand). But with this it should be enough to fool it into thinking it’s installed in the right space.
    - amarshall 3y ago
      > So on VMs, [ring0 layer (the kernel)] doesn’t exist (from what I understand) The kernel still exists in a VM. The “dedicated kernel” bit of Kata distinguishes it from typical containers, not from VMs. Kata is an abstraction atop existing hypervisors that are themselves just an abstraction over KVM. There’s nothing new here w.r.t. VM detection evasion. > fool it into thinking it’s installed in the right space Most VM detection is about observing devices, drivers, timings, or other side-channel type data that is often only seen in a VM.
- insanitybit 3y ago
  Most VM detection I've seen is done through fingerprinting. Checking what the hardware is, certain CPU features/flags, config files, etc.
danjc 3y ago
Serious question - at this point why not just use actual VM's?
- rcoveson 3y ago
  That's what this is. "Containers" implemented as actual VMs.
- spicyusername 3y ago
  The API for containers is widely used.
- dharmab 3y ago
  1. So you can run any container image that someone made into a Docker container instead of building your own VM images 2. So you can run in Kubernetes and use the Kubernetes API, alongside any non-VM Pods and many other features in your Kubernetes cluster 3. Because traditional hypervisors have a higher overhead cost than the microVMs used by Kata and Firecracker 4. Because traditional hypervisors have a longer startup time (e.g. Firecracker is used by Amazon to run Lambda functions)
- solarkraft 3y ago
  They are VMs, but without the annoying "pet" UX.
zackmorris 3y ago
I wish that someone would put up a honeypot website running various containers holding a root user, where the only goal is to break out of the container. I suspect that nearly all containerization software is insecure. Especially with timing attacks like side-channel attacks (Row Hammer, etc): https://en.wikipedia.org/wiki/Timing_attack https://en.wikipedia.org/wiki/Timing_attack https://en.wikipedia.org/wiki/Side-channel_attack https://en.wikipedia.org/wiki/Side-channel_attack In the end, the only way to "prove" container security is to be able to point to the fact that nobody has broken out of it yet. It's ..remarkable that our entire cloud infrastructure runs on containers that have never been audited by brute-force in this manner.
- bozhark 3y ago
  Quantum hack? Nah, it’s 2023, brute it
- zsims 3y ago
  You mean like http://www.lambdashell.com/ http://www.lambdashell.com/ ?
- Quarrel 3y ago
  > I suspect that nearly all containerization software is insecure. This seems, well, naive? You think there aren't a lot of people that have tried to break out of cloud containers? Both EC2 and gcloud have had issues over the years with container breakout and leaks. Complex (ie literally layers of operating systems) software has bugs. Yes. But so does the non-containerised base-case. We have the best style of honeypots you could ever ask for already running- payment infrastructure on the internet. Go get 'em.
- wmf 3y ago
  That honeypot was called VPS hosting and over ten years it didn't get hacked much if at all.
- insanitybit 3y ago
  Lots of people have tested these systems. Here's work we did to exploit Firecracker. We had a known, promising vulnerability, and still failed to break out. https://web.archive.org/web/20220927150915/https://www.graplsecurity.com/post/attacking-firecracker https://web.archive.org/web/20220927150915/https://www.grapl... Exploitation is extremely hard. No one has ever exploited RowHammer in the wild, to my knowledge, and there are a lot of reasons why - but even still, RowHammer isn't magic, you can mitigate against it by increasing your refresh rate, or by limiting the attacker's execution time. Not to mention that vendors have deployed numerous methods of reducing the likelihood of an attack and it is quite complex to pull off these days.
dsp_person 3y ago
Vagrant also kind-of fits the description, but I've been looking for an alternative due to it's libvirt support being iffy (the plugin breaks often or has issues compared to the virtualbox provider, also not many official image providers). Here's one option I came across recently that's pretty nice and simple, using virsh and cloud images, and making it easy to spin up a VM and ssh into it. [1] One use case I'm looking for is to provision system images including ZFS pools/dataset, and not sharing the host kernel module. [1] https://earlruby.org/2023/02/quickly-create-guest-vms-using-virsh-cloud-image-files-and-cloud-init/ https://earlruby.org/2023/02/quickly-create-guest-vms-using-...
- alexeldeib 3y ago
  I love the vagrant UX, I’d be thrilled to see a refresh for 2023. Box management is odd. Weaveworks ignite/footloose are also good but don’t quite scratch my itch. I think vagrant UX with ignite style kernel/rootfs as oci image is a nice flow.
  - flanked-evergl 3y ago
    > Weaveworks ignite/footloose are also good but don’t quite scratch my itch. Are these still going? They seem a bit dead to me.
    - alexeldeib 3y ago
      that’s my take too. At least for ignite, I think Lucas had to take time off for college. Not sure about footloose. I guess another reason I suggested a refresh ;) Firecracker and firectl themselves are maintained of course, but lower level.
- sureglymop 3y ago
  I also use cloud init and one can quickly (5 seconds) spin up VMs with it. I use virt-install which has command line options for the cloud init related files which makes everything very easy.
solarkraft 3y ago
I'm very interested in the concept, but found Kata to be hard to get going with. Last time I looked (a few months ago), the documentation was pretty sparse or outdated. A lot of documentation I found stated something like "we broke that in the new version, you can't actually do that right now", like using it with Docker, which I would much prefer over setting up Kubernetes. I still think it's an absolutely great thing, but the on onboarding could have been a lot less rocky. FWIW, I eventually kind of DIYed it with QEMU MicroVM and virtiofs - never did anything with it though.
- solarkraft 3y ago
  On the onboarding point: Almost all projects would do themselves a big service by putting work into the onboarding experience because it's a key factor in how many people will use and help developing the project.
- flanked-evergl 3y ago
  > Last time I looked (a few months ago), the documentation was pretty sparse or outdated. It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main/utils/README.md#install-a-minimal-kata-containers-system https://github.com/kata-containers/kata-containers/blob/main... Though only one of the hypervisors works well.
  - rcoveson 3y ago
    > Though only one of the hypervisors works well. Well don't leave us hanging, which one? (My money's on QEMU)
    - flanked-evergl 3y ago
      Sorry, I should have said, but yes it is QEMU which is the default.
revskill 3y ago
Should i run `docker run helloworld` or use katacontainer VM to output Hello world ?
- flanked-evergl 3y ago
  If you install kata with https://github.com/kata-containers/kata-containers/blob/main/utils/README.md#install-a-minimal-kata-containers-system https://github.com/kata-containers/kata-containers/blob/main... Then you just use it with: docker run --runtime io.containerd.kata.v2 --rm -it hello-world
symlinkk 3y ago
So why would you use this over regular containers? Is this for people who think containers are insecure?
- insanitybit 3y ago
  Containers are not appropriate for some scenarios. If your threat model is "attacker has arbitrary code execution" you should be very wary of a container holding them for long - you're one kernel privesc away from a container escape. It's nice to have if your situation is "first the attacker would need to exploit my service and then they'd have code exec", but if you provide RCE as a service a normal container is just not a good barrier. By contrast, running attacker code in Firecracker is much safer - for one thing, the attacker needs to escalate within the VM in order to then expose the necessary primitives to then escape the VM. So it immediately adds an additional required vuln. But also, instead of the entire Linux kernel being the attack surface (it is for the first vuln, but...) you have to attack a much smaller codebase that implements the VM. In the case of Firecracker this is really hard and, depending on your exploit, if you end up controlling the Firecracker VM itself you're actually in yet another sandbox - so you now need another vuln to escape (although that sandbox is not amazing imo).
  - symlinkk 3y ago
    But containers run everywhere. ECS for example. If they were insecure then half the web services we use every day would get hacked.
    - insanitybit 3y ago
      ECS is not using containers for isolation though. A single ESC cluster will only ever run on single tenant infrastructure. When AWS requires isolation they never use containers.
- kbumsik 3y ago
  Many serverless services use MicroVM rather because they think containers doesn't have enough isolation (=insecure?) to run untrusted code.
sharts 3y ago
How does this compare to Singularity containers?

bionsystem 3y ago

I don't understand how can they be as fast as regular containers if they run an entire kernel on top of an hypervisor ?

Alifatisk 3y ago
Yeah I'd like to see some numbers on that, like startup time.
- theaiquestion 3y ago
  Depending on your use case there's potentially negligible startup time. On the scale of single digit seconds to less then half a second depending on how much work you put into optimizing it. For some applications this will be too slow (mainly the type where you boot a container per request, although flyio seems to make it work), I think for a _lot_ of applications this wouldn't be noticed. Kata gives you a few different options for what/how you'd like to boot including firecracker. This isn't exclusive to firecracker but if you stay lightweight you can have vm's booting under a half second if you're using slim images. https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... I honestly think for a lot of people, vm's with the convenience/orchestration tools of containers make more sense for a lot of general use cases simply because of the security benefits. The convenience still needs some work though.
  - insanitybit 3y ago
    Unless you're dealing with a multi-tenant situation I'm not super convinced that a VM is worth the effort. It's not the perf, it's the need to make your kernel, root file system, and other infra needed to make it all work. Compare that to a docker container where there's basically 0 additional work that has to be done to be up and running. For most cases I'd be really tempted to work on hardening the docker container than on setting up a VM. Things like Apparmor and seccomp in particular would likely go a very long way.
    - deleted 3y ago
      [deleted]
rwmj 3y ago
I don't have numbers either, but it's a combination of extreme focus on the boot path and virtio drivers, and traditional containers now being quite heavyweight to start (especially when run via Kubernetes). The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. This can practically limit the number of Katacontainers you can run to something much smaller than is possible with ordinary containers, since RAM is the constrained resource on most servers. Nevertheless, with confidential computing coming along, it's likely that at some point in the future many containers will really be VMs, since current CPUs implement confidential computing on top of existing VM primitives (and that's basically necessary due to the way the guest RAM is encrypted). It's likely that any workload that touches PII, finance, health, etc will be required to use confidential computing.
- cogman10 3y ago
  > but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front. Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces. The down side is, of course, security. But that was always the case with containers.
  - tracker1 3y ago
    I think in the longer run, WASM might displace a lot of both in practical terms.
    - ehutch79 3y ago
      Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system? We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.
mugsie 3y ago
there is pretty low overhead if you are opinionated - this is very similar to firecracker (AWS) tooling, so cut down hypervisor with ~ 0 devices, and a cut down guest OS means pretty quick boot times
kobalsky 3y ago
there's probably a big asterisk there. the correct term is probably "fast enough" virtualization adds very overhead, a Windows VM running with a dedicated GPU can get 95% of the host's score on 3dmark. the biggest issue on these cases is IO which can be handled in a few ways.
- lproven 3y ago
  > virtualization adds very overhead Missing word? "Very little"? 5% is enough to turn this year's high-end machine into last year's model. Nested virtualisation is also a thing now, and 5% per layer adds up fast.
insanitybit 3y ago
The short version is that kernels support guest/host relationships natively so that guests can pass operations directly to the host without having to go through an additional system call. Everywhere you do this is attack surface where an attacker in the guest can communicate with privileged facilities, so you want to minimize this where you can. There's usually overhead in the places where the communication requires an additional hop. If you want your host filesystem isolated you're going to need a translation layer and it will be slower. If you're willing to open up your host OS's filesystem, you can basically get ~0 overhead.

blastonico 3y ago

[flagged]

jitl 3y ago

Kata used “Linux kernel Direct Access filesystem (DAX)” to directly share access of the host filesystem to the guest kernel. I thought this was pretty interesting, but it sounds like a possible spot to start a jailbreak. I’m guessing these kinds of optimizations along with using super simple virtualized devices is what gives Kata its almost-cgroups-like performance. > Mapping files using DAX provides a number of benefits over more traditional VM file and device mapping mechanisms: > Mapping as a direct access device allows the guest to directly access the host memory pages (such as via Execute In Place (XIP)), bypassing the guest kernel's page cache. This zero copy provides both time and space optimizations. > Mapping as a direct access device inside the VM allows pages from the host to be demand loaded using page faults, rather than having to make requests via a virtualized device (causing expensive VM exits/hypercalls), thus providing a speed optimization. > Utilizing mmap(2)'s MAP_SHARED shared memory option on the host allows the host to efficiently share pages. From https://github.com/kata-containers/kata-containers/tree/main/docs/design/architecture#dax https://github.com/kata-containers/kata-containers/tree/main...

hinkley 3y ago
Would it be simple enough to stick a union file system over the top of the host file system?
- jitl 3y ago
  Do you mean inside the guest? I'm sure they do stuff like that; they operate the normal sort of cgroups + fs mappings inside the guest VM to create inner containers. Eg all the containers in a Kubernetes pod run inside the same VM, but different containers within the VM.
- rwmj 3y ago
  virtiofs supports DAX.
insanitybit 3y ago
Yeah, it's worth understanding the attack surface of DAX - if someone has information I'd be very interested. That said, you could mitigate it in other ways depending on your use case. Having gone through an evaluation of Firecracker's security my main conclusion was that sandboxing the processes in the guest is the highest 'bang for your buck' way to reduce escapes.

grahamgooch 3y ago

From Baidu https://katacontainers.io/collateral/ApplicationOfKataContainersInBaiduAICloud.pdf https://katacontainers.io/collateral/ApplicationOfKataContai...

airocker 3y ago

Is it possible to add kata containers in eks or gke?

otterley 3y ago
Not in EKS. EC2 doesn't support nested virtualization today.
- rwmj 3y ago
  Peer pods are meant to solve this (one day) by running the "containers" (ie VMs) to the side as regular AWS instances and peering the communications: https://www.redhat.com/en/blog/red-hat-openshift-sandboxed-containers-peer-pods-hands https://www.redhat.com/en/blog/red-hat-openshift-sandboxed-c...
  - otterley 3y ago
    AWS already has an implementation of this, namely, Nitro Enclaves. A Nitro Enclave is an isolated VM used for confidential computing purposes. You provide the Enclave an EIF file, which is basically an OCI image with cryptographic measurements. The Nitro hypervisor launches the VM and allows communications with its parent VM only over a Linux vsock connection. Also, AWS has Fargate, which is basically containers-as-VMs. Every ECS task or EKS Pod is launched as a separate EC2 instance under the hood. This obviates a lot of the need for a solution like Kata there.

xyst 3y ago

> Runs in a dedicated kernel, providing isolation of network, I/O and memory and can utilize hardware-enforced isolation with virtualization VT extensions. So if it’s a dedicated kernel, can this fool game anti cheat systems into thinking it’s not in a VM? Or still the same problem?

jitl 3y ago
If a game anti-cheat system can detect that a regular VM is a VM, then it will also detect Kata VM is a VM. In both cases you're running a game in a VM with a dedicated kernel. Kata VMs are especially VM-y because they use a lot of VM-only features that wouldn't work with real hardware to enhance performance by sharing work between the guest and the host.
- xyst 3y ago
  Well it’s not that it “detects” it but most game AC (Valgrind or BattleEye) install at the ring0 layer (kernel). So on VMs, that doesn’t exist (from what I understand). But with this it should be enough to fool it into thinking it’s installed in the right space.
  - amarshall 3y ago
    > So on VMs, [ring0 layer (the kernel)] doesn’t exist (from what I understand) The kernel still exists in a VM. The “dedicated kernel” bit of Kata distinguishes it from typical containers, not from VMs. Kata is an abstraction atop existing hypervisors that are themselves just an abstraction over KVM. There’s nothing new here w.r.t. VM detection evasion. > fool it into thinking it’s installed in the right space Most VM detection is about observing devices, drivers, timings, or other side-channel type data that is often only seen in a VM.
insanitybit 3y ago
Most VM detection I've seen is done through fingerprinting. Checking what the hardware is, certain CPU features/flags, config files, etc.

danjc 3y ago

Serious question - at this point why not just use actual VM's?

rcoveson 3y ago
That's what this is. "Containers" implemented as actual VMs.
spicyusername 3y ago
The API for containers is widely used.
dharmab 3y ago
1. So you can run any container image that someone made into a Docker container instead of building your own VM images 2. So you can run in Kubernetes and use the Kubernetes API, alongside any non-VM Pods and many other features in your Kubernetes cluster 3. Because traditional hypervisors have a higher overhead cost than the microVMs used by Kata and Firecracker 4. Because traditional hypervisors have a longer startup time (e.g. Firecracker is used by Amazon to run Lambda functions)
solarkraft 3y ago
They are VMs, but without the annoying "pet" UX.

zackmorris 3y ago

I wish that someone would put up a honeypot website running various containers holding a root user, where the only goal is to break out of the container. I suspect that nearly all containerization software is insecure. Especially with timing attacks like side-channel attacks (Row Hammer, etc): https://en.wikipedia.org/wiki/Timing_attack https://en.wikipedia.org/wiki/Timing_attack https://en.wikipedia.org/wiki/Side-channel_attack https://en.wikipedia.org/wiki/Side-channel_attack In the end, the only way to "prove" container security is to be able to point to the fact that nobody has broken out of it yet. It's ..remarkable that our entire cloud infrastructure runs on containers that have never been audited by brute-force in this manner.

bozhark 3y ago
Quantum hack? Nah, it’s 2023, brute it
zsims 3y ago
You mean like http://www.lambdashell.com/ http://www.lambdashell.com/ ?
Quarrel 3y ago
> I suspect that nearly all containerization software is insecure. This seems, well, naive? You think there aren't a lot of people that have tried to break out of cloud containers? Both EC2 and gcloud have had issues over the years with container breakout and leaks. Complex (ie literally layers of operating systems) software has bugs. Yes. But so does the non-containerised base-case. We have the best style of honeypots you could ever ask for already running- payment infrastructure on the internet. Go get 'em.
wmf 3y ago
That honeypot was called VPS hosting and over ten years it didn't get hacked much if at all.
insanitybit 3y ago
Lots of people have tested these systems. Here's work we did to exploit Firecracker. We had a known, promising vulnerability, and still failed to break out. https://web.archive.org/web/20220927150915/https://www.graplsecurity.com/post/attacking-firecracker https://web.archive.org/web/20220927150915/https://www.grapl... Exploitation is extremely hard. No one has ever exploited RowHammer in the wild, to my knowledge, and there are a lot of reasons why - but even still, RowHammer isn't magic, you can mitigate against it by increasing your refresh rate, or by limiting the attacker's execution time. Not to mention that vendors have deployed numerous methods of reducing the likelihood of an attack and it is quite complex to pull off these days.

dsp_person 3y ago

Vagrant also kind-of fits the description, but I've been looking for an alternative due to it's libvirt support being iffy (the plugin breaks often or has issues compared to the virtualbox provider, also not many official image providers). Here's one option I came across recently that's pretty nice and simple, using virsh and cloud images, and making it easy to spin up a VM and ssh into it. [1] One use case I'm looking for is to provision system images including ZFS pools/dataset, and not sharing the host kernel module. [1] https://earlruby.org/2023/02/quickly-create-guest-vms-using-virsh-cloud-image-files-and-cloud-init/ https://earlruby.org/2023/02/quickly-create-guest-vms-using-...

alexeldeib 3y ago
I love the vagrant UX, I’d be thrilled to see a refresh for 2023. Box management is odd. Weaveworks ignite/footloose are also good but don’t quite scratch my itch. I think vagrant UX with ignite style kernel/rootfs as oci image is a nice flow.
- flanked-evergl 3y ago
  > Weaveworks ignite/footloose are also good but don’t quite scratch my itch. Are these still going? They seem a bit dead to me.
  - alexeldeib 3y ago
    that’s my take too. At least for ignite, I think Lucas had to take time off for college. Not sure about footloose. I guess another reason I suggested a refresh ;) Firecracker and firectl themselves are maintained of course, but lower level.
sureglymop 3y ago
I also use cloud init and one can quickly (5 seconds) spin up VMs with it. I use virt-install which has command line options for the cloud init related files which makes everything very easy.

solarkraft 3y ago

I'm very interested in the concept, but found Kata to be hard to get going with. Last time I looked (a few months ago), the documentation was pretty sparse or outdated. A lot of documentation I found stated something like "we broke that in the new version, you can't actually do that right now", like using it with Docker, which I would much prefer over setting up Kubernetes. I still think it's an absolutely great thing, but the on onboarding could have been a lot less rocky. FWIW, I eventually kind of DIYed it with QEMU MicroVM and virtiofs - never did anything with it though.

solarkraft 3y ago
On the onboarding point: Almost all projects would do themselves a big service by putting work into the onboarding experience because it's a key factor in how many people will use and help developing the project.
flanked-evergl 3y ago
> Last time I looked (a few months ago), the documentation was pretty sparse or outdated. It still is, though it works somewhat seamlessly when installing with https://github.com/kata-containers/kata-containers/blob/main/utils/README.md#install-a-minimal-kata-containers-system https://github.com/kata-containers/kata-containers/blob/main... Though only one of the hypervisors works well.
- rcoveson 3y ago
  > Though only one of the hypervisors works well. Well don't leave us hanging, which one? (My money's on QEMU)
  - flanked-evergl 3y ago
    Sorry, I should have said, but yes it is QEMU which is the default.

revskill 3y ago

Should i run `docker run helloworld` or use katacontainer VM to output Hello world ?

flanked-evergl 3y ago
If you install kata with https://github.com/kata-containers/kata-containers/blob/main/utils/README.md#install-a-minimal-kata-containers-system https://github.com/kata-containers/kata-containers/blob/main... Then you just use it with: docker run --runtime io.containerd.kata.v2 --rm -it hello-world

symlinkk 3y ago

So why would you use this over regular containers? Is this for people who think containers are insecure?

insanitybit 3y ago
Containers are not appropriate for some scenarios. If your threat model is "attacker has arbitrary code execution" you should be very wary of a container holding them for long - you're one kernel privesc away from a container escape. It's nice to have if your situation is "first the attacker would need to exploit my service and then they'd have code exec", but if you provide RCE as a service a normal container is just not a good barrier. By contrast, running attacker code in Firecracker is much safer - for one thing, the attacker needs to escalate within the VM in order to then expose the necessary primitives to then escape the VM. So it immediately adds an additional required vuln. But also, instead of the entire Linux kernel being the attack surface (it is for the first vuln, but...) you have to attack a much smaller codebase that implements the VM. In the case of Firecracker this is really hard and, depending on your exploit, if you end up controlling the Firecracker VM itself you're actually in yet another sandbox - so you now need another vuln to escape (although that sandbox is not amazing imo).
- symlinkk 3y ago
  But containers run everywhere. ECS for example. If they were insecure then half the web services we use every day would get hacked.
  - insanitybit 3y ago
    ECS is not using containers for isolation though. A single ESC cluster will only ever run on single tenant infrastructure. When AWS requires isolation they never use containers.
kbumsik 3y ago
Many serverless services use MicroVM rather because they think containers doesn't have enough isolation (=insecure?) to run untrusted code.

sharts 3y ago

How does this compare to Singularity containers?