10 ms·

My VM is lighter (and safer) than your container (2017)

309 points by gaocegege 4y ago

fnord123 4y ago
Title is kinda clickbaity (wha-? how can a VM be lighter than a container). It's about unikernels.
- JeanSebTr 4y ago
  Exactly, unikernels are great for performance and isolation, but that can't be compared to a full application stack running in a container or VM.
  - OrvalWintermute 4y ago
    Not all containers are a "full application stack" A unikernel can definitely be compared.
- nailer 4y ago
  Firecracker VMs are considered lighter than a container and are pretty old at this point.
  - sidkshatriya 4y ago
    I would say that firecracker VMs are not more lightweight than Linux containers. Linux containers are essentially the separation of Linux processes via various namespaces e.g. mount, cgroup, process, network etc. Because this separation is done by Linux internally there are not that many overheads. VMs provide a different kind of separation one that is arguably more secure because it is backed up hardware -- each VM thinks it has the whole hardware to itself. When you switch between the VM and the host there is quite a heavyweight context switch (VMEXIT/VMENTER in Intel parlance). It can take a long time compared to just the usual context switch from one Linux container (process) to another host (process) or another Linux container (process). But coming back to your point, no firecracker VMs are not lighter/lightweight than a Linux container. They are quite heavyweight actually. But the firecracker VMM is probably the most nimble of all VMMs.
- throwaway894345 4y ago
  > how can a VM be lighter than a container It's still clickbaity, but the title implies a comparison between a very lightweight VM and a heavy-weight container (presumably a container based on a full Linux distro). You could imagine an analogous article about a tiny house titled "my house is smaller than your apartment".
  - marcosdumay 4y ago
    It is still lighter in memory only. CPU is also a relevant thing to compare them.
  - turkishmonky 4y ago
    Not to mention, in the paper, the lightvm only had an advantage on boot times. Menory usage was marginally worse than docker, even with the unikernel, and debian on lightvm was drastically worse for cpu usage than docker (the unikernel cpu usage was neck and neck with the debian docker contaner). I could see it being an improvement over other VM control planes, but docker still wins in performance for any equivalant comparisons.
- deleted 4y ago
  [deleted]
- pjmlp 4y ago
  Which in case someone has failed to notice, is pretty much how serverless happens to work out from development point of view. No one cares how the language runtime is actually running.
kasperni 4y ago
[2017]
JStanton617 4y ago
This paper references consistently mischaracterizes AWS Lambda as a "Container as a Service" technology, when in fact it is exactly the sort of lightweight VM that they are describing - https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...
- Jtsummers 4y ago
  In fairness to this paper, it was written and published before that Firecracker article (2017 vs 2018). From another paper on Firecracker providing a bit of history: > When we first built AWS Lambda, we chose to use Linux containers to isolate functions, and virtualization to isolate between customer accounts. In other words, multiple functions for the same customer would run inside a single VM, but workloads for different customers always run in different VMs. We were unsatisfied with this approach for several reasons, including the necessity of trading off between security and compatibility that containers represent, and the difficulties of efficiently packing workloads onto fixed-size VMs. And a bit about the timeline: > Firecracker has been used in production in Lambda since 2018, where it powers millions of workloads and trillions of requests per month. https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
  - runnerup 4y ago
    Thank you for this detail!
- wodenokoto 4y ago
  For what it’s worth, Google’s cloud functions are a container service. You can even download the final docker container.
  - raggi 4y ago
    KVM gVisor is a hybrid model in this context. It shares properties with both containers and lightweight VMs.
- deleted 4y ago
GekkePrutser 4y ago
Sometimes the less strict separation is a feature, not a bug. Without folder sharing with dockers for example, it would be pretty useless.
- 1MachineElf 4y ago
  While a flawed comparison, WSL does use a VM in conjunction with the 9p protocol to achieve folder sharing.
  - liftm 4y ago
    9p-based folder sharing is (used to be?) possible with qemu, too.
    - ksbrooksjr 4y ago
      It looks like it still is supported [1]. I noticed while reading the Lima documentation that they're planning on switching from SSHFS to 9P [2]. [1] https://wiki.qemu.org/Documentation/9psetup https://wiki.qemu.org/Documentation/9psetup [2] https://github.com/lima-vm/lima/blob/3401b97e602083cfc55b34e16d5ae8f119588397/docs/mount.md#9p https://github.com/lima-vm/lima/blob/3401b97e602083cfc55b34e...
    - ParetoOptimal 4y ago
      Yep, can confirm. NixOS configurations can be built and run as a qemu VM. I very recently was using one and found they automatically make a 9p mount to /tmp.
r3mc0 4y ago
Containers and VMs are totally not the same thing. They serve a complete other purpose , as multiple containers can be combined to create an application/service , VMs always use a complete os etc etc anyway the internet is full of the true purpose of containers , they were never meant to use as a "VM" and about security.. meh everything is insecure until proven differently
- wongarsu 4y ago
  VMs can have private networks between each other just as containers do. That's pretty much what EC2 is about.
  - kazen44 4y ago
    heck, when running something in production with multiple underlying servers, you need a system that builds an overlay network regardless if you use containers or Vm's..
- nijave 4y ago
  VMs don't need a full OS. You can run a single process directly from the kernel with no init system or other userland
- R0flcopt3r 4y ago
  Maybe you should read the article? The article basically covers running a tiny kernel that as exactly what you need to run one single binary, which is your application. No printer drivers, video drivers. Probably have network drivers, as that would be quite handy. Kinda up to you on what kernel you pick for this, and what drivers are included. Multiple VMs with networking can also be "combined"... Containers are great because they are small and fast, low overhead etc. And that is why they replaced virtual machines. This article however proves that this is also achievable with virtual machines.
oxfordmale 4y ago
Kubernetes says no... The article is light on detail. Containers and VMs have different use cases. If you self host lightweight VMs is likely the better path, however, once you in the cloud most managed services only provide support for containers.
- lrvick 4y ago
  You can also use a firecracker runner in k8s to wrap each container in a VM for high isolation and security.
- nailer 4y ago
  > in the cloud most managed services only provide support for containers. Respectfully, comments like these are the reason for Kubernetes becoming a meme.
  - oxfordmale 4y ago
    There is a huge difference running on VMs that you have zero access to, and actually owning your own VM infrastructure. Yes AWS Lambda runs on Firecracker, however, it could as well running on a FireCheese VM platform and you would be none the wiser, unless AWS publishes this somewhere. I am also not running on Kubernetes, because Kubernetes. AWS ECS and AWS Batch also only handle containerised applications. Even when deploying on EC2 I tend to use containers, as it ensures they keep working consistently if you apply patches to your EC2 environment.
bongobingo1 4y ago
I'm quite interested in seeing where slim VM's go. Personally I don't use Kubernetes, it just doesn't fit my client work which is nearly all single-server and it makes more sense to just run podman systemd units or docker-compose setups. So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close). Honestly I don't really care about containers vs VMs except the VM alleges better security which is nice, and I guess I like poking at things but they're were a little too rough for weekend poking. Is anyone doing "small scale" lightweight vm deployments - maybe just in your homelab or toy projects? Have you found the experience better than containers?
- xani_ 4y ago
  They went to trash because containers are more convenient to use and saving few MBs of disk/memory is not what most users care. The whole idea was pretty much either use custom kernel (which inevitably have way less info on how to debug anything in it), and re-do all of the network and storage plumbing containers already do via the OS they are running one. OR just very slim linux one which at least people know how to use but STILL is more complexity than "just a blob with some namespaces in it" and STILL requires a bunch of config and data juggling between hypervisor and VM just to share some host files to the guest. Either way to get to the level of "just a slim layer of code between hypervisor and your code" you need to do a quite a lot of deep plumbing and when anything goes wrong debugging is harder. All to get some perceived security and no better performance than just... running the binary in a container. It did percolate into "slim containers" idea where the container is just statically compiled binary + few configs and while it does have same problems with debuggability, you can just attach sidecart to it I guess next big hype will be "VM bUt YoU RuN WebAsSeMbLy In CuStOm KeRnEl"
  - evol262 4y ago
    Virtualization is not just "perceived" security over containerization. From CPU rings on down, it offers dramatically more isolation for security than containerization does. This isn't about 'what most users care' about either. Most users don't really care about 99% of what container orchestration platforms offer. The providers do absolutely care that malicious users cannot punch out to get a shell on an Azure AKS controller or go digging around inside /proc to figure out what other tenants are doing unless the provider is on top of their configuration and regularly updates to match CVEs. "most users" will end up using one of the frameworks written by a "big boy" for their stuff, and they'll end up using what's convenient for cloud providers. The goal of microvms is ultimately to remove everything you're talking about from the equation. Kata and other microvm frameworks aim to be basically jsut another CRI which removes the "deep plumbing" you're talking about. The onus is on them to make this work, but there's an enormous financial payoff, and you'll end up with this whether you think it's worthwhile or not.
opentokix 4y ago
Tell me you don't understand containers wihout telling me you dont understand containers.
- anthk 4y ago
  You don't understand vm's neither. Ever used virtual network interfaces?
ricardobeat 4y ago
How does LightVM compare to Firecracker VMs? Could it be used for on-demand cloud VMs?
jupp0r 4y ago
(2017)
jjtheblunt 4y ago
"orders of magnitude" : Why does anyone ever write "two orders of magnitude" when 100x is shorter? Of course, this presumes 10 as the magnitude and the N orders to be the exponent, but I don't think I've ever, since the 90s, seen that stilted phrasing ever used for a base other than 10.
- IshKebab 4y ago
  Because two orders of magnitude does not mean 100x. It means on the same order as 100x.
  - jjtheblunt 4y ago
    Do you mean folks using the phrase know big-O, big-omega, big-theta, and are thinking along those lines?
    - IshKebab 4y ago
      It's nothing to do with big-O; it's about logarithms. But really I think most people using it just think of it like: "which of these is it closest to? 10x, 100x or 1000x?"
deleted 4y ago
[deleted]
Sohcahtoa82 4y ago
> We achieve lightweight VMs by using unikernels When I attended Infiltrate a few years ago, there was a talk about unikernels. The speaker showed off how incredibly insecure many of them were, not even offering support for basic modern security features like DEP and ALSR. Have they changed? Or did the speaker likely just cherry-pick some especially bad ones?
- sieabah 4y ago
  The headline reads like a reddit post so I'm going to assume the same still holds true.
- eyberg 4y ago
  You are probably talking about this: https://research.nccgroup.com/wp-content/uploads/2020/07/ncc_group-assessing_unikernel_security.pdf https://research.nccgroup.com/wp-content/uploads/2020/07/ncc... In short - not a fundamental limitation - just that kernels (even if they are small) have a ton of work that goes into them. Nanos for instance has page protections, ASLR, virtio-rng (if on GCP), etc.
  - OrvalWintermute 4y ago
    I thought that presentation was a little like looking at a hobby OS of a type, then attempting to draw security conclusions for all of that type. The NanoVMs unikernel for example, is pretty small, DoD supported, and has: ASLR Stack Randomization Heap Randomization Library Randomization Executable Randomization Page Protections Stack Execution off by Default Heap Execution off by Default Null Page is Not Mapped Stack Cookies/Canaries Rodata no execute Text no write STIG
devmor 4y ago
Yes, when you custom engineer a specific, complex solution for a specific use case, it is generally more performative than a general-use solution that's simple.
gavinray 4y ago
The issue with unikernels and things like Firecracker are that you can't run them on already-virtualized platforms I researched Firecracker when I was looking for an alternative to Docker for deploying FaaS functions on an OpenFaaS-like clone I was building It would have worked great if the target deployment was bare metal but if you're asking a user to deploy on IE an EC2 or Fargate or whatnot, you can't use these things so all points are moot This is relevant if you're self-hosting or you ARE a service provider I guess. (Yes, I know about Firecracker-in-Docker, but I mean real production use)
- gamegoblin 4y ago
  This is a limitation of whatever virtualized instance you're running on, not Firecracker itself. Firecracker depends on KVM, and AWS EC2 virtualized instances don't enable KVM. But not all virtualized instance services disable KVM. Obviously, Firecracker being developed by AWS and AWS disabling KVM is not ideal :) Google Cloud, for instance, allows nested virtualization, IIRC.
  - verdverm 4y ago
    Ive used GCP nested virtualization. You pay for that overhead in performance so I wouldn't recommend it without more investigation. We were trying to simulate using luks and the physical is key insert / removal. Would have used it more if we could get GPU passthrough working
  - shepherdjerred 4y ago
    Azure and Digital Ocean allowed nested virt as well!
  - gavinray 4y ago
    Yeah but imagine trying to convince people to use an OSS tool where the catch is that you have to deploy it on special instances, only on providers that support nested virtualization Not a great DX, haha I wound up using GraalVM's "Polyglot" abilities alongside it's WASM stuff
wyager 4y ago
It's not clear to me that VMs actually do offer better isolation than well-designed containers (i.e. not docker). It's basically a question of: do you trust the safety of kernel-mode drivers (for e.g. PV network devices or emulated hardware) for VMs, or do you trust the safety of userland APIs + the limited set of kernel APIs available to containers. On my FreeBSD server, I kind of trust jails with strict device rules (i.e. there are only like 5 things in /dev/) over a VM with virtualized graphics, networking, etc.
- nijave 4y ago
  I think it gets even more complicated with something like firecracker where they recommend you run firecracker in a jail (and provide a utility to set that up)
xahrepap 4y ago
This reminds me: in 2015 I went to Dockercon and one booth that was fun was VMWare's. Basically they had implemented the Docker APIs on top of VMWare so that they could build and deploy VMs using Dockerfiles, etc. I've casually searched for it in the past and it seems to not exist anymore. For me, one of the best parts of Docker is building a docker-image (and sharing how it was done via git). It would be cool to be able to take the same Dockerfiles and pivot them to VMs easily.
- verdverm 4y ago
  They might have built it into Google Anthos as part of their partnership. I recall seeing a demo where you could deploy & run any* VMWare image on Kubernetes without any changes
- All4All 4y ago
  Isn't that essentially what Vagrant and Vagrantfiles do?
  - xahrepap 4y ago
    Yes, which is what I'm using now. But it doesn't use the Docker APIs to allow you to (mostly) reuse a dockerfile to build a VM or a container. not sure if it would be better than Vagrant. But it was still very interesting.
  - hinkley 4y ago
    What is your theory for why Docker won and Vagrant didn't? Mine is that all of the previous options were too Turing Complete, while the Dockerfile format more closely follows the Principle of Least Power. Power users always complain about how their awesome tool gets ignored while 'lesser' tools become popular. And then they put so much energy into apologizing for problems with the tool or deflecting by denigrating the people who complain. Maybe the problem isn't with 'everyone'. Maybe Power Users have control issues, and pandering to them is not a successful strategy.
    - pphysch 4y ago
lamontcg 4y ago
Containers should really be viewed as an extension of packages (like RPM) with a bit of extra sauce with the layered filesystem, a chroot/jail and cgroups for some isolation between different software running on the same server. Back in 2003 or so we tried doing this with microservices that didn't need an entire server with multiple different software teams running apps on the same physical image to try to avoid giving entire servers to teams that would be only using a few percent of the metal. This failed pretty quickly as software bugs would blow up the whole image and different software teams got really grounchy at each other. With containerization the chroot means that the software carries along all its own deps and the underlying server/metal image can be managed seperately, and the cgroups means that software groups are less likely to stomp on each other due to bugs. This isn't a cloud model of course, it was all on-prem. I don't know how kubernetes works in the cloud where you can conceivably be running containers on metal sharing with other customers. I would tend to assume that under the covers those cloud vendors are using Containers on VMs on Metal to provide better security guarantees than just containers can offer. Containers really shouldn't be viewed as competing with VMs in a strict XOR sense.
- nikokrock 4y ago
  I don’t remember where i read it but as far as i know when using Fargate to run containers (with k8s or ecs) AWS will just allocate an ec2 instance for you. Your container will never run on the same vm as other customer. This explain i think the latency you can have to start a container. To improve that you need to handle your own ec2 cluster with an autoscaling group
- pjmlp 4y ago
  Back in 2000 we were doing this with HP-UX Vaults, IT keeps going in circles.
  - kazen44 4y ago
    ASlso, docker to me seems quite leaky in terms of abstractions (especially the networking side) compared to something like a freeBSD jail. why does docker networking need to be infinitely complex? freebsd solved this issue far more cleanly two decades ago without having to emulate entirely new networking spaces.
faeriechangling 4y ago
What I see happening now on the cloud is containers from different companies and different security domain running on the same VM. I have to think this is fundamentally insecure and that VMs are underrated. I hear people advocate QubesOS for security which is based on XEN when it comes to running my client. They say my banking should be done in a different VM than my email for instance. Well if that’s the case, why do we run many containers doing different security sensitive functions on the same VM when containers are not really considered a very good security boundary? From a security design perspective I imagine hardware being exclusive to a person/organization, vms being exclusive to some security function, and containers existing on top of that makes more sense from a security function but we seem to be playing things more loosely on the server side.
- bgm1975 4y ago
  Doesn’t AWS use firecracker with its Fargate container service (and lambda too)?
djhaskin987 4y ago
Not surprising that VMs running unikernels are as nimble as containers, but not quite useful either, at least in general. Much easier to just use a stock docker image.
dirkg 4y ago
Why is a 5yr old article being posted now? If this were to catch on, it would've. I just dont see it being used anywhere. Having a full Linux kernel available is a major benefit that you lose, right?
deleted 4y ago
[deleted]
thomasjudge 4y ago
This is 5 years old. What's the current state of the art?

309 points by gaocegege 4y ago

fnord123 4y ago

Title is kinda clickbaity (wha-? how can a VM be lighter than a container). It's about unikernels.

JeanSebTr 4y ago
Exactly, unikernels are great for performance and isolation, but that can't be compared to a full application stack running in a container or VM.
- OrvalWintermute 4y ago
  Not all containers are a "full application stack" A unikernel can definitely be compared.
nailer 4y ago
Firecracker VMs are considered lighter than a container and are pretty old at this point.
- sidkshatriya 4y ago
  I would say that firecracker VMs are not more lightweight than Linux containers. Linux containers are essentially the separation of Linux processes via various namespaces e.g. mount, cgroup, process, network etc. Because this separation is done by Linux internally there are not that many overheads. VMs provide a different kind of separation one that is arguably more secure because it is backed up hardware -- each VM thinks it has the whole hardware to itself. When you switch between the VM and the host there is quite a heavyweight context switch (VMEXIT/VMENTER in Intel parlance). It can take a long time compared to just the usual context switch from one Linux container (process) to another host (process) or another Linux container (process). But coming back to your point, no firecracker VMs are not lighter/lightweight than a Linux container. They are quite heavyweight actually. But the firecracker VMM is probably the most nimble of all VMMs.
throwaway894345 4y ago
> how can a VM be lighter than a container It's still clickbaity, but the title implies a comparison between a very lightweight VM and a heavy-weight container (presumably a container based on a full Linux distro). You could imagine an analogous article about a tiny house titled "my house is smaller than your apartment".
- marcosdumay 4y ago
  It is still lighter in memory only. CPU is also a relevant thing to compare them.
- turkishmonky 4y ago
  Not to mention, in the paper, the lightvm only had an advantage on boot times. Menory usage was marginally worse than docker, even with the unikernel, and debian on lightvm was drastically worse for cpu usage than docker (the unikernel cpu usage was neck and neck with the debian docker contaner). I could see it being an improvement over other VM control planes, but docker still wins in performance for any equivalant comparisons.
deleted 4y ago
[deleted]
pjmlp 4y ago
Which in case someone has failed to notice, is pretty much how serverless happens to work out from development point of view. No one cares how the language runtime is actually running.

kasperni 4y ago

[2017]

JStanton617 4y ago

This paper references consistently mischaracterizes AWS Lambda as a "Container as a Service" technology, when in fact it is exactly the sort of lightweight VM that they are describing - https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...

Jtsummers 4y ago
In fairness to this paper, it was written and published before that Firecracker article (2017 vs 2018). From another paper on Firecracker providing a bit of history: > When we first built AWS Lambda, we chose to use Linux containers to isolate functions, and virtualization to isolate between customer accounts. In other words, multiple functions for the same customer would run inside a single VM, but workloads for different customers always run in different VMs. We were unsatisfied with this approach for several reasons, including the necessity of trading off between security and compatibility that containers represent, and the difficulties of efficiently packing workloads onto fixed-size VMs. And a bit about the timeline: > Firecracker has been used in production in Lambda since 2018, where it powers millions of workloads and trillions of requests per month. https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
- runnerup 4y ago
  Thank you for this detail!
wodenokoto 4y ago
For what it’s worth, Google’s cloud functions are a container service. You can even download the final docker container.
- raggi 4y ago
  KVM gVisor is a hybrid model in this context. It shares properties with both containers and lightweight VMs.
deleted 4y ago

GekkePrutser 4y ago

Sometimes the less strict separation is a feature, not a bug. Without folder sharing with dockers for example, it would be pretty useless.

1MachineElf 4y ago
While a flawed comparison, WSL does use a VM in conjunction with the 9p protocol to achieve folder sharing.
- liftm 4y ago
  9p-based folder sharing is (used to be?) possible with qemu, too.
  - ksbrooksjr 4y ago
    It looks like it still is supported [1]. I noticed while reading the Lima documentation that they're planning on switching from SSHFS to 9P [2]. [1] https://wiki.qemu.org/Documentation/9psetup https://wiki.qemu.org/Documentation/9psetup [2] https://github.com/lima-vm/lima/blob/3401b97e602083cfc55b34e16d5ae8f119588397/docs/mount.md#9p https://github.com/lima-vm/lima/blob/3401b97e602083cfc55b34e...
  - ParetoOptimal 4y ago
    Yep, can confirm. NixOS configurations can be built and run as a qemu VM. I very recently was using one and found they automatically make a 9p mount to /tmp.

r3mc0 4y ago

Containers and VMs are totally not the same thing. They serve a complete other purpose , as multiple containers can be combined to create an application/service , VMs always use a complete os etc etc anyway the internet is full of the true purpose of containers , they were never meant to use as a "VM" and about security.. meh everything is insecure until proven differently

wongarsu 4y ago
VMs can have private networks between each other just as containers do. That's pretty much what EC2 is about.
- kazen44 4y ago
  heck, when running something in production with multiple underlying servers, you need a system that builds an overlay network regardless if you use containers or Vm's..
nijave 4y ago
VMs don't need a full OS. You can run a single process directly from the kernel with no init system or other userland
R0flcopt3r 4y ago
Maybe you should read the article? The article basically covers running a tiny kernel that as exactly what you need to run one single binary, which is your application. No printer drivers, video drivers. Probably have network drivers, as that would be quite handy. Kinda up to you on what kernel you pick for this, and what drivers are included. Multiple VMs with networking can also be "combined"... Containers are great because they are small and fast, low overhead etc. And that is why they replaced virtual machines. This article however proves that this is also achievable with virtual machines.

oxfordmale 4y ago

Kubernetes says no... The article is light on detail. Containers and VMs have different use cases. If you self host lightweight VMs is likely the better path, however, once you in the cloud most managed services only provide support for containers.

lrvick 4y ago
You can also use a firecracker runner in k8s to wrap each container in a VM for high isolation and security.
nailer 4y ago
> in the cloud most managed services only provide support for containers. Respectfully, comments like these are the reason for Kubernetes becoming a meme.
- oxfordmale 4y ago
  There is a huge difference running on VMs that you have zero access to, and actually owning your own VM infrastructure. Yes AWS Lambda runs on Firecracker, however, it could as well running on a FireCheese VM platform and you would be none the wiser, unless AWS publishes this somewhere. I am also not running on Kubernetes, because Kubernetes. AWS ECS and AWS Batch also only handle containerised applications. Even when deploying on EC2 I tend to use containers, as it ensures they keep working consistently if you apply patches to your EC2 environment.

bongobingo1 4y ago

I'm quite interested in seeing where slim VM's go. Personally I don't use Kubernetes, it just doesn't fit my client work which is nearly all single-server and it makes more sense to just run podman systemd units or docker-compose setups. So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close). Honestly I don't really care about containers vs VMs except the VM alleges better security which is nice, and I guess I like poking at things but they're were a little too rough for weekend poking. Is anyone doing "small scale" lightweight vm deployments - maybe just in your homelab or toy projects? Have you found the experience better than containers?

xani_ 4y ago
They went to trash because containers are more convenient to use and saving few MBs of disk/memory is not what most users care. The whole idea was pretty much either use custom kernel (which inevitably have way less info on how to debug anything in it), and re-do all of the network and storage plumbing containers already do via the OS they are running one. OR just very slim linux one which at least people know how to use but STILL is more complexity than "just a blob with some namespaces in it" and STILL requires a bunch of config and data juggling between hypervisor and VM just to share some host files to the guest. Either way to get to the level of "just a slim layer of code between hypervisor and your code" you need to do a quite a lot of deep plumbing and when anything goes wrong debugging is harder. All to get some perceived security and no better performance than just... running the binary in a container. It did percolate into "slim containers" idea where the container is just statically compiled binary + few configs and while it does have same problems with debuggability, you can just attach sidecart to it I guess next big hype will be "VM bUt YoU RuN WebAsSeMbLy In CuStOm KeRnEl"
- evol262 4y ago
  Virtualization is not just "perceived" security over containerization. From CPU rings on down, it offers dramatically more isolation for security than containerization does. This isn't about 'what most users care' about either. Most users don't really care about 99% of what container orchestration platforms offer. The providers do absolutely care that malicious users cannot punch out to get a shell on an Azure AKS controller or go digging around inside /proc to figure out what other tenants are doing unless the provider is on top of their configuration and regularly updates to match CVEs. "most users" will end up using one of the frameworks written by a "big boy" for their stuff, and they'll end up using what's convenient for cloud providers. The goal of microvms is ultimately to remove everything you're talking about from the equation. Kata and other microvm frameworks aim to be basically jsut another CRI which removes the "deep plumbing" you're talking about. The onus is on them to make this work, but there's an enormous financial payoff, and you'll end up with this whether you think it's worthwhile or not.

opentokix 4y ago

Tell me you don't understand containers wihout telling me you dont understand containers.

anthk 4y ago
You don't understand vm's neither. Ever used virtual network interfaces?

ricardobeat 4y ago

How does LightVM compare to Firecracker VMs? Could it be used for on-demand cloud VMs?

jupp0r 4y ago

(2017)

jjtheblunt 4y ago

"orders of magnitude" : Why does anyone ever write "two orders of magnitude" when 100x is shorter? Of course, this presumes 10 as the magnitude and the N orders to be the exponent, but I don't think I've ever, since the 90s, seen that stilted phrasing ever used for a base other than 10.

IshKebab 4y ago
Because two orders of magnitude does not mean 100x. It means on the same order as 100x.
- jjtheblunt 4y ago
  Do you mean folks using the phrase know big-O, big-omega, big-theta, and are thinking along those lines?
  - IshKebab 4y ago
    It's nothing to do with big-O; it's about logarithms. But really I think most people using it just think of it like: "which of these is it closest to? 10x, 100x or 1000x?"

deleted 4y ago

[deleted]

Sohcahtoa82 4y ago

> We achieve lightweight VMs by using unikernels When I attended Infiltrate a few years ago, there was a talk about unikernels. The speaker showed off how incredibly insecure many of them were, not even offering support for basic modern security features like DEP and ALSR. Have they changed? Or did the speaker likely just cherry-pick some especially bad ones?

sieabah 4y ago
The headline reads like a reddit post so I'm going to assume the same still holds true.
eyberg 4y ago
You are probably talking about this: https://research.nccgroup.com/wp-content/uploads/2020/07/ncc_group-assessing_unikernel_security.pdf https://research.nccgroup.com/wp-content/uploads/2020/07/ncc... In short - not a fundamental limitation - just that kernels (even if they are small) have a ton of work that goes into them. Nanos for instance has page protections, ASLR, virtio-rng (if on GCP), etc.
- OrvalWintermute 4y ago
  I thought that presentation was a little like looking at a hobby OS of a type, then attempting to draw security conclusions for all of that type. The NanoVMs unikernel for example, is pretty small, DoD supported, and has: ASLR Stack Randomization Heap Randomization Library Randomization Executable Randomization Page Protections Stack Execution off by Default Heap Execution off by Default Null Page is Not Mapped Stack Cookies/Canaries Rodata no execute Text no write STIG

devmor 4y ago

Yes, when you custom engineer a specific, complex solution for a specific use case, it is generally more performative than a general-use solution that's simple.

gavinray 4y ago

The issue with unikernels and things like Firecracker are that you can't run them on already-virtualized platforms I researched Firecracker when I was looking for an alternative to Docker for deploying FaaS functions on an OpenFaaS-like clone I was building It would have worked great if the target deployment was bare metal but if you're asking a user to deploy on IE an EC2 or Fargate or whatnot, you can't use these things so all points are moot This is relevant if you're self-hosting or you ARE a service provider I guess. (Yes, I know about Firecracker-in-Docker, but I mean real production use)

gamegoblin 4y ago
This is a limitation of whatever virtualized instance you're running on, not Firecracker itself. Firecracker depends on KVM, and AWS EC2 virtualized instances don't enable KVM. But not all virtualized instance services disable KVM. Obviously, Firecracker being developed by AWS and AWS disabling KVM is not ideal :) Google Cloud, for instance, allows nested virtualization, IIRC.
- verdverm 4y ago
  Ive used GCP nested virtualization. You pay for that overhead in performance so I wouldn't recommend it without more investigation. We were trying to simulate using luks and the physical is key insert / removal. Would have used it more if we could get GPU passthrough working
- shepherdjerred 4y ago
  Azure and Digital Ocean allowed nested virt as well!
- gavinray 4y ago
  Yeah but imagine trying to convince people to use an OSS tool where the catch is that you have to deploy it on special instances, only on providers that support nested virtualization Not a great DX, haha I wound up using GraalVM's "Polyglot" abilities alongside it's WASM stuff

wyager 4y ago

It's not clear to me that VMs actually do offer better isolation than well-designed containers (i.e. not docker). It's basically a question of: do you trust the safety of kernel-mode drivers (for e.g. PV network devices or emulated hardware) for VMs, or do you trust the safety of userland APIs + the limited set of kernel APIs available to containers. On my FreeBSD server, I kind of trust jails with strict device rules (i.e. there are only like 5 things in /dev/) over a VM with virtualized graphics, networking, etc.

nijave 4y ago
I think it gets even more complicated with something like firecracker where they recommend you run firecracker in a jail (and provide a utility to set that up)

xahrepap 4y ago

This reminds me: in 2015 I went to Dockercon and one booth that was fun was VMWare's. Basically they had implemented the Docker APIs on top of VMWare so that they could build and deploy VMs using Dockerfiles, etc. I've casually searched for it in the past and it seems to not exist anymore. For me, one of the best parts of Docker is building a docker-image (and sharing how it was done via git). It would be cool to be able to take the same Dockerfiles and pivot them to VMs easily.

verdverm 4y ago
They might have built it into Google Anthos as part of their partnership. I recall seeing a demo where you could deploy & run any* VMWare image on Kubernetes without any changes
All4All 4y ago
Isn't that essentially what Vagrant and Vagrantfiles do?
- xahrepap 4y ago
  Yes, which is what I'm using now. But it doesn't use the Docker APIs to allow you to (mostly) reuse a dockerfile to build a VM or a container. not sure if it would be better than Vagrant. But it was still very interesting.
- hinkley 4y ago
  What is your theory for why Docker won and Vagrant didn't? Mine is that all of the previous options were too Turing Complete, while the Dockerfile format more closely follows the Principle of Least Power. Power users always complain about how their awesome tool gets ignored while 'lesser' tools become popular. And then they put so much energy into apologizing for problems with the tool or deflecting by denigrating the people who complain. Maybe the problem isn't with 'everyone'. Maybe Power Users have control issues, and pandering to them is not a successful strategy.
  - pphysch 4y ago

lamontcg 4y ago

Containers should really be viewed as an extension of packages (like RPM) with a bit of extra sauce with the layered filesystem, a chroot/jail and cgroups for some isolation between different software running on the same server. Back in 2003 or so we tried doing this with microservices that didn't need an entire server with multiple different software teams running apps on the same physical image to try to avoid giving entire servers to teams that would be only using a few percent of the metal. This failed pretty quickly as software bugs would blow up the whole image and different software teams got really grounchy at each other. With containerization the chroot means that the software carries along all its own deps and the underlying server/metal image can be managed seperately, and the cgroups means that software groups are less likely to stomp on each other due to bugs. This isn't a cloud model of course, it was all on-prem. I don't know how kubernetes works in the cloud where you can conceivably be running containers on metal sharing with other customers. I would tend to assume that under the covers those cloud vendors are using Containers on VMs on Metal to provide better security guarantees than just containers can offer. Containers really shouldn't be viewed as competing with VMs in a strict XOR sense.

nikokrock 4y ago
I don’t remember where i read it but as far as i know when using Fargate to run containers (with k8s or ecs) AWS will just allocate an ec2 instance for you. Your container will never run on the same vm as other customer. This explain i think the latency you can have to start a container. To improve that you need to handle your own ec2 cluster with an autoscaling group
pjmlp 4y ago
Back in 2000 we were doing this with HP-UX Vaults, IT keeps going in circles.
- kazen44 4y ago
  ASlso, docker to me seems quite leaky in terms of abstractions (especially the networking side) compared to something like a freeBSD jail. why does docker networking need to be infinitely complex? freebsd solved this issue far more cleanly two decades ago without having to emulate entirely new networking spaces.

faeriechangling 4y ago

What I see happening now on the cloud is containers from different companies and different security domain running on the same VM. I have to think this is fundamentally insecure and that VMs are underrated. I hear people advocate QubesOS for security which is based on XEN when it comes to running my client. They say my banking should be done in a different VM than my email for instance. Well if that’s the case, why do we run many containers doing different security sensitive functions on the same VM when containers are not really considered a very good security boundary? From a security design perspective I imagine hardware being exclusive to a person/organization, vms being exclusive to some security function, and containers existing on top of that makes more sense from a security function but we seem to be playing things more loosely on the server side.

bgm1975 4y ago
Doesn’t AWS use firecracker with its Fargate container service (and lambda too)?

djhaskin987 4y ago

Not surprising that VMs running unikernels are as nimble as containers, but not quite useful either, at least in general. Much easier to just use a stock docker image.

dirkg 4y ago

Why is a 5yr old article being posted now? If this were to catch on, it would've. I just dont see it being used anywhere. Having a full Linux kernel available is a major benefit that you lose, right?

deleted 4y ago

[deleted]

thomasjudge 4y ago

This is 5 years old. What's the current state of the art?