7 ms·
My VM is lighter (and safer) than your container (2017)
- jonahbenton 3y ago(2017)
- mrAssHat 3y agoIntegrate that with kubernetes and I'm sold.
- _joel 3y agoWhat, like https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/ ?
- d3m0t3p 3y agoI think it's from the AWS team, they made firecracker (micro VM) So it does exist. Funnily that's what fly does: take your container uncompress it to a full micro VM and run it on their infra
- deleted 3y ago[deleted]
- iamstan23 3y agofly.io uses Firecracker. Firecracker is Open Sourced with an Apache 2 license. It's faster than LightVM mentioned in the post. Firecracker also has containerd support (https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container...). There are a few ways to run Kubernetes with Firecracker, including FireKube.
- throwaway482945 3y agoIs it really faster? I thought firecracker boot times were something like 100ms. LightVM claims 2.3ms?
- tpetry 3y agoLightVM is stating a VM creation of 2.3ms while Firecracker states 125ms of time from VM creation to a working user space. So this comparing apples and oranges.
- fhuici 3y agoThe 125ms is using Linux. Using a unikernel and tweaking Firecracker a bit (on KraftCloud) we can get, for example, 20 millis cold starts for NGINX, and have features on the way to reduce this further.
- imtringued 3y agoI know it's cool to talk about these insane numbers, but from what I can tell people have AWS lambdas that boot slower than this to the point where people send warmup calls just to be sure. What exactly warrants the ability to start a VM this quickly?
- fhuici 3y agoBack when we did the paper, Firecracker wasn't mainstream so we ended up doing a (much hackier) version of a fast VMM by modifying's Xen's VMM; but yeah, a few millis was totally feasible back then, and still now (the evolution of that paper is Unikraft, a LF OSS project at www.unikraft.org). (Cold) boot times are determined by a chain of components, including (1) the controller (eg, k8s/Borg), (2) the VMM (Firecracker, QEMU, Cloud Hypervisor), (3) the VM's OS (e.g., Linux, Windows, etc), (4) any initialization of processes, libs, etc and finally (5) the app itself. With Unikraft we build extremely specialized VMs (unikernels) in order to minimize the overhead of (3) and (4). On KraftCloud, which leverages Unikraft/unikernels, we additionally use a custom controller to optimize (1) and Firecracker to optimize (2). What's left is (5), the app, which hopefully the developers can optimize if needed.
- crabbone 3y agoThere are NVidia's Kata containers: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html https://docs.nvidia.com/datacenter/cloud-native/gpu-operator... . I'm not sure you need the physical GPUs to run them though. Most likely not. I'm wondering though what value will Kubernetes add beside integrating with existing (presumably Kubernetes-based) infrastructure? At least, this is my understanding of the rationale for Kata containers. Other than that, it seems like it'd be just getting in the way...
- bjconlan 3y agoI believe this work originated at Intel as "clear containers" (which I believe started life from an acquisition (but could be mixing this up...my memory isn't what it used to be). Either way it's great they are being used like this and at Nvidia (I know Alibaba cloud also use this tech)
- fhuici 3y agoYes, Kata started as clear containers. And yes, the main purpose is compatibility with containers -- though generally speaking, adding layers to the cloud stack never helps to make a deployment more efficient. On kraft.cloud we use Dockerfiles to specify app/filesystem, but then at deploy time automatically and transparently convert that to a specialized VM/unikernel for best performance.
- basemi 3y agoSe also: https://katacontainers.io/ https://katacontainers.io/
- nderjung 3y agoWe ended up doing this over at https://unikraft.io https://unikraft.io :-)
- jsheard 3y agoFrom 2017, before rootless containers caught on I think. The conclusions on safety might be due for re-evaluating.
- deleted 3y ago[deleted]
- Aardwolf 3y agoWhy weren't containers rootless from the start anyway? What did they need that user space doesn't provide? Wine, emulators and VMs didn't require it either (with the exception of some VMs needing a kernel module for performance reasons like memory management, which I also find stupid, the OS should provide all the performance in user space).
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- BobbyTables2 3y agoYou mean for the container launcher to not require root? Root isn’t required — look at Podman. Docker spent so much on marketing, the world is too blind to pivot to a superior alternative!
- zokier 3y agoPeople were running containers for a decade before rootless podman came around. There has been lot of sharp corners around userns and related tech that needed to get resolved. Notably Debian& Ubuntu disabled unprivileged userns for some legitimate security concerns
- Aardwolf 3y agoI really don't get that: having to run something substantial as root seems a much bigger security concern, than what it is shielding from user space (example: hosting a web server at port 80)
- _joel 3y agoShould be tagged with [2017]
- andix 3y agoI think containers are often misunderstood: The main benefit is not isolation and security, it's defined and reproducible environments and builds. If there is some additional isolation required, just run the container in a VM.
- brabel 3y agoBut if you can get isolation, security AND reproducible environments using a VM, specially one that's nearly as fast as a OS process, the case for using containers instead pretty much disappears. I don't know this LiteVM thing but I will definitely investigate that, specially given that on my Mac I need to use a VM anyway to run containers!
- hmottestad 3y agoI wouldn't discount the massive user base of containers among developers.
- alickz 3y agoi think at that point it's kind of just semantics i don't think devs care if they use containers or VMs, as long as it's easy and they don't have to worry about which version of Python the host is running
- jackcviers3 3y agoThis. It's why vagrant was popular before the container revolution. The killer app of Docker isn't the container, it's the depth and uniformity of the UX surrounding the container system. When that is broken by something on the host (non x86 cpu was a major pain for a while before popular images were x-built) and emulation gets in the way and is not as easy, or just mildly different (windows behind corporate firewalls that assign ips used by the docker engine for example), the ease of use falls away for non-power users and it's all painful again. Tech like Docker for windows and Rancher Desktop and lima has largely matured at this point, but somebody could make a new machine and then the process of gradual improvement starts all over again.
- 3y ago
- deleted 3y ago[deleted]
- codedokode 3y ago"VM" means it has its own kernel? Why have 2 kernels on the same machine? All processes in a proper OS are already isolated and there is no need for VM.
- Cthulhu_ 3y agoIsolated, but are they isolated enough? The article states that containers offer weaker isolation than VMs. (it doesn't quantify it though and I don't know this kind of thing offhand)
- codedokode 3y agoProcesses run in a userspace and cannot do anything without OS approval.
- akdev1l 3y agoThe kernel can be attacked and exploited. Container escape exploits are more common than VM escape exploits.
- kevincox 3y agoThe Linux kernel has far too large of an attack surface to be trusted as a hard security boundary. It is good enough to prevent mostly trusted software from accidentally interfering with each other but I would not trust it to protect me from an untrusted workload. For example GCP and AWS both have container running services. They both use hardware VMs to isolate different tenants. You will never share a kernel with another customer (I don't even think you will share one with yourself by default).
- codedokode 2y agoMaybe you need a better kernel then? For example, a microkernel.
- fhuici 3y agoI agree with the other comments. On the cloud, the VM is still the golden standard for strong (hardware-level isolation): if you deploy a container in the cloud, you can almost be sure there's a VM underneath. Given this, what we tried to do in that paper, in the LF Unikraft project (www.unikraft), and on kraft.cloud, is ensure that each VM only has the thinnest possible layer between the application and the hypervisor underneath -- strong isolation and hopefully max efficiency. We do use Dockerfiles to have users specify the app/filesystem, but then we transparently convert them to unikernels (specialized VMs) at deploy time.
- nashashmi 3y agoStupid question but forgive me: Whats the difference between a container and a VM?
- deleted 3y ago[deleted]
- akdev1l 3y agoA VM is a virtualized instance with virtual hardware and can therefore run its own operating system with its own kernel to interface with the virtual hardware. A container is basically a process restricted by multiple kernel namespace isolation mechanisms. It shares the same kernel with the host and does not present any “virtual hardware”.
- volkadav 3y agohigh level, a vm is an entire virtual machine with its own kernel/operating system/filesystem/etc. a container is a process (and associated files/archived filesystem) with a (more or less) isolated view of the world (network/filesystem/etc.) running on top of the same kernel/os as other processes on the same machine. examples: a) vm - an entire windows install running in a window on my linux workstation so i can use tax software once a year. two kernels running at the same time. (N+1 for N VMs) b) container - a small python service, its dependencies, and various filesystem bits from alpine-minimal packaged into a file that docker/containerd/whatever can turn into the service running in a little isolated portion of my machine. no matter how many i run, one kernel. the various processes just don't see the host or other procs' files/memory/etc. via namespace trickery (unless there's a security problem, lol)
- belter 3y agoTechnically, and simplifying enormously, the VM emulates the whole machine while the Container scopes the OS process. I prefer the analogy of an office building. Your VM is your whole office building and overnight maybe a whole new company can move in but still using the whole building. Your Container is a set of rules, somebody told you when arrived to the reception desk. About where is the only office in the building you can use, plus maybe some common access to shared areas once in a while, like WC and Kitchen. :-)
- sanjay_0508 3y ago[dead]
- epr 3y agoNo mention of user namespaces whatsoever, which is the primary security isolation mechanism for containers on linux. This is what enables "rootless" mode. Of course, this is from 2017, but user namespaces were released with linux 3.8 in February 2013. Docker particularly has always required extra work to run in rootless mode because it was released soon after in March 2013, and for whatever reason it hasn't been a priority to rework the codebase to make that the default. I switched to podman for exactly this reason as my go-to oci implementation and haven't looked back. The linux kernel features that enable various forms of isolation all require root privileges (CAP_SYS_ADMIN). Once user namespaces were a thing, that allowed you to use user namespaces to get around the root requirements for all the other isolation namespaces. All of the below still require CAP_SYS_ADMIN: CLONE_NEWCGROUP: cgroup namespace, for resource control (mem/cpu/block io/devices/network bandwith) CLONE_NEWIPC: ipc namespace for sysv ipc objects and message queues CLONE_NEWNET: network namespace, for isolated virtual networking CLONE_NEWNS: mount namespace, for isolated mounting (filesystems, etc.) CLONE_NEWPID: pid namespace, for isolated view of running processes CLONE_NEWUTS: unix timesharing system namespace, for isolation of hostname and domain name see: https://man7.org/linux/man-pages/man2/clone.2.html https://man7.org/linux/man-pages/man2/clone.2.html
- ledgerdev 3y agoThanks for the suggestion, I was unaware of podman and will be trying it out because root has always bothered me.
- angra_mainyu 3y agoPodman is fantastic! It can also handle kubernetes and generate systemd units for containers.
- ledgerdev 2y ago> generate systemd units for containers Oh man that is something I've been wanting as well.
- sureglymop 3y ago
- zerof1l 3y agoDocker has a big community, lots of guides, and ready-to-use containers. It became pretty much a de facto standard for self-hosting things. You also have a very high chance of getting a piece of software to work out of the box as intended with Doker. The only way this or some other way of running stuff will overtake Doker is if it will match the Docker in these aspects. As much as I'd love to try this lightweight VM idea, I don't have the time or energy to convert 20+ projects I'm self-hosting into this and then keep everything updated. I'd rather invest this time into learning Docker more and making my existing setup more secure and robust.
- k8sToGo 3y agoEspecially with something like linuxserver.io
- jeltz 3y agoThe ready-to-use is only true for the most popular software like PostgreSQL. Anything else is often broken and/or unmaintained.
- gbalduzzi 3y agoIn web development I had never experienced an issue with the commonly used softwares
- jve 3y agoAhem, this is a research paper. You should look at this stuff as "Innovation" and someone may just consider building a tool or product on the idea... or not.
- fhuici 3y agoAuthor here, we did this, first by continuing the research alongside the creation of the Unikraft LF OSS project -- the result of which was the Eurosys 2021 best paper award (https://dl.acm.org/doi/10.1145/3447786.3456248 https://dl.acm.org/doi/10.1145/3447786.3456248). Commercially, we leverage Unikraft on kraft.cloud to provide a cloud platform with millisecond semantics.
- datadeft 3y agoIs LightVM actively developed or used? The repo is empty.
- koprulusector 3y agoFor those that didn’t take the time to read, this is about unikernels
- uticus 3y agoOS: I provide isolation where needed, handle safely interacting with outside world, and abstract away all the pesky stuff so programmers can just get stuff done. Container / VM: I provide isolation where needed, handle safely interacting with outside world, and abstract away all the pesky stuff so programmers can just get stuff done. I get that a dev machine (OS) isn't usually suitable for deployment or shared development (Container/VM). But seems to me the promise of the Operating System has fallen short, if we are striving to meet so many of the same goals of the OS, with something on the OS that tries to abstract away the OS.
- vilunov 3y agoI guess this came to be due to the poor original security model of classic OSs, which led to prolification of viruses and complex management of shared resources. Users, groups and access flags are not enough to manage security of a system. Linux tried to fix that with namespaces and it turned out to be more or less successful, but Linux is not an OS, it's just a kernel, and it's up to real OSs built atop Linux to use namespaces as an implementation detail for real application isolation. One way to do that is OCI-containers, the other way is Flatpak. Neither of those is not a proper OS yet, but you could call Kubernetes an operating system which uses containers as means for application and resource isolation. Naturally that means Kubernetes is a complex beast, but that's what it takes to provide what users expect from an OS. Android also comes to mind, they managed to isolate applications between each other quite safely.
- clan 3y agoI say this with great care as I do not want to launch a flamewar. If you do not consider Linux with namespaces an OS (because of fragmented userland): Would you then consider FreeBSD with jails or Solaris with zones for fully fledged? If you still consider those flawed (maybe because thet do not force you into jails/zones) should we at least no consider OS/390 or z/OS as proper operating systems to that/your (not meant inflamatory!) standard? Yes. Though you do not mention them directly DOS and Windows has ruled the world for years and they opened the door for the nasties. But they were not all there was - only the popular/easy choice. Everything is a trade off.
- flemhans 3y agoDid something come out of it? What's the best way to run thousands of superlight VMs in 2024? WebAssembly?
- wg0 3y agoLooking around Kraftcloud, it looks like fly.io but vocabulary is lot simpler and I can totally see it as a viable upcoming PaaS player. Seems like you can run state full workloads too.
- macspoofing 3y ago>On the downside, containers offer weaker isolation than VMs, to the point where people run containers in virtual machines to achieve proper isolation. That's not really why containers are deployed in VMs, especially in the context of on-prem enterprise software. I think that's more of a legacy issue. For example, for on-prem enterprise software, the enterprise already invested millions into their VM infrastructure so deploying a containerized stack means deploying into their VM infrastructure. I think when centralized container orchestrators get enough market penetration with properly trained IT, you'll probably see that change. Also, very few people choose containers for security and isolation. Typically it's for flexibility in deployment, and control of the environment (no more dependency hell).
- TrueDuality 3y agoThere is a lot of discussion on here about the different isolation levels available, but these micro-VMs aren't playing in the same field and can't be compared apples-to-apples. If you go read the paper this requires a specialized Xen kernel, which in turn requires processor virtualization extensions directly available where you're running these containers. Those extensions aren't generally available if you're already running inside of a VM. This is a solution that only works on bare metal which I would bet money the vast majority of people using containers, outside of development environments at least, are not running their containers in bare metal but in an existing VM such as on AWS or GCP where this solution is simply a non-starter. Neat, niche, and doesn't operate in the same world as containers.
- deleted 3y ago[deleted]
- hi-v-rocknroll 3y agoContainers seem light and cheap, but they have subtle problems lacking solid guarantees, prioritization, or limits on compute, network, and storage resources that type-1 v12n provides.
- 1vuio0pswjnm7 2y agoFirecracker micro VM uses musl not glibc. Is it true.