17 ms·
Containers in 2019: They're Calling It a Hypervisor Comeback
- pjmlp 7y agoJust waiting for the new re-discovery of hypervisors, but with better marketing names.
- pojntfx 7y ago"Workload Orchestrators" K8s can already do VMs with Kube-virt, so yeah.
- tyingq 7y agoK8S is unnecessarily complicated. I fully expect "serverless", warts and all, to take all comers. And, I get the irony. It's basically cgi-bin 2.0. It will win not because it is better, but because it is better "understood".
- pjmlp 7y agoThe irony is that now we have Azure services to keep state across serverless requests. Excuse me while I go implement a servlet over there....
- meddlepal 7y agoServerless has been around now for several years and it hasn't taken off yet... definitely not to the same degree containers have. I'm skeptical it will. To adopt serverless you need to be willing to rearchitect your product and retool your developers... that's expensive.
- kazen44 7y agocontainers are still relatively new and definetly not as ubiquitous as virtual machines. Virtual machines are everywhere, from SME to large enterprises, tech hubs, industrial software et al. Containers are mainly used for modern web and app development.
- Havoc 7y ago>(Serverless) I'm skeptical it will. They'd get more uptake if it was easier I think. Not re-architect your product...but small things here and there. I wanted to play with azure python functions but despite vs enterprise & lots of credits I can't. Without admin rights on local machine it's basically impossible. (Need VSCode & AZ toolkit) (Unrelated - that kinda blew my mind - no you can't do that in the 2,000 USD VS enterprise...you need to use the free one)
- snazz 7y ago> (Unrelated - that kinda blew my mind - no you can't do that in the 2,000 USD VS enterprise...you need to use the free one) I’m guessing this is because Microsoft wants it to be more accessible—they probably realize that there isn’t much money to be made in $2,000 developer tools. Visual Studio was never a Python IDE; VS Code is much more language-agnostic.
- Havoc 7y agoSure, but the whole "this feature is impossible in our 2000usd suit but its possible in our free one" doesn't seem strange to you? If I'm buying the top end product I'm expecting full feature set, no?
- alchemism 7y agoThe rabbit hole goes deeper than that, my friend. https://cloud.google.com/run/ https://cloud.google.com/run/
- aloknnikhil 7y agoOh God, why? Btw, this uses gVisor (https://gvisor.dev/ https://gvisor.dev/) as the container runtime (instead of runc/containerd). That seriously limits what syscalls the containers can leverage. And the pricing is complicated too.
- regularfry 7y agoCGI-bin 2.0 is actually quite neat: having your script kick off on as many machines as needed is something CGI-bin 1.0 never cracked on any significant scale, and (eventually) having something like a standard interface to do that on Someone Else's Datacenter will be fun.
- sedachv 7y ago> And, I get the irony. It's basically cgi-bin 2.0. More like "fancy inetd."
- panpanna 7y agoBare-metal containers. No, that was even worse.
- ahartmetz 7y agoServerless servers?
- PopeDotNinja 7y agoFederated subsystem authoritator
- hinkley 7y agoAnd let me state again for the record that all of these promises being made by container systems sound an awful lot like the promises I was offered by 'real' operating systems in the early nineties. I think the only real difference is that there has been a sea change in public opinion on this kind of aggressive isolation by default being worthwhile. But a hypervisor publishing a bunch of services that talk to the world and each other? Things are beginning to look a bit more like microkernels as time goes on.
- BiteCode_dev 7y agoPlus now, the app devs, now all turned devops because talent is scarse and demand is high, get to handle the complexity of concurrency, networking and reliability. This adds to deployment/testing/development setup, and when you change stack or job, the architecture slightly different and you need to learn it all over again. Funnily enough, we couldn't make the microkernel concept works despite an homogeneous, integrated environment as a basis. But we are fine building frankenstein OS images, hacking multiple graph-shaped systems to make them play together, and call that a success. Now don't get me wrong, I understand perfectly the benefits of it. But somehow, it feels like we arrived at the right result by the worst possible path, and at a price way higher than it should be.
- panpanna 7y agoWhat if the pure microkernel approach is flawed and hypervisors offer a sensible middle ground?
- jacquesm 7y agoMost of the people that rail against pure microkernels have zero experience actually building stuff with them. Those that do have that experience have hard proof that systems built in this way are more reliable and easier to build than others. Hypervisors have their own place in the hierarchy, they do not compete effectively with microkernels on their own turf.
- 7y ago
- blablabla123 7y ago> The Docker engine default seccomp profile blocks 44 system calls today, leaving containers running in this default Docker engine configuration with just around 300 syscalls available. ...preventing devs/ops people to run tools like iotop, unless extra capabilities are added. I'm all in for containers, cgroups/namespaces but at the moment it's namespace isolation for the price of less features. Unless namespaces become first-class citizens in the Linux kernel, it will always be more efficient to just run on VMs or even Bare Metal. At least for non-planet scale workloads. :-)
- panpanna 7y agoNamespaces _are_ first class citizens in the kernel. Have you tried any solutions that are not based on docker?
- ris 7y ago> ...preventing devs/ops people to run tools like This is because docker makes the fundamental mistake of conflating packaging with isolation. "Packaging" is achieved in docker by using the OS to do an amount of sandboxing and then letting the user perform whatever non-reproducible crap they like before balling the whole thing up and calling it a package. If instead you make an app author actually figure out what their dependencies are and how to fetch/build them - in a system such as Nix, you not only get reproducible packages, you also get to decide to apply actual os-level isolation on a case by case basis - a developer doesn't necessarily need/want these barriers on their dev machine.
- panpanna 7y agoFor the record, hypervisor were introduced almost before operating systems. In the begining it was just a switch to partition memmory and run multiple jobs on a big expensive machine. So this is not the first time hypervisors are making a comeback.
- curt15 7y agoIs there still any place in 2019 for "system containers" like LXC/LXD?
- simosx 7y agoThey are quite useful as "VMs at the speed of application containers". When you use application containers, you tend to create a more complicated setup and lose visibility as to what's going on in the system. Just like you use VMs on a baremetal server, you use system containers in a VM. But I find more important to use LXD as a tool for software development and also for Linux desktop use. If you want to setup nodejs or something similar, it makes sense to put it in a LXD container so that it does not mess up your host. Each step of the workflow (creation, deletion, etc) takes a few seconds or less. You can also setup LXD containers to run GUI applications. It makes big sense if you want to run games (like Steam) so that your desktop does not get polluted with i386 packages. You can also use for development tools, such as those from JetBrains, Android Studio and more.
- bonzini 7y agoThey're certainly very useful for CI systems.
- josteink 7y agoThey may not be mainstream but I use them as isolated VM-like application environments, where everything “just works” without having to learn/apply a whole lot of new tools or workflows. I’m sure there must be others who see the benefit of this approach too?
- ris 7y agoI'm quite hopeful for the limited re-introduction of hypervisors to "containment", if only because I've become disappointed with the lack of power app authors have to specify fine-grained, strict security details. This is because most of the fun lockdown toys that linux has, selinux, seccomp et al aren't easily "stackable", and those toys have been used by the orchestrator to perform the containment. And that's great, but it means containers all end up with broad, generic policies (that only care about container breakout) which can't be further restricted by app/container authors. My hope is that lightweight virtualisation will give back the ability for app authors to tighten their container's straight jackets. Personally I've got my eye on kata containers.
- nunchuckninja 7y agoSo I read something succinct a while back - docker et al are distribution platforms not security platforms . They add "0" security against an adversary (think padlocks). How true is it ? And what high perf securely contained systems are there? OpenVZ?
- deleted 7y ago[deleted]
- upofadown 7y agoThe generic issue seems to be that stuff like containers can be escaped with pretty much any privilege escalation exploit ... and such exploits are reasonably common in the world of Linux.
- nunchuckninja 7y agoSo either take perf hit or don't expect isolation at all?
- dilyevsky 7y agoFor completely untrusted workloads basically - yeah. For semi-trusted, there’s lots of tech that provides reasonable, lightweight isolation. There’s no reason why hardware vendors cant ship products that are both virtualizable with high performance and secure, so that may still come.
- aritmo 7y agoBut if the container is unprivileged?
- naasking 7y agoSandstorm has had a pretty good track record [1]. Then again, it was designed and run by capability security folks, who take security pretty seriously. [1] https://sandstorm.io/ https://sandstorm.io/
- 7y ago
- kresten 7y agoKey point: AWS Firecracker does NOT run on AWS. Unless you want to pay for bare metal instances. AWS Firecracker DOES run on Google Cloud, Azure and Digital Ocean.
- jacques_chester 7y agoFirecracker is used as the runtime for Lambda, I believe.
- denton-scratch 7y agoI have a Xen hypervisor at home, running on a (well-configured) NUC. VMs boot in about 15 seconds. I'm patient! I'm not doing devops stuff; I no longer code, so I don't need a testing pipeline. I looked into containers based on LXC, when it was first introduced. I decided to stay away - I don't want to get tied into Poettering's code. Yeah, I'm running systemd on some of the VMs, but you don't really have much choice nowadays. I still use Debian, but I'm bitter about their adoption of systemd - and my hypervisor machine runs SysV Init, because I know how that works. I never mixed it with Docker. From what I've been reading, Docker is already old-hat (it's only about 3 years old; how did that happen?) Kubernetes seems to be the thing nowadays. I don't even know how to pronounce "Kubernetes". What was wrong with LXC? Like, LXC comes with the OS. Nothing to install. Why do people love these 3rd-party container engines? And "wrappers"? what purpose is served by container wrappers? Serious questions, I'm not trolling. Promise. I'm just a bit out-of-date.
- meddlepal 7y agoSome of us have problems that are actually well solved by these tools and run on more sophisticated hardware than our home computer... It doesn't sound like you've hit that set of problems.
- denton-scratch 7y agoYeah, thanks. It's a home computer because I don't work any more - I set-up and operated a continuous production VM pipeline on my boss's rack, back when I had a boss. Docker had turned up; I evaluated it (quite briefly - I had work to do, and nobody had asked me to evaluate Docker). We had fairly specific requirements that I couldn't match to Docker. I recall now that Kubernetes is principally a deployment system - sorry. Perhaps it is well-integrated with some container systems, such as Docker. Kubernetes didn't exist when I had a boss. I use Ansible. I get that LXC is not very friendly. But it works OK, doesn't it? Just wrap a shiny skin around it. Don't re-invent the wheel; we already have enough wheel inventions. I'm not by any means knowledgeable about containers; I evaluated LXC way back when, and decided that VMs were more secure. And never revisited that decision.
- 7y ago
- jacques_chester 7y ago"Containers" is an unfortunate term, since it really better describes the container image than an actual running process with API virtualisation. I think VMs-as-containers is where we'll wind up. The container image has turned out to be the real thing of interest, the runtime is almost secondary. Virtual machine systems have closed the performance gap in a variety of ways. For example: tearing out kernel checks for devices that will never be connected to the VM; taking advantage of hyper-privileged CPU instructions; being able to make and restore higher-fidelity checkpoints than an OS can for faster launches etc. At which point the isolation benefits of VMs really begin to outweigh everything else. A hypervisor has a much smaller attack surface and has a much simpler role than a full monolithic OS kernel. It partitions the hardware and that's about it. It doesn't exist in a constant tension between kernel-as-resource-manager and kernel-as-service-provider.
- LIV2 7y agoThis is kinda where VMware is going with project pacific & vSphere integrated containers; containers running as individual VMs on a hypervisor. I wonder if we will see others following the same pattern? I’m not sure what the drawbacks might be though
- jacques_chester 7y agoI think yes. Red Hat and others are working on KubeVirt, plus I believe there are CRI implementations for gVisor and Firecracker. Disclosure: I work for Pivotal, which is in the middle of acquisition by VMware.
- depr 7y agoYes and no, there's the attack surface of the guest operating system.
- dopylitty 7y agoI've had this sneaking but hard to articulate suspicion that datacenters, bare metal servers, VMs, operating systems, containers, OS processes, language VMs, and threads are all really attempts to abstract the same thing. You want to run business code in a way that's protected from other business code but also able to interact with other business code and data in a well defined way. I also have this sneaking suspicion that new generations are re-inventing the wheel in a lot of ways. If you have minimal containers running on a hypervisor how is that different from processes running on an operating system? You have all these CPU provided virtualization instructions to protect guests from each other and the host from the guests but there's no reason those instructions couldn't have been developed to protect processes from each other. You have indirections to protect one guest from accessing another's storage but there's no reason processes couldn't have the same protections (and they do in many operating systems). Why container orchestration and overlay networks instead of OS scheduling and IPC? I'm sure people in academic computer science have already published many a paper about this but it feels different seeing it from inside an IT organization where people don't seem to apply the lessons from older technologies to newer ones and we end up in this constant churn of reinvention which, as far as I can tell, is mostly a way for people, both in management and in the trenches, to keep their jobs, at least until you're "too old to learn new things" and pushed aside.
- BubRoss 7y agoThreads are not the same because they are separate computation in the same address space. At some point though, permissions to disk, network, screen, input devices, etc. need to be an integrated part of computing for anything that runs more than one program. Take a look at QNX though, at some point it will be reinvented and heralded as a breakthrough, 40 years after it happened the first time.
- dopylitty 7y agoThat's a good point. But as I think about it in a sense threads are also a form of isolation in that they allow you to isolate units of business code so that they can run independently and be (potentially) protected from other business code monopolizing a shared resource while not actually doing work.
- jasoneckert 7y agoWait....this isn't new - hypervisor-based container isolation has been in Windows Server since 2016 (it's called Hyper-V containers).
- sayhello 7y agoDid the Windows Server isolation bundle a slim kernel? One of the core elements of this for the Serverless fast boot requirement is using a kernel that has legacy modules excised.
- xfitm3 7y agoI see container systems, such as Docker, as a packaging system more than anything else.
- hinkley 7y agoA Turing complete, language agnostic one. If you were trying to write one that wasn’t container focused, how would that look?
- spion 7y agoThe future is probably WASI: a sandboxed compile target with a capabilities based security model not owned by a single corporation. https://wasi.dev https://wasi.dev
- amscanne 7y agoI’ve been thinking about these problems for a while. Previously, I thought that the “put a VM on it” approach was the right one. In 2015, I wrote novm [1], which I think served as inspiration for some developments that followed. My thinking has changed over the years and I actually work on gVisor today (disclaimer!). I’d like to share some thoughts here. Hypervisors never left. They are a fundamental building block for infrastructure and will continue to be. The question is whether there will be a broad shift to start relying on hypervisors to isolate every individual application. In my opinion, just wrapping containers in VMs is not a solution. (Nor do I find it technologically interesting, but that’s me.) I agree that the approach addresses some of the challenges of isolation, but is one step forward, two steps back in other ways. Virtualizing at the hardware boundary lets you do some things very well. For example, device state is simple, and hardware support lets you track dirty memory passively and efficiently, so you can implement live migration for virtual machines much better than you could for processes. It can divide big machines into fungible, commodity sizes (allowing applications from having to care about NUMA, etc.). It lets you pass though and assign hardware devices. It gives you a strong security baseline. But abstractions work best when they are faithful. Virtual machines operate on virtual CPUs, memory and devices, and operating systems work best when those abstractions behave like the real thing. That is, CPUs and memory are mostly available, and hardware acts like hardware (works independently, interactions don’t stall). Containers and applications operate on OS-level abstractions: threads, memory mappings, futexes, etc. These abstractions are the basis for container efficiency — not because startup time is fast, but because these abstractions allow for a lot of statistical multiplexing and over-subscription while still performing well. The abstractions provide a lot of visibility for the OS to make good choices with global information (e.g. informing the scheduler, reclaim policy, etc.). A problem arises when you decide that you want to bind single applications to single VMs, and then run many VMs instead of many containers. Effectively, the abstractions that you expose are now CPUs and memory, and these just don’t work as well for over-subscription and overall infrastructure efficiency. There’s no shared scheduler or cooperative synchronization (e.g. in an OS, threads waking each other will be moved to the same core), there’s no shared page cache, etc. There are other problems too: virtualization gives you a very strong security baseline, but you have to start punching significant holes to get the container semantics you want. E.g. the cited virtfs is a great example: it’s easy to reason about the state of a block device, but an effective FUSE api (and shared memory for metadata) is a much larger system surface. The hardware interface itself is not a silver bullet. Devices are still complex (escapes happen), and the last few years have taught us that even the hardware mechanisms can have flaws. For example, AFAIK Kata containers is still vulnerable to L1TF unless you’re using instance-exclusive cpusets or have disabled hyper-threading. (Whereas native processes and containers are not vulnerable to this particular bug.) The “put a VM on it” approach also may not have the standard image problems that plain hypervisors have, but you’ve got portability challenges. It seems non-ideal that a container isolation solution can run in infrastructure X and Y, but not in standard public clouds or your on-prem VMWare hosts, etc. (There might be specific technologies for each case, but that’s rather the point.) That’s my 2c. I’m pretty optimistic that we can have strong isolation while still preserving the efficiency, portability and features of container-based infrastructure. I like a lot of these projects (especially the ones doing technologically interesting things, e.g. nabla, x-containers, virtfs, etc.) but I don’t think the straight-up “put a VM on it” approach is going to get us there.