13 ms·

Unfashionably secure: why we use isolated VMs

• PedroBatista 2y ago
  As a permanent "out of style" curmudgeon in the last ~15 years, I like that people are discovering that maybe VMs are in fact the best approach for a lot of workloads and the LXC cottage industry and Docker industrial complex that developed around solving problems created by themselves or solved decades ago might need to take a hike. Modern "containers" were invented to make things more reproducible ( check ) and simplify dev and deployments ( NOT check ). Personally FreeBSD Jails / Solaris Zones are the thing I like to dream are pretty much as secure as a VM and a perfect fit for a sane dev and ops workflow, I didn't dig too deep into this is practice, maybe I'm afraid to learn the contrary, but I hope not. Either way Docker is "fine" but WAY overused and overrated IMO.
  • ganoushoreilly 2y ago
    Docker is great, way overused 100%. I believe a lot of it started as "cost savings" on resource usage. Then it became the trendy thing for "scalability". When home enthusiasts build multi container stacks for their project website, it gets a bit much.
    • applied_heat 2y ago
      Solves dependency version hell also
      • theLiminator 2y ago
        Solves it in the same sense that it's a giant lockfile. It doesn't solve the other half where updates can horribly break your system and you run into transitive version clashes.
        • bornfreddy 2y ago
          But at least you can revert back to the original configuration (as you can with VM, too).
        • Spivak 2y ago
          It solves it in the sense that it empowers the devs to update their dependencies on their own time and ops can update the underlying infrastructure fearlessly. It turned a coordination problem into a non-problem.
        • ktosobcy 2y ago
          Having been running a VPS with manually maintained services moving to docker saved me a lot of headache... things definitely breaks less often (almost never) and if they do it's quite easy to revert back to previous version...
      • sitkack 2y ago
        It doesn't solve it, it makes it tractable so you can use the scientific method to fix problems as opposed to voodoo.
    • Yodel0914 2y ago
      > When home enthusiasts build multi container stacks for their project website, it gets a bit much. I don't know - docker has been a godsend for running my own stuff. I can get a docker-compose file working on my laptop, then run it on my VPS with a pretty high certainty that it will work. Updating has also (to date) been incredibly smooth.
  • everforward 2y ago
    > Modern "containers" were invented to make thinks more reproducible ( check ) and simplify dev and deployments ( NOT check ). I do strongly believe deployments of containers are easier. If you want something that parallels a raw VM, you can "docker run" the image. Things like k8s can definitely be complicated, but the parallel there is more like running a whole ESXi cluster. Having done both, there's really only a marginal difference in complexity between k8s and an ESXi cluster supporting a similar feature set. The dev simplification is supposed to be "stop dealing with tickets from people with weird environments", though it admittedly often doesn't apply to internal application where devs have some control over the environment. > Personally FreeBSD Jails / Solaris Zones are the thing I like to dream are pretty much as secure as a VM and a perfect fit for a sane dev and ops workflow I would be interested to hear how you use them. From my perspective, raw jails/zones are missing features and implementing those features on top of them ends up basically back at Docker (probably minus the virtual networking). E.g. jails need some way to get new copies of the code that runs in them, so you can either use Docker or write some custom Ansible/Chef/etc that does basically the same thing. Maybe I'm wrong, and there is some zen to be found in raw-er tools.
  • dboreham 2y ago
    For me it's about the ROAC property (Runs On Any Computer). I prefer working with stuff that I can run. Running software is live software, working software, loved software. Software that only works in weird places is bad, at least for me. Docker is pretty crappy in most respects, but it has the ROAC going for it. I would love to have a "docker-like thing" (with ROAC) that used VMs not containers (or some other isolation tech that works). But afaik that thing does not yet exist. Yes there are several "container-tool, but we made it use VMs" (firecracker and downline), but they all need weirdo special setup, won't run on my laptop, or a generic Digitalocean VM.
    • 01HNNWZ0MV43FF 2y ago
      Yeah that's kind of a crummy tradeoff. Docker is "Runs on any Linux, mostly, if you have a new enough kernel" meaning it packages a big VM anyway for Windows and macOS VMs are "Runs on anything! ... Sorta, mostly, if you have VM acceleration" meaning you have to pick a VM software and hope the VM doesn't crash for no reason. (I have real bad luck with UTM and VirtualBox on my Macbook host for some reason.) All I want is everything - An APE-like program that runs on any OS, maybe has shims for slightly-old kernels, doesn't need a big installation step, and runs any useful guest OS. (i.e. Linux)
      • neaanopri 2y ago
        The modern developer yearns for Java
        • smallmancontrov 2y ago
          I had to use eclipse the other day. How the hell is it just as slow and clunky as I remember from 20 years ago? Does it exist in a pocket dimension where Moore's Law doesn't apply?
          • TurningCanadian 2y ago
            That's not Java's fault though. IntelliJ IDEA is also built on Java and runs just fine.
  • gryfft 2y ago
    I've been meaning to do a bhyve deep dive for years, my gut feelings being much the same as yours. Would appreciate any recommended reading.
    • Gud 2y ago
      Read the fine manual and handbook.
  • nimish 2y ago
    Clear Containers/Kata Containers/firecracker VMs showed that there isn't really a dichotomy here. Why we aren't all using HW assisted containers is a mystery.
    • turtlebits 2y ago
      Engineers are lazy, especially Ops. Until it's easier to get up and running and there are tangible benefits, people won't care.
    • tptacek 2y ago
      It's not at all mysterious: to run hardware-virtualized containers, you need your compute hosted on a platform that will allow KVM. That's a small, expensive, tenuously available subset of AWS, which is by far the dominant compute platform.
      • Spivak 2y ago
        So… Lambda, Fargate, and EC2. The only thing you can't really do this with is EKS. Like Firecracker was made by AWS to run containers on their global scale KVM, EC2.
        • tptacek 2y ago
          Lambda and Fargate are implementations of the idea, not a way for you yourself to do any kind of KVM container provisioning. You can't generally do this on EC2; you need special instances for it. For a variety of reasons, I'm pretty familiar with Firecracker.
          • Spivak 2y ago
            What I'm I missing? AWS offers (virtual) hardware backed containers as a service, I would go so far as to say that a significant number of people are running vm backed containers. And I've been at a few shops where EC2 is used as the poor-man's-firecracker by building containers and then running 1(ish) per VM. AWS's architecture actively encourages this because that's by far the easiest security boundary to manipulate. The moment you start thinking about two privilege levels in the same VM you're mostly on your own. The number of people running production workloads who, knowingly or not, believe that the security boundary is not between containers but between the vms enclosing those containers is probably almost everyone.
  • turtlebits 2y ago
    Honestly, it really doesn't matter whether it's VMs or Docker. The docker/container DX is so much better than VMWare/QEMU/etc. Make it easy to run workloads in VMs/Firecracker/etc and you'll see people migrate.
    • packetlost 2y ago
      I mean, Vagrant was basically docker before docker. People used it. But it turns out the overhead over booting a full VM + kernel adds latency which is undesirable for development workloads. The techniques used by firecracker could be used, but I suspect the overhead of allocating a namespace and loading a process will always be less than even restoring from a frozen VM, so I wouldn't hold my breath on it swinging back in VM's direction for developer workloads ever.
      • yjftsjthsd-h 2y ago
        It would be interesting to see a microvm (kata/firecracker/etc.) version of vagrant. And open source, of course. I can't see any technical reason why it would be particularly difficult.
        • mountainriver 2y ago
          Oh they exist! Several of them in fact, they have never picked up a ton of steam though
        • packetlost 2y ago
          I don't think they're that valuable tbh. Outside of cases where you're running completely untrusted code or emulating a different architecture, there's no strong reason to pick any VM over one of the several container paradigms.
          • yjftsjthsd-h 2y ago
            One more usecase - which I admit is niche - is that I want a way to run multiple OSs as transparently as possible. A microvm that boots freebsd in less than a second and that acts almost like a native container would be excellent for certain development work. Edit: Actually it's not just cross-OS work in the freebsd/linux sense; it would also be nice for doing kernel dev work. Edit my linux kernel module, compile, and then spawn a VM that boots and starts running tests in seconds.
  • compsciphd 2y ago
    As the person who created docker (well, before docker - see https://www.usenix.org/legacy/events/atc10/tech/full_papers/Potter.pdf https://www.usenix.org/legacy/events/atc10/tech/full_papers/... and compare to docker), I argued that it wasn't just good for containers, but could be used to improve VM management as well (i.e. a single VM per running image - seehttps://www.usenix.org/legacy/events/lisa11/tech/full_papers/Potter.pdf https://www.usenix.org/legacy/events/lisa11/tech/full_papers...) I then went onto built a system with kubernetes that enabled one to run "kubernetes pods" in independent VMs - https://github.com/apporbit/infranetes https://github.com/apporbit/infranetes (as well as create hybrid "legacy" VM / "modern" container deployments all managed via kubernetes.) - as a total aside (while I toot my own hort on the topic of papers I wrote or contributed to), note the reviewer of this paper that originally used the term Pod for a running container - https://www.usenix.org/legacy/events/osdi02/tech/full_papers/osman/osman.pdf https://www.usenix.org/legacy/events/osdi02/tech/full_papers... - explains where Kubernetes got the term from. I'd argue that FreeBSD Jails / Solaris Zones (Solaris Zone/ZFS inspired my original work) really aren't any more secure than containers on linux, as they all suffer from the same fundamental problem of the entire kernel being part of one's "tcb", so any security advantage they have is simply due lack of bugs, not simply a better design.
    • ysnp 2y ago
      Would you say approaches like gvisor or nabla containers provide more/enough evolution on the security front? Or is there something new on the horizon that excites you more as a prospect?
      • compsciphd 2y ago
        been out of the space for a bit (though interviewing again, so might get back into it), gvisor at least as the "userspace" hypervisor, seemed to provide minimal value vs modern hypervisor systems with low overhead / quick boot VMs (ala firecracker). With that said, I only looked at it years ago, so I could very well be out of date on it. Wasn't aware of Nabla, but they seem to be going with the unikernel approach (based on a cursory look at them). Unikernels have been "popular" (i.e. multiple attempts) in the space (mostly to basically run a single process app without any context switches), but it creates a process that is fundamentally different than what you develop and is therefore harder to debug. while the unikernels might be useful in the high frequency trading space (where any time savings are highly valued), I'm personally more skeptical of them in regular world usage (and to an extent, I think history has born this out, as it doesn't feel like any of the attempts at it, has gotten real traction)
  • anonfordays 2y ago
    >Personally FreeBSD Jails / Solaris Zones are the thing I like to dream are pretty much as secure as a VM and a perfect fit for a sane dev and ops workflow, I didn't dig too deep into this is practice, maybe I'm afraid to learn the contrary, but I hope not Having run both at scale, I can confirm and assure you they are not as secure as VMs and did not produce sane devops workflows. Not that Docker is much better, but it is better from the devops workflow perspective, and IMHO that's why Docker "won" and took over the industry.
    • deleted 2y ago
      [deleted]
      • deleted 2y ago
        [deleted]
    • kkfx 2y ago
      A sane DevOps workflow is with declarative systems like NixOS or Guix System, definitively not on a VM infra in practice regularly not up to date, full of useless deps, on a host definitively not up to date, with the entire infra typically not much managed nor manageable and with an immense attack surface... VMs are useful for those who live on the shoulder of someone else (i.e. *aaS) witch is ALL but insecure.
      • secondcoming 2y ago
        I'm not sure what you're referring to here? Our cloud machines are largely VMs. Deployments mean building a new image and telling GCP to deploy that as machines come and go due to scaling. The software is up to date, dependencies are managed via ansible. Maybe you think VMs means monoliths? That doesn't have to be the case.
        • kkfx 2y ago
          That's precisely the case: instead of owning hw, witch per-machine it's a kind-of monolith (even counting blades and other modular solution), you deploy a full OS or half-full to run just a single service, on top of another "OS". Of course yes, this is the cloud model, and is also the ancient and deprecated mainframe model, with much more added complexity and no unique ownership with an enormously big attack surface. Various return of experience prove that cloud model is not cheap nor reliable than owning iron, it's just fast since you live on the shoulders of someone else. A speed you will pay at an unknown point in time when something happen and you have zero control other that. DevOps meaning the Devs taking over the Ops without having the needed competences, it's a modern recipe to a failing digital ecosystems and we witnessed that more and more with various "biblical outages" from "Roomba devices briked due to an AWS mishap, cars of a certain vendor with a slice or RCEs, payment systems outages, ... a resilient infra it's not a centrally managed decentralized infra, it's a vast and diverse ecosystem interoperating with open and standard tools and protocols. Classic mail or Usenet infra are resilient, GMail backed by Alphabet infra is not. What if Azure tomorrow collapse? What's the impact? What's the attack surface of living on the shoulder of someone else, typically much bigger than you and often in other countries where getting even legal protections is costly and complex? Declarative systems on iron means you can replicate your infra ALONE on the iron, VMs meaning you need much more resources and you do not even know the entire stack of your infra, you can't essentially replicate nothing. VMs/images are still made the classical '80s style semi-manual way with some automation written by a dev knowing just how to manage his/her own desktop a bit and others will use it careless "it's easy to destroy and re-start", as a result we have seen in production images with someone unknown SSH authorized keys because to be quick someone pick the first ready made image from Google Search and add just few things, we are near the level of crap of the dot-com bubble, with MUCH more complexity and weight.
  • analognoise 2y ago
    What do you think of Nix/NixOS?
    • egberts1 2y ago
      Nix is trying to be like macOS's DMG but its image file is bit more parse-able.
    • reddit_clone 2y ago
      But that comes _after_ you have chosen VMs over Containers yes? If you are using VMs, I think NixOs/Guix is a good choice. Reproducible builds, Immutable OS, Immutable binaries and Dead easy rollback. It still looks somewhat futuristic. Hopefully gets traction.
      • solarpunk 2y ago
        if you're using nixos, just to do provisioning, I would argue OStree is a better fit.
      • bspammer 2y ago
        Nix is actually a really nice tool for building docker images: https://xeiaso.net/talks/2024/nix-docker-build/ https://xeiaso.net/talks/2024/nix-docker-build/
  • vundercind 2y ago
    Docker’s the best cross-distro rolling-release package manager and init system for services—staying strictly out of managing the base system, which is great—that I know of. I don’t know of anything that’s even close, really. All the other stuff about it is way less important to me than that part.
    • pxc 2y ago
      This is wrong in pretty much every way I can imagine. Docker's not a package manager. It doesn't know what packages are, which is part of why the chunks that make up Docker containers (image layers) are so coarse. This is also part of why many Docker images are so huge: you don't know exactly the packages you need, strictly speaking, so you start from a whole OS. This is also why your Dockerfiles all invoke real package managers— Docker can't know how to install packages if it doesn't know what they are! It's also not cross-platform, or at least 99.999% of images you might care about aren't— they're Linux-only. It's also not a service manager, unless you mean docker-compose (which is not as good as systemd or any number of other process supervisors) or Docker Swarm (which has lost out to Kubernetes). (I'm not sure what you even mean by 'init system for containers' since most containers don't include an init system.) There actually are cross-platform package managers out there, too. Nix, Pkgsrc, Homebrew, etc. All of those I mentioned and more have rolling release repositories as well. ('Rolling release' is not a feature of package managers; there is no such thing as a 'rolling release package manager'.)
      • vundercind 2y ago
        > This is wrong in pretty much every way I can imagine. Nope! It’s not wrong in any way at all! You’re thinking of how it’s built. I’m thinking of what it does (for me). I tell it a package (image) to fetch, optionally at a version. It has a very large set of well maintained up-to-date packages (images). It’s built-in, I don’t even have to configure that part, though I can have it use other sources for packages if I want to. It fetches the package. If I want it to update it, I can have it do that too. Or uninstall it. Or roll back the version. I am 100% for-sure using it as a package manager, and it does that job well. Then I run a service with a simple shell script (actually, I combine the fetching and running, but I’m highlighting the two separate roles it performs for me). It takes care of managing the process (image, which is really just a very-fat process for these purposes). It restarts it if it crashes, if I like. It auto-starts it when the machine reboots—all my services come back up on boot, and I’ve never touched systemd (which my Debian uses), Docker is my interface to that and I didn’t even have to configure it to do that part. I’m sure it’s doing systemd stuff under the hood, at least to bring the docker daemon up, but I’ve never touched that and it’s not my interface to managing my services. The docker command is. Do I see what’s running with systemd or ps? No, with docker. Start, restart, or stop a service? Docker. I’ve been running hobbyist servers at home (and setting up and administrating “real” ones for work) since 2000 or so and this is the smoothest way to do it that I’ve seen, at least for the hobbyist side. Very nearly the only roles I’m using Docker to fill, in this scenario, are package manager and service manager. I don’t care how it works—I know how, but the details don’t matter for my use case, just the outcomes. The outcome is that I have excellent, updated, official packages for way more services than are in the Debian repos, that leave my base system entirely alone and don’t meaningfully interact with it, with config that’s highly portable to any other distro, all managed with a common interface that would also be the same on any other distro. I don’t have to give any shits about my distro, no “oh if I want to run this I have to update the whole damn distro to a new major version or else manually install some newer libraries and hope that doesn’t break anything”, I just run packages (images) from Docker, update them with Docker, and run them with Docker. Docker is my UI for everything that matters except ZFS pool management. > It's also not cross-platform, or at least 99.999% of images you might care about aren't— they're Linux-only. I specifically wrote cross-distro for this reason. > There actually are cross-platform package managers out there, too. Nix, Pkgsrc, Homebrew, etc. Docker “packages” have a broader selection and better support than any of those, as far as services/daemons go; it’s guaranteed to keep everything away from the base system and tidy for better stability; and it provides a common interface for configuring where to put files & config for easier and more-confident backup. I definitely use it mainly as a package manager and service manager, and find it better than any alternative for that role.
  • topspin 2y ago
    Isn't this discussion based on a false dichotomy? I, too, use VMs to isolate customers, and I use containers within those VMs, either with or without k8s. These tools solve different problems. Containers solve software management, whereas VMs provide a high degree of isolation. Container orchestration is where I see the great mistake in all of this. I consider everything running in a k8s cluster to be one "blast domain." Containers can be escaped. Faulty containers impact everyone relying on a cluster. Container orchestration is the thing I believe is "overused." It was designed to solve "hyper" scale problems, and it's being misused in far more modest use cases where VMs should prevail. I believe the existence of container orchestration and its misapplication has retarded the development of good VM tools: I dream of tools that create, deploy and manage entire VMs with the same ease as Docker, and that these tools have not matured and gained popularity because container orchestration is so easily misapplied. Strongly disagree about containers and dev/deployment ("NOT check"). I can no longer imagine development without containers: it would be intolerable. Container repos are a godsend for deployment.
    • rodgerd 2y ago
      > Container orchestration is the thing I believe is "overused." It was designed to solve "hyper" scale problems, and it's being misused in far more modest use cases where VMs should prevail. As a relatively early corporate adopter of k8s, this is absolutely correct. There are problems where k8s is actually easier than building the equivalent capability elsewhere, but a lot of uses it's put to seem to be driven more by a desire to have kubernetes on one's resume.
      • topspin 2y ago
        For what k8s was designed to do -- herding vast quantities of ephemeral compute resources across a global network -- it's great. That's not my problem with it. My problem is that by being widely misapplied it has stunted the development of good solutions to everything else. K8s users spend their efforts trying to coax k8s to do things it was never intended to do, and so the k8s "ecosystem" has spiraled into this duplicative, esoteric, fragile, and costly bazar of complexity and overengineering.
  • tptacek 2y ago
    Jails/Zones are not pretty much as secure as a VM. They're materially less secure: they leave cotenant workloads sharing a single kernel (not just the tiny slice of the kernel KVM manages). Most kernel LPEs are probably "Jail" escapes, and it's not feasible to filter them out with system call sandboxing, because LPEs occur in innocuous system calls, too.
  • tomjen3 2y ago
    If anything Docker is underused. You should have a very good reason to make a deploy that is not Docker, or (if you really need the extra security) a VM that runs one thing only (and so is essentially a more resource requiring Docker). If you don’t, then it becomes much harder to answer the question of what exactly is deployed on a given server and what it takes to bring it up again if it goes down hard. If you but everything in Docker files, then the answer is whatever is set in the latest docker-compose file.
  • m463 2y ago
    I've always hated the docker model of the image namespace. It's like those cloud-based routers you can buy. Docker actively prevents you from having a private repo. They don't want you to point away from their cloud. Redhat understood this and podman allows you to have a private docker infrastructure, disconnected from docker hub. For my personal stuff, I would like to use "FROM scratch" and build my personal containers in my own ecosystem.
    • Carrok 2y ago
      > Docker actively prevents you from having a private repo. In what ways? I use private repos daily with no issues.
      • m463 2y ago
        If you reference a container without a domain, you pull from docker.io With podman, you can control this with $HOME/.config/containers/registries.conf or /etc/containers/registries.conf with docker, not possible (though you can hack mirrors) https://stackoverflow.com/questions/33054369/how-to-change-the-default-docker-registry-from-docker-io-to-my-private-registry https://stackoverflow.com/questions/33054369/how-to-change-t...
        • Carrok 2y ago
          So.. just use a domain. This seems like a nothing burger.
          • m463 2y ago
            Not all dockerfiles (especially multi-stage builds) are easily sanitized for this. think FROM python:latest or FROM ubuntu:20.04 AS build They've put deliberate barriers in the way of using docker commands without accessing their cloud.
        • Cyph0n 2y ago
          Or even easier: just fully qualify all images. With Podman: nginx => docker.io/library/nginx linuxserver/plex => docker.io/linuxserver/plex
  • ranger207 2y ago
    Docker's good at packaging, and Kubernetes is good at providing a single API to do all the infra stuff like scheduling, storage, and networking. I think that if someone sat down and tried to create a idealized VM management solution that covered everything between "dev pushes changes" to "user requests website" then it'd probably have a single image for each VM to run (like Docker has a single image for each container to run) then management of VM hosts, storage, networking, and scheduling VMs to run on which host would wind up looking a lot like k8s. You could certainly do that with VMs but for various path dependency reasons people do that with containers instead and nobody's got a well adopted system for doing the same with VMs
    • crabbone 2y ago
      I'm sorry, but: * Docker isn't good at packaging. When people talk about packaging, they usually understand it to include dependency management. For Docker to be good at packaging it should be able to create dependency graphs and allow users to re-create those graphs on their systems. Docker has no way of doing anything close to that. Aside from that, Docker suffers from the lack of reproducible builds, lack of upgrade protocols... It's not good at packaging... maybe it's better than something else, but there's a lot of room for improvement. * Kubernetes doesn't provide a single API to do all the infra stuff. In fact, it provides so little, it's a mystery why anyone would think that. All those stuff like "storage", "scheduling", "networking" that you mentioned comes as add-ons (eg. CSI, CNI) which aren't developed by Kubernetes, aren't following any particular rules, have their own interfaces... Not only that, Kubernetes' integration with CSI / CNI is very lacking. For example, there's no protocol for upgrading these add-ons when upgrading Kubernetes. There's no generic interface that these add-ons have to expose to the user in order to implement common things. It's really anarchy what's going on there... There are lots of existing VM management solutions, eg. OpenStack, VSphere -- you don't need to imagine them, they exist. They differ from Kubernetes in many ways. Very superficially, yet importantly, they don't have an easy way to automate them. For very simple tasks Kubernetes offers a very simple solution for automation. I.e. write some short YAML file. Automating eg. ESX comes down to using a library like govmomi (or something that wraps it, like Terraform). But, in the mentioned case, Terraform only managed deployment, and doesn't take care of the post-deployment maintenance... and so on. However, the more you deal with the infra, the more you realize that the initial effort is an insignificant fraction of the overall complexity of the task you need to deal with. And that's where the management advantages of Kubernetes start to seem less appealing. I.e. you realize that you will have to write code to manage your solution, and there will be a lot of it... and a bunch of YAML files won't cut it.
  • mountainriver 2y ago
    Docker is fantastic and VMs are fantastic. I honestly can’t imagine running all the services we have without containers. It would be wildly less efficient and harder to develop on. VMs are wonderful when you need the security
  • diego_sandoval 2y ago
    > and Docker industrial complex that developed around solving problems created by themselves or solved decades ago. From my perspective, it's the complete opposite: Docker is a workaround for problems created decades ago (e.g. dynamic linking), that could have been solved in a better manner, but were not.
    • ktosobcy 2y ago
      there are flatpacks/appimage/whatever but they are linux-only (mostly) and still lack something akin of docker-compose...
  • tiffanyh 2y ago
    Are Jails/Zones/Docker even security solutions? I always used them as process isolation & dependency bundling.
  • icelancer 2y ago
    Same. We're still managing ESXi here at my company. Docker/K8s/etc are nowhere close to prod and probably never will be. Been very pleased with that decision. I will say that Docker images get one HUGE use case at our company - CUDA images with consistent environments. CUDA/pytorch/tensorflow hell is something I couldn't imagine dealing with when I was in college studying CS a few decades ago.
  • lkrubner 2y ago
    I agree. VMs rely on old technologies, and are reliable in that way. By contrast, the move to Docker then necessitated additional technologies, such as Kubernetes, and Kubernetes brought an avalanche of new technologies to help manage Docker/Kubernetes. I am wary of any technology that in theory should make things simpler but in fact draws you down a path that requires you to learn a dozen new technologies. The Docker/Kubernetes path also drove up costs, especially the cost associated with the time needed to set up the devops correctly. Anything that takes time costs money. When I was at Averon the CEO insisted on absolutely perfect reliability and therefore flawless devops, so we hired a great devops guy to help us get setup, but he needed several weeks to set everything up, and his hourly rate was expensive. We could have just "push some code to a server" and we would have saved $40,000. When I consult with early stage startups, and they worry about the cost of devops, I point out that we can start simply, by pushing some code to a server, as if this was still 2001, and we can proceed slowly and incrementally from there. While Docker/Kubernetes offers infinite scalability, I warn entrepreneurs that their first concern should be keeping things simple and therefore low cost. And then the next step is to introduce VMs, and then use something like Packer to enable the VMs to be uses as AMIs and so allow the devops to develop to the point of using Terraform -- but all of that can wait till the product actually gains some traction.
  • benreesman 2y ago
    Namespaces and cgroups and LXC and the whole alphabet soup, the “Docker Industrial Complex” to borrow your inspired term, this stuff can make sense if you rack your own gear: you want one level of indirection. As I’ve said many times, putting a container on a serverless on a Xen hypervisor so you can virtualize while you virtualize? I get why The Cloud wants this, but I haven’t the foggiest idea why people sit still for it. As a public service announcement? If you’re paying three levels of markup to have three levels of virtual machine? You’ve been had.
    • Spivak 2y ago
      You're only virtualizing once. Serverless/FaaS is just a way to run a container, and a container is just a Linux process with some knobs to let different software coexist more easily. You're still just running VMs, same as you always were, but just have a new way of putting the software you want to run on them.
  • khana 2y ago
    [dead]
  • dhx 2y ago
    The article doesn't read to me to be an argument about whether sharing a kernel is better or worse (multiple virtual machines each with their own kernel versus multiple containers isolated by a single kernel). The article instead reads to me as an argument for isolating customers to their own customer-specific systems so there is no web server daemon, database server, file system path or other shared system used by multiple customers. As an aside to the article, two virtual machines each with their own kernel are generally forced to communicate with each in more complex ways through network protocols which add more complexity and increase risk of implementation flaws and vulnerabilities existing. Two processes in different cgroups with a common kernel have other simpler communication options available such as being able to read the same file directly, UNIX domain sockets, named pipes, etc.
    • didntcheck 2y ago
      Yep, the article just seems to be talking about single tenancy vs multi tenancy. The VMs vs containers thing seems mostly orthogonal
  • ktosobcy 2y ago
    > Modern "containers" were invented to make things more reproducible ( check ) and simplify dev and deployments ( NOT check ). Why? I have my RPi4 and absolutely love docker(-compose) - deploying stuff/services on in it just a breeze compared to previous clusterf*k of relying on system repository for apps (or if something doesnt work)... with docker compose I have nicely separated services with dedicated databases in required version (yes, I ran into an issue that one service required newer and another older version of the database, meh) As for development - I do development natively but again - docker makes it easier to test various scenarios...
    • skydhash 2y ago
      I’ve been using LXC/Incus as lightweight VMs (my home server is an old mac mini) and I think too many software is over reliant on Docker. It’s very much “ship your computer” with a bunch of odd scripts in addition to the Dockerfiles.
      • ktosobcy 2y ago
        Could you elaborate the "ship your computer"? Majority of the images are base OS (which is as lean as possible) and then just the app... To that end, full blown VM seems even more "ship your computer" thing? Btw. isn't LXC base for the Docker as well? It looks somewhat similar to docker and podman?
  • markandrewj 2y ago
    I wish people would stop going on about BSD jails as if they are the same. I would recommend at least using jails first. Most people using container technologies are well versed in BSD jails, as well as other technologies such as LXD, CRI-O, Micro VM's, and traditional virtualization technologies (KVM). You will encounter rough edges with any technology if you use it long enough. Container technologies require learning new skills, and this is where I personally see people often get frustrated. There is also the lean left mentality of container environments, where you are expected to be responsible for your environment, which is difficult for some. I.E. users become responsible for more then in a traditional virtualizated environment. People didn't stop using VM's, they just started using containers as well. What you should use is dependent on the workload. When you have to manage more then a single VM, and work on a larger team, the value of containers becomes more apparent. Not to mention the need to rapidly patch and update in today's environment. Often VM's don't get patched because applications aren't architected in a way to allow for updates without downtime, although it is possible. There is a mentality of 'if it's not broke, don't fix it'. There is some truth that virtualized hardware can provide bounds of seperation as well, but other things like selinux also enforce these boundaries. Not to mention containers are often running inside VM's as well. Using ephemeral VM's is not a new concept. The idea of 'cattle vs pets', and cloud, was built on KVM (OpenStack/AWS).
  • cryptonector 2y ago
    I mean, yeah, but things like rowhammer and Spectre/Meltdown, and many other side-channels are a big deal. VMs are not really enough to prevent abuse of the full panoply of side-channels known and unknown.
  • cryptonector 2y ago
    Jails/Zones are just heavy-duty containers. They're still not VMs. Not that VMs are enough either, given all the side-channels that abound.
  • TheNewsIsHere 2y ago
    I feel the exact same way. There are so many use cases that get shoved into the latest, shiniest box just because it’s new and shiny. A colleague of mine once suggested running a CMS we manage for customers on a serverless stack because “it would be so cheap”. When you face unexpected traffic bursts or a DDoS, it becomes very expensive, very fast. Customers don’t really want to be billed per execution during a traffic storm. It would also have been far outside the normal environment that CMS expects, and wouldn’t have been supported by any of our commercial, vendored dependencies. Our stack is so much less complicated without running everything in Docker, and perhaps ironically, about half of our stack runs in Kubernetes. The other half is “just software on VMs” we manage through typical tools like SSH and Ansible.
• osigurdson 2y ago
  When thinking about multi-tenancy, remember that your bank doesn't have a special VM or container, just for you.
  • 01HNNWZ0MV43FF 2y ago
    My bank doesn't even have 2FA
    • jmnicolas 2y ago
      Mine neither and they use a 6 numbers pincode! This is ridiculous, in comparison my home wifi password is 60+ random chars long.
      • leononame 2y ago
        But they do ask you only two digits of the pin on each try and they probably will lock your account after three incorrect attempts. Not saying 6 digits is secure, but it's better than everyone using "password" if they have a string policy on incorrect attempts. And don't hm they have 2FA for executing transactions? I'm pretty sure banks are some of the most targeted IT systems. I don't trust them blindly, but when it comes to online security, I trust that they built a system that's reasonably well secured and other cases, I'd get my money back, similar to credit cards.
      • dspillett 2y ago
        Mine, FirstDirect in the UK, recently dropped the password from “between 5 and 9 case-sensitive alphanumeric characters” to “exactly six digits” and claimed that this was just as secure as before…¹² My guess is that either they were cutting support costs and wanted to reduce the number of calls from people who forgot their more complicated password!. Either that or they are trying to integrate a legacy system, don't have the resources/access to improve that, so reduced everything else down to its level. When raised one on of their public facing online presences someone pointed out that it is no less than other online banks do, but if they are happy being just as good but no better than other banks there is nothing for me to be loyal to should another bank come up with a juicy looking offer. ---- [1] because of course 13,759,005,982,823,100 possible combinations is no better than exactly 1,000,000 where you know most people are going to use some variant of a date of birth/marriage and makes shoulder-surfing attacks no more difficult </snark> [2] The only way it is really just as secure as before is if there is a significant hole elsewhere so it doesn't matter what options are available there. Going from zero security to zero security is just as secure as before, no lie!
• jonathanlydall 2y ago
  Sure, it’s an option which eliminates the possibility of certain types of errors, but it’s costing you the ability to pool computing resources as efficiently as you could have with a multi-tenant approach. The author did acknowledge it’s a trade off, but the economics of this trade off may or may not make sense depending on how much you need to charge your customers to remain competitive with competing offerings.
  • bbqbbqbbq 2y ago
    [dead]
• bobbob1921 2y ago
  My big struggle with docker/containers vs VMs is the storage layer (on containers). I’m sure it’s mostly lack of experience / knowledge on my end, but I never have a doubt or concern that my storage is persistent and clearly defined when using a VM based workload. I cannot say the same for my docker/container based workloads, I’m always a tad concerned about the persistence of storage, (or the resource management in regards to storage). This becomes even more true as you deal with networked storage on both platforms
  • imp0cat 2y ago
    Mount those paths that you care about to local filesystem. Otherwise, you're always one `docker system prune -a -f --volumes` from a disaster.
  • amluto 2y ago
    It absolutely boggles my mind that read-only mode is not the default in Docker. By default, every container has an extra, unnamed, writable volume: its own root. Typo in your volume mount? You’re writing to root, and you will lose data. Of course, once this is fixed and you start using read-only containers, one wonders why “container” exists as a persistent, named concept.
    • danhor 2y ago
      Because unless you resort to stuff like in-ram overlayfs which will also result in data loss, a lot of system software assumes it can write anywhere and will bitterly complain if not, even if it's not "real" data, and can be very annoying to fix. That's fina for carefully engineered containers, but the usual thrown together stuff docker started with gets a lot more annoying.
• fsckboy 2y ago
  just as a meta idea, i'm mystified that systems folks find it impossible to create protected mode operating systems that are protected, and then we all engage in wasteful kluges like VMs. i'm not anti-VM, they're great technology, i just don't think it should be the only way to get protection. VMs are incredibly inefficient... what's that you say, they're not? ok, then why aren't they integrated into protected mode OSes so that they will actually be protected?
  • bigbones 2y ago
    Because it would defeat the purpose. Turns out we don't trust the systems folks all that much
  • Veserv 2y ago
    [flagged]
    • elric 2y ago
      Hah, I was going to post the same quote when I read the parent comment. Glad to see I'm not the only grump who remembers TDR quotes. But he's right. And with the endless stream of leaky CPUs and memory (spectre, rowhammer, etc) he's even more right now than he was 17 years ago. There are all kinds of things being done to mitigate multi-tenant security risks in the Confidential Computing space (with Trusted Execution Environments, Homomorphic Encryption, or even Secure Multiparty Computation), but these are all incredibly complex and largely bolted on to an insecure base. It's just really, *really*, hard to make something non-trivial fully secure. "It depends on your threat model" used to be a valid statement, but with everyone running all of their code on top of basically 3 platforms owned by megacorps, I'm not sure even that is true anymore.
      • tptacek 2y ago
        Microarchitectural attacks are an even bigger problem for shared-kernel multitenant systems!
    • tptacek 2y ago
• deleted 2y ago
  [deleted]
• stacktrust 2y ago
  A modern virtualization architecture can be found in the OSS pKVM L0 nested hypervisor for Android Virtualization Framework, which has some architectural overlap with HP/Bromium AX L0 + [Hyper-V | KVM | Xen] L1 + uXen L2 micro-VMs with copy-on-write memory. A Bromium demo circa 2014 was a web browser where every tab was an isolated VM, and every HTTP request was an isolated VM. Hundreds of VMs could be launched in a couple of hundred milliseconds. Firecracker has some overlap. > Lastly, this approach is almost certainly more expensive. Our instances sit idle for the most part and we pay EC2 a pretty penny for the privilege. With many near-idle server VMs running identical code for each customer, there may be an opportunity to use copy-on-memory-write VMs with fast restore of unique memory state, using the techniques employed in live migration. Xen/uXen/AX: https://www.platformsecuritysummit.com/2018/speaker/pratt/ https://www.platformsecuritysummit.com/2018/speaker/pratt/ pKVM: https://www.youtube.com/watch?v=9npebeVFbFw https://www.youtube.com/watch?v=9npebeVFbFw
• jefurii 2y ago
  Using VMs as the unit allows them to move to another provider if they need to. They could even move to something like an on-prem Oxide rack if they wanted. [Yes I know, TFA lists this as a "false benefit" i.e. something they think doesn't benefit them.]
• mikewarot 2y ago
  It's nice to see the Principle Of Least Access (POLA) in practical use. Some day, we'll have operating systems that respect it as well. As more people wake up to the realization that we shouldn't trust code, I expect that the number of civilization wide outages will decrease. Working in the cloud, they're not going to be able to use my other favorite security tool, the data diode. Which can positively guarantee ingress of control, while still allowing egress of reporting data.
  • fsflover 2y ago
    > Some day, we'll have operating systems that respect it as well. Qubes OS has been relying on it for many years. My daily driver, can't recommend it enough.
  • nrr 2y ago
    If you're coming by after the fact and scratching your head at what a data diode is, Wikipedia's page on the subject is a decent crib document. <https://en.wikipedia.org/wiki/Unidirectional_network https://en.wikipedia.org/wiki/Unidirectional_network>
• SunlitCat 2y ago
  VMs are awesome for what they can offer. Docker (and the like) are kinda a lean VM for a specific tool scenario. What I would like to see, would be more App virtualization software which isolates the app from the underlying OS enough to provide an safe enough cage for the app. I know there are some commercial offerings out there (and a free one), but maybe someone can chime in has some opinions about them or know some additional ones?
  • stacktrust 2y ago
    HP business PCs ship with SureClick based on OSS uXen, https://news.ycombinator.com/item?id=41071884 https://news.ycombinator.com/item?id=41071884
    • SunlitCat 2y ago
      Thank you for sharing, didn't know that one!
      • stacktrust 2y ago
        It's from the original Xen team. Subsequently cloned by MS as MDAG (Defender Application Guard).
        • SunlitCat 2y ago
          Cool! I know MDAG and actually it's a pretty neat concept, kinda.
  • peddling-brink 2y ago
    That’s what containers attempt to do. But it’s not perfect. Adding a layer like gvisor helps, but again the app is still interacting with the host kernel so kernel exploits are still possible. What additional sandboxing are you thinking of?
    • SunlitCat 2y ago
• smitty1e 2y ago
  > Switching to another provider would be non-trivial, and I don’t see the VM as a real benefit in this regard. The barrier to switching is still incredibly high. This point is made in the context of VM bits, but that switching cost could (in theory, haven't done it myself) be mitigated using, e.g. Terraform. The brace-for-shock barrier at the enterprise level is going to be exfiltrating all of that valuable data. Bezos is running a Hotel California for that data: "You can checkout any time you like, but you can never leave" (easily).
  • tetha 2y ago
    Heh. We're in the process of moving a service for a few of our larger customers over due to some variety of emergencies, let's keep it at that. It took us 2-3 days of hustling to get the stuff running and production ready and providing the right answers. This is the "Terraform and Ansible-Stuff" stage of a real failover. In a full infrastructure failover, I'd expect it to take us 1-2 very long days to get 80% running and then up to a week to be fully back on track and another week of shaking out strange issues. And then a week or two of low-availability from the ops-team. However, for 3 large customers using that product, cybersecurity and compliance said no. They said no about 5-6 weeks ago and project to have an answer somewhere within the next 1-2 months. Until then, the amount of workarounds and frustration growing around it is rather scary. I hope I can contain it to some places in which there is no permanent damage for the infrastructure. Tech isn't necessarily the hardest thing in some spaces.
• kkfx 2y ago
  As much stuff you add as much attack surface you have. Virtualized infra are a commercial need, an IT and Operation OBSCENITY definitively never safe in practice.
• tptacek 2y ago
  The cool kids have been combining containers and hardware virtualization for something like 10 years now (back to QEMU-Lite and kvmtool). Don't use containers if the abstraction gets in your way, of course, but if they work for you --- as a mechanism for packaging and shipping software and coordinating deployments --- there's no reason you need to roll all the way back to individually managed EC2 instances. A short survey on this stuff: https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/
  • mwcampbell 2y ago
    Since you're here, I was just thinking about how feasible it would be to run a microVM-per-tenant setup like this on Fly. I guess it would require some automation to create a Fly app for each customer. Is this something you all have thought about?
    • tptacek 2y ago
      Extraordinarily easy. It's a design goal of the system. I don't want to crud up the thread; this whole "container vs. VM vs. dedicated hardware" debate is dear to my heart. But feel free to drop me a line if you're interested in our take on it.
      • heeton 2y ago
        I’m also interested in your take on it, if you wanted to publish a response publicly. Would love something like this for enterprise SaaS clients.
    • DAlperin 2y ago
      Also to add, we already have lots of customers who use this model.
• vin10 2y ago
  > If you wouldn't trust running it on your host, you probably shouldn't run it in a container as well. - From a Docker/Moby Maintainer
• ploxiln 2y ago
  > we operate in networks where outbound MQTT and HTTPS is simply not allowed (which is why we rely on encrypted DNS traffic for device-to-Console communication) HTTPS is not allowed (locked down for security!), so communication is smuggled over DNS? uhh ... I suspect that a lot of what the customer "security" departments do, doesn't really make sense ...
  • jmprspret 2y ago
    DNS tunneling, or smuggling through DNS requests, is like a known malware C2 method. Seems really weird to (ab)use it for ""security""
    • thinkst 2y ago
      Product builders can learn loads from malware in terms of deployment and operational ease. Malware needs to operate without any assistance in unknown environments. Nobody is allowing outbound comms deliberately for malware, so tunnel methods were developed. Networks have these capabilities, inherently they're part of the specs. But only malware seems to realise that and use it. We love reusing offensive techniques for defence (see our Canarytokens stuff), and DNS comms fits that perfectly. Our customers get an actual 2-minute install, not a 2-minute-and-then-wait-a-week-for-the-firewall-rules install.
      • jmprspret 2y ago
        I didn't mean to imply that so for security was a bad thing. Now I read back my comment I see that is exactly how it sounds. I agree with you
        • thinkst 2y ago
          We've got several product features that have been driven by our offensive security background... thanks for the prod, we'll blog it.
      • marcosdumay 2y ago
        The problem is that when you apply the malware lessons to your software, every anti-virus starts to work against you.
• er4hn 2y ago
  One thing I wasn't able to grok from the article is orchestration of VMs. Are they using AWS to manage the VM lifecycles, restart them, etc? Last time I looked into this for on-prem the solutions seemed very enterprise, pay the big bux, focused. Not a lot in the OSS space. What do people use for on-prem VM orchestration that is OSS?
  • jinzo 2y ago
    Depends what is your scale, but I used oVirt and Proxmox in the past, and it was (especially oVirt) very enterprisey but OSS.
• deleted 2y ago
  [deleted]
• JohnCClarke 2y ago
  Question: Could you get the customer isolation by running all console access through customer specific lambdas which simply add a unique (and secret) header to all requests. Then you can run a single database with sets of tables keyed by that secret header value. Would give you very nearly as good isolation for much lower cost.
• coppsilgold 2y ago
  If you think about it virtualization is just a narrowing of the application-kernel interface. In a standard setting the application has a wide kernel interface available to it with dozens (ex. seccomp) to 100's of syscalls. A vulnerablility in any one of which could result in full system compromise. With virtualization the attack surface is narrowed to pretty much just the virtualization interface. The problem with current virtualization (or more specifically, the VMM's) is that it can be cumbersome, for example memory management is a serious annoyance. The kernel is built to hog memory for cache and etc. but you don't want the guest to be doing that - since you want to overcommit memory as guests will rarely use 100% of what is given to them (especially when the guest is just a jailed singular application), workarounds such as free page reporting and drop_caches hacks exist. I would expect eventually to see high performance custom kernels for a application jails - for example: gVisor[1] acts as a syscall interceptor (and can use KVM too!) and a custom kernel. Or a modified linux kernel with patched pain points for the guest. In effect what virtualization achieves is the ability to rollback much of the advantage of having an operating system in the first place in exchange for securely isolating the workload. But because the workload expects an underlying operating system to serve it, one has to be provided to it. So now you have a host operating system and a guest operating system and some narrow interface between the two to not be a complete clown show. As you grow the interface to properly slave the guest to the host to reduce resource consumption and gain more control you will eventually end up reimagining the operating system perhaps? Or come full circle to the BSD jail idea - imagine the host kernel having hooks into every guest kernel syscall, is this not a BSD jail with extra steps? [1] <https://gvisor.dev/ https://gvisor.dev/>
• Melatonic 2y ago
  Eventually we'll get a great system managing some form of micro VM that lots of people use and we have years of documentation and troubleshooting on Until then the debate between VM and Containerisation will continue
• Havoc 2y ago
  So you end up with thousands of near idle AWS instances? There has got to be a better middle ground. Like mult tenant but strong splits ( each customer on db etc )
• deleted 2y ago
  [deleted]
• ianpurton 2y ago
  I've solved the same problem but used Kubernetes namespaces instead. Each customer gets their own namespace and a namespace is locked down in terms of networking and I deploy Postgres in each namespace using the Postgres operator. I've built an operator for my app, so deploying the app into a namespace is as simple as deploying the manifest.
• deleted 2y ago
  [deleted]
• sim7c00 2y ago
  i wish nanoVMs were better. its a cool concept leveraging the actual VM extensions for security. but all the ones i've seen hardly get into user-mode, dont have stack protectors or other trivial security features enabled etc. (smap/smep) making it super insecure anyway. maybe someday that market will boom a bit more, so we can run hypervisors with vms in there that host single application kind of things. like a BSD kernel that runs postgres as its init process or something. (i know thats oversimplified probarbly ::P). there's a lot of room in the VM space for improvement ,but pretty much all of it is impossible if you need to load an entire OS multi-purpose-multi-user into the vm.....
• udev4096 2y ago
  > Nothing here will earn us a speaking invite to CNCF events This made me laugh for some reason
• solatic 2y ago
  There's nothing in Kubernetes and containers that prevents you from running single-tenant architectures (one tenant per namespace), or from colocating all single-tenant services on the same VM, and preventing multiple customers from sharing the same VM (pod affinity and anti-affinity). I'm not sure why the author doesn't understand that he could have his cake and eat it too.
• javier_e06 2y ago
  Months ago I went to the movie theater. Why a $20.00 USD bill in my hand I asked the young one (yes I am that old) for a medium pop corn. "Credit Card" only. He warned me. "You have to take cash" I reminded me. He directed me to the box office where I had to purchase a $20 USD gift card which I then used to purchase the pop-corn. I never used the remaining balance. Management does not trust the crew of low wage minions, with cash, who would? I had my popcorn right? What is the complain here? I network comes done, stores will have no choice but to hand the food for free. I am currently not trouble shooting my solutions. I am trouble shooting the VM.
• Thaxll 2y ago
  You could use different nodepool per customers using the same k8s control plane.
• JackSlateur 2y ago
  Boarf This can be boiled down to "we use AWS' built-in security, not our own". Using EC2 instances is then nothing but a choice. You could do the exact same thing with containers (with fargate, perhaps ?) : one container per tenant, no relations between containers => same things (but cheaper).