4 ms·

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff lik

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry if I'm misunderstanding, but the hypervisor in the cloud (i.e. the VM running under your "instance") is to protect the cloud provider from you. The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads. It is totally within your power to choose to run 1 cloud "instance" per workload so you don't have to worry about cross-workload contamination, but if you want to effectively multiplex your workload on constantly available machines, you're going to have to mix your workloads, and that almost certainly requires protection of some kind. In an ideal world we might all be getting VMs that were unikernels that are a bit safer to use but in general it feels like you need to isolate these workloads somehow if you're going to run them on the same "instance". What does your perfect/preferred world look like?

paulfurtado 7y ago
Kata works pretty well. You set it as a runtime for docker and it gives a fantastic illusion that the container isn't different from any other docker container but uses a KVM VM with minimal hardware to run the container. It starts in under 200ms (and faster if you make it use firecracker), etc. The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers. I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-binary-translation https://blogs.oracle.com/ravello/nested-virtualization-with-...
- dragonsh 7y ago
  We tried kata the user experience is not as good as lxd. Moreover with lxd just use the same deployment platform, scripts as used for bare-metal and VM. No need to fiddle with mix of shell scripts and DSL for container orchestration and how to integrate with your own code. No need for zombie process like in docker.
  - paulfurtado 7y ago
    Kata is great if you're already using docker, it also is increased isolation versus lxd. If you don't already have a use case for docker, kubernetes, the image repository features, etc then of course kata wouldn't be useful to you over lxc. Also, in case you're dealing with zombie processes in docker again and can't ditch docker, running a container with `docker run --init` gets you a minimal init process that reaps the zombies. Adding that flag to dockerd instead does this for all containers by default. It's insane that this isn't turned on by default since it's such a common problem. Additionally, if you want to run systemd inside the container similar to lxc, you can install the systemd-oci-hook and it will do the necessary setup for systemd to be happy.
    - dragonsh 7y ago
      Thanks for the information and indeed I looked at Kata again. But just it didn't click like the way LXD did. I found with LXD the image repository are so easy, deployment my team can re-use whatever they learned for bare-metal and virtual machines and last but not the least its performant with nice tooling. Initially when I started using had few networking issues due to different support in GCP, AWS and Azure. But now my team is proficient enough to use it even on them. Also the constant updates Stephen and his team did is fantastic. If you have time please look into weekly updates and play with LXD. It works great for reasonable sized container cluster which covers most of the startups. They won't be good for big google size of kind yet without more tooling like Kubernetes.
- hardwaresofton 7y ago
  As a person who runs on (and really likes) Hetzner... I totally misunderstood, thanks for clearing it up.
nailer 7y ago
> The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads. I think it's more likely that the hypervisor someone iss running on top of their cloud provider's hypervisor exists because many people like building their own cloud providers and like to add layers of unnecessary complexity (like a second layer of hypervisors) at their employer's or investor's expense. If someone doesn't trust their own workloads, it is vastly simpler and less expensive to run them on separate cloud instance.
- sansnomme 7y ago
  Spawn time is fairly significant for your typical VMs. An entire class of applications (such as serverless) cannot be used as your usual traditional VM creation takes several minutes while the hybrid container-VM types do it in milliseconds, allowing for near real-time applications.
  - nailer 7y ago
    It's not clear how what you're saying relates to nested hypervisors, but I don't think anyone does serverless on nested hypervisors. AWS Lambda uses MicroVMs which to my knowledge are not nested.
    - sansnomme 7y ago
      Your typical open source "serverless" systems that run on Kubernetes has a security story that's "trusted code only". If your application involves routinely running hostile code e.g. something like Cloudflare Workers. Your usual put-it-in-a-container story just won't cut it. You need on-demand, rapidly deployed containers, that is capable of providing VM level of security and isolation with the spawn time and resource consumption of a container. This isn't new technology, AWS, Google etc. have had some variant of it from day 1. Heroku and a lot of the secondary PaaS all run on the big players' underneath, they don't do it themselves. It is only recently that the technologies involved were open sourced and standardized.
eyberg 7y ago
In the public cloud (aws/gce) we provision unikernels as vms themselves - that is there is no underlying linux instance. It's just the unikernel riding on top of the hypervisor (in the case of google that's kvm, for the t2s on aws it's xen but their c5s are kvm). For small services you use t2-micros (aws) and f1-micros (gce). Both will give you a thread. If you need more you scale more. This works great if you have a handful of services - if you have hundreds or thousands of services yes you might want to look at running on your own hardware as the public cloud could get expensive at that point or conversely you can re-evaluate if you need hundreds of services and maybe just opt for larger services that use heftier amounts of threads/memory (eg: larger instance sizes). As for running your own hardware we have crappy (eg: $800) servers in our office that have 32 threads and we haven't really stressed it at all but we can easily run thousands of unikernels on those boxes. Also, in the bay area you can grab a cabinet from Hurricane Electric for $400/month. Imagine what you could do with real hardware.
- eximius 7y ago
  What domain is the problem you're solving in that you can effectively run unikernels? i.e., you don't need something that ultimately doesn't run well in unikernels. For example, our application is a web app - that by itself can run reasonably well in a unikernel (not with how we've written it, but it can be done). But we call out to imagemagick for some processing, which would pretty much preclude a unikernel approach (barring breaking it out into its own service, which is totally doable - its hard to provide a perfect example). Also, what are you using to create the unikernels?
sansnomme 7y ago
We don't need unikernels. The technology for running containers securely has existed for years. The only problem is that few mainstream/major cloud providers support them, it requires nested virtualization among other things. And your traditional companies like AWS and GCE make you pay out of the nose for it because if it becomes widely available, then the whole "Serverless", elastic container service etc. industry may be eroded. Right now you have two choices: run your own co-location bare metal server with all the complexity that comes with it, or pay for a VPS with nested virtualization support that is a magnitude more expensive.

paulfurtado 7y ago

Kata works pretty well. You set it as a runtime for docker and it gives a fantastic illusion that the container isn't different from any other docker container but uses a KVM VM with minimal hardware to run the container. It starts in under 200ms (and faster if you make it use firecracker), etc. The problem your parent is pointing out is that AWS instances don't support nested virtualization (and it sounds like neither does Digital Ocean) so on both of these cloud providers you can't use something like kata containers or any nested VMs. GCP does support nested virtualization in any VM and aws `.metal` instance types do as well, but are rather expensive. It's really a shame that even with the new EC2 kvm-based hypervisor they still didn't enable nested virtualization for most instance types, otherwise we'd definitely be making heavy use of kata containers. I wish Oracle hadn't bought Ravello Systems and that Ravello open sourced their binary translation stuff that made nested VMs possible in EC2 without the full overhead of software virtualization. Unfortunately, there are no open source implementations of similar software that I know of. Their blog is now hosted on oracle's site: https://blogs.oracle.com/ravello/nested-virtualization-with-binary-translation https://blogs.oracle.com/ravello/nested-virtualization-with-...

dragonsh 7y ago
We tried kata the user experience is not as good as lxd. Moreover with lxd just use the same deployment platform, scripts as used for bare-metal and VM. No need to fiddle with mix of shell scripts and DSL for container orchestration and how to integrate with your own code. No need for zombie process like in docker.
- paulfurtado 7y ago
  Kata is great if you're already using docker, it also is increased isolation versus lxd. If you don't already have a use case for docker, kubernetes, the image repository features, etc then of course kata wouldn't be useful to you over lxc. Also, in case you're dealing with zombie processes in docker again and can't ditch docker, running a container with `docker run --init` gets you a minimal init process that reaps the zombies. Adding that flag to dockerd instead does this for all containers by default. It's insane that this isn't turned on by default since it's such a common problem. Additionally, if you want to run systemd inside the container similar to lxc, you can install the systemd-oci-hook and it will do the necessary setup for systemd to be happy.
  - dragonsh 7y ago
    Thanks for the information and indeed I looked at Kata again. But just it didn't click like the way LXD did. I found with LXD the image repository are so easy, deployment my team can re-use whatever they learned for bare-metal and virtual machines and last but not the least its performant with nice tooling. Initially when I started using had few networking issues due to different support in GCP, AWS and Azure. But now my team is proficient enough to use it even on them. Also the constant updates Stephen and his team did is fantastic. If you have time please look into weekly updates and play with LXD. It works great for reasonable sized container cluster which covers most of the startups. They won't be good for big google size of kind yet without more tooling like Kubernetes.
hardwaresofton 7y ago
As a person who runs on (and really likes) Hetzner... I totally misunderstood, thanks for clearing it up.

nailer 7y ago

> The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads. I think it's more likely that the hypervisor someone iss running on top of their cloud provider's hypervisor exists because many people like building their own cloud providers and like to add layers of unnecessary complexity (like a second layer of hypervisors) at their employer's or investor's expense. If someone doesn't trust their own workloads, it is vastly simpler and less expensive to run them on separate cloud instance.

sansnomme 7y ago
Spawn time is fairly significant for your typical VMs. An entire class of applications (such as serverless) cannot be used as your usual traditional VM creation takes several minutes while the hybrid container-VM types do it in milliseconds, allowing for near real-time applications.
- nailer 7y ago
  It's not clear how what you're saying relates to nested hypervisors, but I don't think anyone does serverless on nested hypervisors. AWS Lambda uses MicroVMs which to my knowledge are not nested.
  - sansnomme 7y ago
    Your typical open source "serverless" systems that run on Kubernetes has a security story that's "trusted code only". If your application involves routinely running hostile code e.g. something like Cloudflare Workers. Your usual put-it-in-a-container story just won't cut it. You need on-demand, rapidly deployed containers, that is capable of providing VM level of security and isolation with the spawn time and resource consumption of a container. This isn't new technology, AWS, Google etc. have had some variant of it from day 1. Heroku and a lot of the secondary PaaS all run on the big players' underneath, they don't do it themselves. It is only recently that the technologies involved were open sourced and standardized.

eyberg 7y ago

In the public cloud (aws/gce) we provision unikernels as vms themselves - that is there is no underlying linux instance. It's just the unikernel riding on top of the hypervisor (in the case of google that's kvm, for the t2s on aws it's xen but their c5s are kvm). For small services you use t2-micros (aws) and f1-micros (gce). Both will give you a thread. If you need more you scale more. This works great if you have a handful of services - if you have hundreds or thousands of services yes you might want to look at running on your own hardware as the public cloud could get expensive at that point or conversely you can re-evaluate if you need hundreds of services and maybe just opt for larger services that use heftier amounts of threads/memory (eg: larger instance sizes). As for running your own hardware we have crappy (eg: $800) servers in our office that have 32 threads and we haven't really stressed it at all but we can easily run thousands of unikernels on those boxes. Also, in the bay area you can grab a cabinet from Hurricane Electric for $400/month. Imagine what you could do with real hardware.

eximius 7y ago
What domain is the problem you're solving in that you can effectively run unikernels? i.e., you don't need something that ultimately doesn't run well in unikernels. For example, our application is a web app - that by itself can run reasonably well in a unikernel (not with how we've written it, but it can be done). But we call out to imagemagick for some processing, which would pretty much preclude a unikernel approach (barring breaking it out into its own service, which is totally doable - its hard to provide a perfect example). Also, what are you using to create the unikernels?

sansnomme 7y ago

We don't need unikernels. The technology for running containers securely has existed for years. The only problem is that few mainstream/major cloud providers support them, it requires nested virtualization among other things. And your traditional companies like AWS and GCE make you pay out of the nose for it because if it becomes widely available, then the whole "Serverless", elastic container service etc. industry may be eroded. Right now you have two choices: run your own co-location bare metal server with all the complexity that comes with it, or pay for a VPS with nested virtualization support that is a magnitude more expensive.