5 ms·

Virtlet: run VMs as Kubernetes pods

179 points by ivan4th 8y ago

rwmj 8y ago
Interesting to see how this really compares to KubeVirt, which seems to be doing the same thing. I don't think KubeVirt is "just" pets as far as I understand. [disclaimer: I've been very peripherally involved with KubeVirt because they asked me about integrating virt-v2v support].
- ivan4th 8y ago
  You can't have a StatefulSet of VMs on KubeVirt, although you may want one if you're using unikernels or want to run a nested Kubernetes cluster for testing, like here: https://github.com/Mirantis/virtlet/blob/master/examples/k8s.yaml https://github.com/Mirantis/virtlet/blob/master/examples/k8s... On the other hand, you can't have VM migrations with Virtlet.
  - rwmj 8y ago
    Be nice if there was just one "virt for Kube" technology. I guess virt-v2v will end up having to support both.
    - wmf 8y ago
      Given that the container ecosystem is a huge land rush with many obviously missing pieces, there are going to be multiple projects for every piece.
  - rmohr 8y ago
    Yes, right now we were more focusing on the 1:1 case. Our OfflineVirtualMachine is basically a StatefulSet with replica 1. For stateless scaling we have a VirtualMachineReplicaSet. A StatefulVirtualMachineSet will probably be added soon to KubeVirt.
- sofaofthedamned 8y ago
  As an aside - I saw you a few years ago in Gloucester UK and you taught me loads about virtualisation. Thank you Richard.
- outworlder 8y ago
  It seems that KubeVirt is implemented as an operator and registers new resource types, which is an approach I like. I'd have for someone to do a kubectl delete pod --all thinking they are only deleting stateless apps and getting rid of VMs too.
agumonkey 8y ago
reminds me of vmlets term from the virtual virtual machines https://pages.lip6.fr/vvm/publications/0008SBAC.pdf https://pages.lip6.fr/vvm/publications/0008SBAC.pdf
darren0 8y ago
The real reason for these projects (Virlet, KubeVirt, RancherVM) is that OpenStack is too damn hard. Even with all this focus on containers these days people still need an onprem VM solution. But the requirements are really pretty simple for most use cases and the cost and complexity of OpenStack is not justified. Edit: disclaimer: my company does RancherVM.
- jell 8y ago
  The reason for this project was that people want to migrate to containers as industry is pushing them forward to it, but - there will always be cases in which "normal" containers will be out of scope - when you will be dependent on piece which is: a) based on other than linux based operating system b) you need a specific linux module which for some reason should not be loaded on host kernel c) you need a hardware separation using virtualization because of security reasons All that can be achieved using "somewhere around" openstack, but what about integration/common interface with other parts of stack? That's the reason why we did Virtlet - to have the same interface for "normal" (docker image based) pods and VMs (hard disk image based), with the same kubectl command, with the same api (so it can be used for deployments, daemonsets, statefulsets and so on), with the same thing configuring networking in whole cluster (CNI, hopefully any of your choice - if it's not working, please fill the bug on GH).
  - darren0 8y ago
    I don't fully buy that explanation. If OpenStack was better they would have already had success on that platform and then would be asking for a bridge between k8s and OpenStack (openstack could easily be a virtual kubelet). Instead people are looking to abandon OpenStack in favor of something hopefully better.
    - jell 8y ago
      I'm not saying that OpenStack is better/worse - it was only out of our goals. For sure OpenStack has better tenancy support, much more mature approach to networking or storage (in containers world we are still working on https://github.com/container-storage-interface/spec https://github.com/container-storage-interface/spec ) but in the same time - we only needed to have a qcow2 based vm running between already deployed pods, with the same user interface. That's all, without other story around that.
s0l1dsnak3123 8y ago
I'm using kubevirt for this. My usecase is to allow preconfigured windows VMs to boot up inside of kubernetes - so that both Windows VMs and Linux containers are manipulated using the same API. It works very well!
- ivan4th 8y ago
  Yes, you can use KubeVirt for Windows. In some cases, it may be even better than Virtlet for that purpose. But if you want a real Kubernetes Deployment / StatefulSet / DaemonSet / etc. built out of VMs (examples being unikernels, nested Kubernetes cluster for testing etc.) Virtlet may be a better choice.
  - dummdida 8y ago
    True - Today it does not directly fit into existing workload controllers (Deployment, ReplicaSet etc) but SIG Api Machinery actually saw that others would also like to see CRD support in templates of controllers. If this lands, then KubeVirt would get that support "for free" (tm)
  - rmohr 8y ago
    Regarding the nested cluster: One of the main reasons for having the VirtualMachineReplicaSet in KubeVirt is to allow that use-case. They work pretty nice in combination with CloudInit and e.g. kubeadm to dynamically scale the cluster.
- jell 8y ago
  the key is that with Virtlet you can use you windows VM as other pods in replicasets, daemonsets, statefulsets, what can not be done using CRD offered by kubevirt. In the same time, assuming that your windows image supports cloud init (e.g. using http://cloudbase-init.readthedocs.io http://cloudbase-init.readthedocs.io ) - you can setup it during the boot, configuring users, passwords, running arbitrary scripts - you have full control about what will be done during cloud init phase using only pod annotations (you don't need prepared specific image with predefined content as in KubeVirt case). In case of Virtlet you have a possibility to merge data from different sources (config maps, secrets, annotations) and use the result in output cloud init image. At the end - KubeVirt provides you a way to run a VM as a custom resource in an app on k8s (virt-launcher), while Virtlet provides a way to run VM as first class cluster citizen, as a pod, with the same interfaces as for other pods (so you can use kubectl logs, kubectl attach -it, as with other "normal" (docker image based) pods). ok, with windows vm you will see in kubectl logs only cloud init logs as thats the only part which is using serial console to output something :P with other types of vm images (based on linux, bsd or unikernels) you would probably appreciate more possibility to use kubectl attach -it, or kubectl logs ;)
deleted 8y ago
[deleted]
edaemon 8y ago
What's the key difference between Virtlet/KubeVirt and just running VMs alongside your Kubernetes pods? Is the main goal to centralize management? That seems reasonable, I'm just wondering. Also, how does a Kubernetes-managed VM compare to a plain VM from, say, AWS EC2? I imagine it's a little less efficient, since the situation I'm imagining involves various VMs running a Kubernetes pod running the VM, but I may have this all wrong.
- ivan4th 8y ago
  Besides using kubectl to manage VMs, you also get VMs that join the cluster network as first-class citizens. In this blog post you can see a k8s service pointing to a VM and accessing k8s services from within the VMs. Also, compared to EC2 / GCE / etc., you can use Virtlet for bare metal on-prem clusters, too.
  - edaemon 8y ago
    That makes sense, thanks.

179 points by ivan4th 8y ago

rwmj 8y ago

Interesting to see how this really compares to KubeVirt, which seems to be doing the same thing. I don't think KubeVirt is "just" pets as far as I understand. [disclaimer: I've been very peripherally involved with KubeVirt because they asked me about integrating virt-v2v support].

ivan4th 8y ago
You can't have a StatefulSet of VMs on KubeVirt, although you may want one if you're using unikernels or want to run a nested Kubernetes cluster for testing, like here: https://github.com/Mirantis/virtlet/blob/master/examples/k8s.yaml https://github.com/Mirantis/virtlet/blob/master/examples/k8s... On the other hand, you can't have VM migrations with Virtlet.
- rwmj 8y ago
  Be nice if there was just one "virt for Kube" technology. I guess virt-v2v will end up having to support both.
  - wmf 8y ago
    Given that the container ecosystem is a huge land rush with many obviously missing pieces, there are going to be multiple projects for every piece.
- rmohr 8y ago
  Yes, right now we were more focusing on the 1:1 case. Our OfflineVirtualMachine is basically a StatefulSet with replica 1. For stateless scaling we have a VirtualMachineReplicaSet. A StatefulVirtualMachineSet will probably be added soon to KubeVirt.
sofaofthedamned 8y ago
As an aside - I saw you a few years ago in Gloucester UK and you taught me loads about virtualisation. Thank you Richard.
outworlder 8y ago
It seems that KubeVirt is implemented as an operator and registers new resource types, which is an approach I like. I'd have for someone to do a kubectl delete pod --all thinking they are only deleting stateless apps and getting rid of VMs too.

agumonkey 8y ago

reminds me of vmlets term from the virtual virtual machines https://pages.lip6.fr/vvm/publications/0008SBAC.pdf https://pages.lip6.fr/vvm/publications/0008SBAC.pdf

darren0 8y ago

The real reason for these projects (Virlet, KubeVirt, RancherVM) is that OpenStack is too damn hard. Even with all this focus on containers these days people still need an onprem VM solution. But the requirements are really pretty simple for most use cases and the cost and complexity of OpenStack is not justified. Edit: disclaimer: my company does RancherVM.

jell 8y ago
The reason for this project was that people want to migrate to containers as industry is pushing them forward to it, but - there will always be cases in which "normal" containers will be out of scope - when you will be dependent on piece which is: a) based on other than linux based operating system b) you need a specific linux module which for some reason should not be loaded on host kernel c) you need a hardware separation using virtualization because of security reasons All that can be achieved using "somewhere around" openstack, but what about integration/common interface with other parts of stack? That's the reason why we did Virtlet - to have the same interface for "normal" (docker image based) pods and VMs (hard disk image based), with the same kubectl command, with the same api (so it can be used for deployments, daemonsets, statefulsets and so on), with the same thing configuring networking in whole cluster (CNI, hopefully any of your choice - if it's not working, please fill the bug on GH).
- darren0 8y ago
  I don't fully buy that explanation. If OpenStack was better they would have already had success on that platform and then would be asking for a bridge between k8s and OpenStack (openstack could easily be a virtual kubelet). Instead people are looking to abandon OpenStack in favor of something hopefully better.
  - jell 8y ago
    I'm not saying that OpenStack is better/worse - it was only out of our goals. For sure OpenStack has better tenancy support, much more mature approach to networking or storage (in containers world we are still working on https://github.com/container-storage-interface/spec https://github.com/container-storage-interface/spec ) but in the same time - we only needed to have a qcow2 based vm running between already deployed pods, with the same user interface. That's all, without other story around that.

s0l1dsnak3123 8y ago

I'm using kubevirt for this. My usecase is to allow preconfigured windows VMs to boot up inside of kubernetes - so that both Windows VMs and Linux containers are manipulated using the same API. It works very well!

ivan4th 8y ago
Yes, you can use KubeVirt for Windows. In some cases, it may be even better than Virtlet for that purpose. But if you want a real Kubernetes Deployment / StatefulSet / DaemonSet / etc. built out of VMs (examples being unikernels, nested Kubernetes cluster for testing etc.) Virtlet may be a better choice.
- dummdida 8y ago
  True - Today it does not directly fit into existing workload controllers (Deployment, ReplicaSet etc) but SIG Api Machinery actually saw that others would also like to see CRD support in templates of controllers. If this lands, then KubeVirt would get that support "for free" (tm)
- rmohr 8y ago
  Regarding the nested cluster: One of the main reasons for having the VirtualMachineReplicaSet in KubeVirt is to allow that use-case. They work pretty nice in combination with CloudInit and e.g. kubeadm to dynamically scale the cluster.
jell 8y ago
the key is that with Virtlet you can use you windows VM as other pods in replicasets, daemonsets, statefulsets, what can not be done using CRD offered by kubevirt. In the same time, assuming that your windows image supports cloud init (e.g. using http://cloudbase-init.readthedocs.io http://cloudbase-init.readthedocs.io ) - you can setup it during the boot, configuring users, passwords, running arbitrary scripts - you have full control about what will be done during cloud init phase using only pod annotations (you don't need prepared specific image with predefined content as in KubeVirt case). In case of Virtlet you have a possibility to merge data from different sources (config maps, secrets, annotations) and use the result in output cloud init image. At the end - KubeVirt provides you a way to run a VM as a custom resource in an app on k8s (virt-launcher), while Virtlet provides a way to run VM as first class cluster citizen, as a pod, with the same interfaces as for other pods (so you can use kubectl logs, kubectl attach -it, as with other "normal" (docker image based) pods). ok, with windows vm you will see in kubectl logs only cloud init logs as thats the only part which is using serial console to output something :P with other types of vm images (based on linux, bsd or unikernels) you would probably appreciate more possibility to use kubectl attach -it, or kubectl logs ;)

deleted 8y ago

[deleted]

edaemon 8y ago

What's the key difference between Virtlet/KubeVirt and just running VMs alongside your Kubernetes pods? Is the main goal to centralize management? That seems reasonable, I'm just wondering. Also, how does a Kubernetes-managed VM compare to a plain VM from, say, AWS EC2? I imagine it's a little less efficient, since the situation I'm imagining involves various VMs running a Kubernetes pod running the VM, but I may have this all wrong.

ivan4th 8y ago
Besides using kubectl to manage VMs, you also get VMs that join the cluster network as first-class citizens. In this blog post you can see a k8s service pointing to a VM and accessing k8s services from within the VMs. Also, compared to EC2 / GCE / etc., you can use Virtlet for bare metal on-prem clusters, too.
- edaemon 8y ago
  That makes sense, thanks.