10 ms·

Libvirt – The Unsung Hero of Cloud Computing (2013)

222 points by vikrantrathore 6y ago

catern 6y ago
I really doubt that AWS is using libvirt. They almost certainly have their own abstraction. But still, I agree that libvirt is great; I wrote about it here: http://catern.com/posts/libvirt.html http://catern.com/posts/libvirt.html
- vikrantrathore 6y ago
  Might be true, but don’t see any open source work from Amazon in public domain which shows they built their own libraries from scratch to manage Xen and cloud management in early years from 2008. Indeed it’s 2020 and yet to see any major open source work from Amazon (which has benefited a lot from open source itself using Perl, CPAN, C, Java, Linux etc.). In this respect IBM, google, Microsoft, Facebook and Apple are far better. Here even Oracle fare better due to acuisition of MySQL and sun microsystems. I believe the major contribution from amazon might be hiring some of the open source developers to build proprietary systems. Those developers in spare time or weekends continue their open source project, but I do not have any study or articles on it. Based on my information in 2013, amazon built their cloud using Xen hypervisor and related tools and libraries. Libvirt is one of the key libraries providing beautiful abstractions and language bindings to manage xen on Linux node at that time. It will be nice if you can point to code from Amazon on low level library like Libvirt for cloud computing.
  - lmz 6y ago
    What makes you think it was libvirt instead of the Xen native xm / xl tools?
  - easton 6y ago
    Amazon has been open-sourcing a lot lately. Firecracker (microVM service), Bottlerocket (Linux designed for hosting containers), and they've been distributing OpenJDK builds. Perhaps not as much as Microsoft (these days), but they are certainly giving back.
- exikyut 6y ago
  Or QEMU, which libvirt uses. They're using KVM directly, possibly with Firecracker.
amscanne 6y ago
While I respect the job that libvirt does (it works — high praise for software), it’s unfortunate that it is also the answer to the question “how can I represent all these virtualized things using XML?”, which was in fashion when libvirt was created. It’s also a bit misleading to characterize cloud providers as building on libvirt. Libvirt is useful as an mostly hypervisor-agnostic wrapper, which is super useful for enterprise on-prem software, but kinda of the opposite of what big providers need and build for themselves. I wonder what we will look back on as the XML of today. Everything is schemaless JSON and YAML; surely we’ll look back and wonder WTF everyone was thinking? But alas, it’s probably not a data format at all. Only time will tell.
- catmanjan 6y ago
  Yep, if people think XML has aged poorly, wait until these schema-less formats age...
  - stochastastic 6y ago
    Ha! ... that’s a laugh of bitter jealousy. I’ve dealt with XML, but I’ve never with XML that came with a schema, or which would have reliably followed one. Not saying schema-less formats are great, but at least I can eyeball them to see what is going on.
    - shmerl 6y ago
      Schemas are good. Even JSON now has one.
    - hinkley 6y ago
      XML turned everyone into a language designer during an era where we already knew that language design was a rare skill. If I saw a schema, which I often didn’t, it usually didn’t say what the author thought it said. To a first order approximation, all the good ones I saw came from one tool (XMLSpy possibly?) Namespaces ended up in a sort of uncanny valley that I can’t quite do justice to.
willscott 6y ago
for the last month (v6.6.0 and v6.7.0) libvirt has been released with a new key 453B65310595562855471199CA68BE8010084C9C (first seen:2020-07-20). It hasn't been signed or verified by any other source in libvirt / redhat yet. https://www.redhat.com/archives/libvirt-announce/2020-July/msg00006.html https://www.redhat.com/archives/libvirt-announce/2020-July/m... That is preventing downstream distros, like arch, from admitting these new releases into their package repos. Critical infrastructure, indeed.
- silly-silly 6y ago
  Key exists on pgp.ocf.berkeley.edu.
CSDude 6y ago
Libvirt is really nice. For linux, I also recommend reading the man page of Qemu directly to learn more about internals, it helped me understand a lot. Of course you have to take care networking, disk management on your own but it can be really simple. https://linux.die.net/man/1/qemu-kvm https://linux.die.net/man/1/qemu-kvm
- guerby 6y ago
  There's a nice tool to get qemu command lines from libvirt xml: virsh domxml-to-native qemu-argv myvm.xml
  - tarruda 6y ago
    Nice, thanks for sharing!
ohazi 6y ago
If you haven't tried it, the "Virtual Machine Manager" GUI [1] is also surprisingly usable on Desktop Linux. I use it with both Windows and Linux images, and getting it to work with the usual guest extension niceties is straightforward. From what I remember, getting VirtualBox or VMware to launch and run properly after a few months of automatic upgrades and not launching your images for a while was always kind of a gamble. With libvirt, everything just seems to work, and your images are just ready to go when you need them. [1] https://virt-manager.org/ https://virt-manager.org/
- throwaway8941 6y ago
  >With libvirt, everything just seems to work Strictly speaking, you have qemu to thank for that. libvirt is just a frontend for several hypervisors, including qemu (which is what typically used).
  - bonzini 6y ago
    Libvirt is also responsible for keeping the guest exactly the same after upgrades; a basic QEMU command line does not guarantee that the guest hardware remains the same when you upgrade to a newer version, while Libvirt uses the more complicated and less human-friendly options to ensure that. Libvirt does a lot more for QEMU than for other hypervisors, so much that libvirtd's initial name was qemud.
- lights0123 6y ago
  For simple usage, GNOME Boxes is really nice. It'll automatically download the ISOs of basically any Linux distro when you click on their logo, automatically go through the installer, and install copy/paste, dynamic resizing of the VM screen when you resize the window (I don't believe virt-manager supports that), and drag/drop support. It uses libvirt behind the scenes, so you can use virt-manager with VMs created by Boxes and vice versa.
  - ohazi 6y ago
    I don't think I ever tried drag and drop, but I've gotten copy/paste and window->display resizing to work in virt-manager after manually setting up the kvm/qemu guest tools.
guerby 6y ago
We're currently migrating from VMware to libvirt, we discovered the cockpit project and cockpit-machines to manage VMs: https://cockpit-project.org/ https://cockpit-project.org/ cockpit-machines is available in a recent version in debian backports, installing it is trivial, no configuration, https://hostname:9090/ https://hostname:9090/ and just works. RedHat announced that cockpit will be the long term successor of virt-manager: https://www.redhat.com/en/blog/managing-virtual-machines-rhel-8-web-console https://www.redhat.com/en/blog/managing-virtual-machines-rhe... https://blog.wikichoon.com/2020/06/virt-manager-deprecated-in-rhel.html https://blog.wikichoon.com/2020/06/virt-manager-deprecated-i... cockpit has frequent releases, latest: https://cockpit-project.org/blog/cockpit-227.html https://cockpit-project.org/blog/cockpit-227.html It hasn't all the features of virt-manager, far from it, but looks promising.
- thejosh 6y ago
  That's awesome, thanks for the share. For my homelab I use proxmox on a couple of machines and it works great for managing containers (in terms of LXC containers that would be more of a traditional VM) and it works great. Most people/companies don't need the complexities that come with Kubernetes or other tools like that.
  - guerby 6y ago
    cockpit has many "applications" including cockpit-podman to manage containers : https://github.com/cockpit-project/cockpit-podman https://github.com/cockpit-project/cockpit-podman This is the Cockpit user interface for podman containers. It is being actively developed and has not yet reached feature parity with cockpit-docker. For now you can do basic image and container tasks for both system and user containers.
- sgt 6y ago
  I gave it a shot, and it was able to see my running libvirt VM's although it could not create one due to an obscure error of not supporting the "custom" CPU type. The running VM's are running in full virtualization so it is probably just a minor bug in Cockpit, seeing that it is after all just a libvirtd frontend in this regard. Aside from that, Cockpit looks fairly stable and it has potential, but keep in mind it is very simple.
xfennec 6y ago
We've build our own automated hosting infrastructure* a few years ago on top of libvirt. Using the libvirt API was a breeze and libvirt is rock solid since day one for us, I can only recommend this project. * Shameless plug: https://github.com/OnitiFR/mulch https://github.com/OnitiFR/mulch We're using libvirt-go binding (Daniel Berrangé and his team is doing a excellent job maintaining it!), and KVM/QEMU hypervisior. For a small team like us, it's incredibly valuable to have access to such powerful tools in a such easy way.
- appleflaxen 6y ago
  Mulch looks super cool; you have some great ideas!
shekharshan 6y ago
I couldn't understand this: "Domain is an instance of an operating system (or subsystem in case of container virtualization like OpenVZ and lxc) running on a virtualized machine provided by the hypervisor" So the domain itself is a virtual machine? What makes it different from other guest virtual machinse?
- jeroenhd 6y ago
  I've never used libvirt in combination with OpenVZ or LXC, but when I read that I interpret it as "a domain is a VM or a container" with some kind of VM running on the system to facilitate containers. The article is seven years old, so I can imagine that this was how the project once ran containers, but it doesn't anymore. Reading the documentation[1] I don't the VM is relevant anymore for containers, at least. The docs say libvirt manages LXC container directly through the kernel API, so there's no VM to speak of there. The OpenVZ docs[2] also mention that the containers run on the host, not in a VM. [1]: https://libvirt.org/drvlxc.html https://libvirt.org/drvlxc.html [2]: https://libvirt.org/drvopenvz.html https://libvirt.org/drvopenvz.html
- rwmj 6y ago
  It's unfortunate that when libvirt was originally started (2005) it was a wrapper around Xen, and in Xen a VM is called a domain.
dmacvicar 6y ago
If you want to use libvirt with terraform, this is the project I started to learn Go years ago: https://github.com/dmacvicar/terraform-provider-libvirt https://github.com/dmacvicar/terraform-provider-libvirt

I really doubt that AWS is using libvirt. They almost certainly have their own abstraction. But still, I agree that libvirt is great; I wrote about it here: http://catern.com/posts/libvirt.html http://catern.com/posts/libvirt.html

vikrantrathore 6y ago
Might be true, but don’t see any open source work from Amazon in public domain which shows they built their own libraries from scratch to manage Xen and cloud management in early years from 2008. Indeed it’s 2020 and yet to see any major open source work from Amazon (which has benefited a lot from open source itself using Perl, CPAN, C, Java, Linux etc.). In this respect IBM, google, Microsoft, Facebook and Apple are far better. Here even Oracle fare better due to acuisition of MySQL and sun microsystems. I believe the major contribution from amazon might be hiring some of the open source developers to build proprietary systems. Those developers in spare time or weekends continue their open source project, but I do not have any study or articles on it. Based on my information in 2013, amazon built their cloud using Xen hypervisor and related tools and libraries. Libvirt is one of the key libraries providing beautiful abstractions and language bindings to manage xen on Linux node at that time. It will be nice if you can point to code from Amazon on low level library like Libvirt for cloud computing.
- lmz 6y ago
  What makes you think it was libvirt instead of the Xen native xm / xl tools?
- easton 6y ago
  Amazon has been open-sourcing a lot lately. Firecracker (microVM service), Bottlerocket (Linux designed for hosting containers), and they've been distributing OpenJDK builds. Perhaps not as much as Microsoft (these days), but they are certainly giving back.
exikyut 6y ago
Or QEMU, which libvirt uses. They're using KVM directly, possibly with Firecracker.

amscanne 6y ago

While I respect the job that libvirt does (it works — high praise for software), it’s unfortunate that it is also the answer to the question “how can I represent all these virtualized things using XML?”, which was in fashion when libvirt was created. It’s also a bit misleading to characterize cloud providers as building on libvirt. Libvirt is useful as an mostly hypervisor-agnostic wrapper, which is super useful for enterprise on-prem software, but kinda of the opposite of what big providers need and build for themselves. I wonder what we will look back on as the XML of today. Everything is schemaless JSON and YAML; surely we’ll look back and wonder WTF everyone was thinking? But alas, it’s probably not a data format at all. Only time will tell.

catmanjan 6y ago
Yep, if people think XML has aged poorly, wait until these schema-less formats age...
- stochastastic 6y ago
  Ha! ... that’s a laugh of bitter jealousy. I’ve dealt with XML, but I’ve never with XML that came with a schema, or which would have reliably followed one. Not saying schema-less formats are great, but at least I can eyeball them to see what is going on.
  - shmerl 6y ago
    Schemas are good. Even JSON now has one.
  - hinkley 6y ago
    XML turned everyone into a language designer during an era where we already knew that language design was a rare skill. If I saw a schema, which I often didn’t, it usually didn’t say what the author thought it said. To a first order approximation, all the good ones I saw came from one tool (XMLSpy possibly?) Namespaces ended up in a sort of uncanny valley that I can’t quite do justice to.

willscott 6y ago

for the last month (v6.6.0 and v6.7.0) libvirt has been released with a new key 453B65310595562855471199CA68BE8010084C9C (first seen:2020-07-20). It hasn't been signed or verified by any other source in libvirt / redhat yet. https://www.redhat.com/archives/libvirt-announce/2020-July/msg00006.html https://www.redhat.com/archives/libvirt-announce/2020-July/m... That is preventing downstream distros, like arch, from admitting these new releases into their package repos. Critical infrastructure, indeed.

silly-silly 6y ago
Key exists on pgp.ocf.berkeley.edu.

CSDude 6y ago

Libvirt is really nice. For linux, I also recommend reading the man page of Qemu directly to learn more about internals, it helped me understand a lot. Of course you have to take care networking, disk management on your own but it can be really simple. https://linux.die.net/man/1/qemu-kvm https://linux.die.net/man/1/qemu-kvm

guerby 6y ago
There's a nice tool to get qemu command lines from libvirt xml: virsh domxml-to-native qemu-argv myvm.xml
- tarruda 6y ago
  Nice, thanks for sharing!

ohazi 6y ago

If you haven't tried it, the "Virtual Machine Manager" GUI [1] is also surprisingly usable on Desktop Linux. I use it with both Windows and Linux images, and getting it to work with the usual guest extension niceties is straightforward. From what I remember, getting VirtualBox or VMware to launch and run properly after a few months of automatic upgrades and not launching your images for a while was always kind of a gamble. With libvirt, everything just seems to work, and your images are just ready to go when you need them. [1] https://virt-manager.org/ https://virt-manager.org/

throwaway8941 6y ago
>With libvirt, everything just seems to work Strictly speaking, you have qemu to thank for that. libvirt is just a frontend for several hypervisors, including qemu (which is what typically used).
- bonzini 6y ago
  Libvirt is also responsible for keeping the guest exactly the same after upgrades; a basic QEMU command line does not guarantee that the guest hardware remains the same when you upgrade to a newer version, while Libvirt uses the more complicated and less human-friendly options to ensure that. Libvirt does a lot more for QEMU than for other hypervisors, so much that libvirtd's initial name was qemud.
lights0123 6y ago
For simple usage, GNOME Boxes is really nice. It'll automatically download the ISOs of basically any Linux distro when you click on their logo, automatically go through the installer, and install copy/paste, dynamic resizing of the VM screen when you resize the window (I don't believe virt-manager supports that), and drag/drop support. It uses libvirt behind the scenes, so you can use virt-manager with VMs created by Boxes and vice versa.
- ohazi 6y ago
  I don't think I ever tried drag and drop, but I've gotten copy/paste and window->display resizing to work in virt-manager after manually setting up the kvm/qemu guest tools.

guerby 6y ago

We're currently migrating from VMware to libvirt, we discovered the cockpit project and cockpit-machines to manage VMs: https://cockpit-project.org/ https://cockpit-project.org/ cockpit-machines is available in a recent version in debian backports, installing it is trivial, no configuration, https://hostname:9090/ https://hostname:9090/ and just works. RedHat announced that cockpit will be the long term successor of virt-manager: https://www.redhat.com/en/blog/managing-virtual-machines-rhel-8-web-console https://www.redhat.com/en/blog/managing-virtual-machines-rhe... https://blog.wikichoon.com/2020/06/virt-manager-deprecated-in-rhel.html https://blog.wikichoon.com/2020/06/virt-manager-deprecated-i... cockpit has frequent releases, latest: https://cockpit-project.org/blog/cockpit-227.html https://cockpit-project.org/blog/cockpit-227.html It hasn't all the features of virt-manager, far from it, but looks promising.

thejosh 6y ago
That's awesome, thanks for the share. For my homelab I use proxmox on a couple of machines and it works great for managing containers (in terms of LXC containers that would be more of a traditional VM) and it works great. Most people/companies don't need the complexities that come with Kubernetes or other tools like that.
- guerby 6y ago
  cockpit has many "applications" including cockpit-podman to manage containers : https://github.com/cockpit-project/cockpit-podman https://github.com/cockpit-project/cockpit-podman This is the Cockpit user interface for podman containers. It is being actively developed and has not yet reached feature parity with cockpit-docker. For now you can do basic image and container tasks for both system and user containers.
sgt 6y ago
I gave it a shot, and it was able to see my running libvirt VM's although it could not create one due to an obscure error of not supporting the "custom" CPU type. The running VM's are running in full virtualization so it is probably just a minor bug in Cockpit, seeing that it is after all just a libvirtd frontend in this regard. Aside from that, Cockpit looks fairly stable and it has potential, but keep in mind it is very simple.

xfennec 6y ago

We've build our own automated hosting infrastructure* a few years ago on top of libvirt. Using the libvirt API was a breeze and libvirt is rock solid since day one for us, I can only recommend this project. * Shameless plug: https://github.com/OnitiFR/mulch https://github.com/OnitiFR/mulch We're using libvirt-go binding (Daniel Berrangé and his team is doing a excellent job maintaining it!), and KVM/QEMU hypervisior. For a small team like us, it's incredibly valuable to have access to such powerful tools in a such easy way.

appleflaxen 6y ago
Mulch looks super cool; you have some great ideas!

shekharshan 6y ago

I couldn't understand this: "Domain is an instance of an operating system (or subsystem in case of container virtualization like OpenVZ and lxc) running on a virtualized machine provided by the hypervisor" So the domain itself is a virtual machine? What makes it different from other guest virtual machinse?

jeroenhd 6y ago
I've never used libvirt in combination with OpenVZ or LXC, but when I read that I interpret it as "a domain is a VM or a container" with some kind of VM running on the system to facilitate containers. The article is seven years old, so I can imagine that this was how the project once ran containers, but it doesn't anymore. Reading the documentation[1] I don't the VM is relevant anymore for containers, at least. The docs say libvirt manages LXC container directly through the kernel API, so there's no VM to speak of there. The OpenVZ docs[2] also mention that the containers run on the host, not in a VM. [1]: https://libvirt.org/drvlxc.html https://libvirt.org/drvlxc.html [2]: https://libvirt.org/drvopenvz.html https://libvirt.org/drvopenvz.html
rwmj 6y ago
It's unfortunate that when libvirt was originally started (2005) it was a wrapper around Xen, and in Xen a VM is called a domain.

dmacvicar 6y ago

If you want to use libvirt with terraform, this is the project I started to learn Go years ago: https://github.com/dmacvicar/terraform-provider-libvirt https://github.com/dmacvicar/terraform-provider-libvirt