9 ms·

Last week, I intended to replace VirtualBox with QEMU for my personal use. The spartiate documentation made it really hard. I lack the competence to seriously c

Last week, I intended to replace VirtualBox with QEMU for my personal use. The spartiate documentation made it really hard. I lack the competence to seriously contribute. The official documentation just explains the numerous parameters. It also points to tutorials (wikibook, etc), which unfortunately are very basic and out of sync. Even Arch Linux does not provide a good documentation (lack of clarity, some obsolete syntax, "Networking" section is a mess...). Like often, Debian's wiki is obsolete, and even their QEMU images are 5 years old. I had to combine the following 3 sources and a lot of try and guesses to build my VMs. QEMU Gentoo : https://wiki.gentoo.org/wiki/QEMU/Options https://wiki.gentoo.org/wiki/QEMU/Options QEMU official : https://qemu.weilnetz.de/doc/qemu-doc.html https://qemu.weilnetz.de/doc/qemu-doc.html KVM Performance : http://www.linux-kvm.org/page/Tuning_KVM http://www.linux-kvm.org/page/Tuning_KVM

tobbyb 8y ago
We need to spin up a lot of VMs of various Linux distributions for testing Flockport and we found the existing tooling too heavy and involved. In the end we rolled our own solution that is much more lightweight, uses QEMU directly and provides better visibility, ease of use and even some potentially useful networking features. Here are a few screencasts [1]. This is mainly designed as a module for Flockport and we will preview this soon. [1] https://www.youtube.com/watch?v=uEnk9m-egeA https://www.youtube.com/watch?v=uEnk9m-egeA
Double_a_92 8y ago
Welcome to the Open Source experience... sadly.
- darkwater 8y ago
  https://www.virtualbox.org/wiki/Licensing_FAQ https://www.virtualbox.org/wiki/Licensing_FAQ
- Retra 8y ago
  As opposed to what? Most proprietary tools don't have good documentation either. And then you may not even be able to look at the code to see what it does.
  - oblio 8y ago
    I don't get why he's downvoted. Virtualbox (which is not proprietary), VMWare, Hyper-V are way easier to use. That's a fact. And not many people need to look at the code in real life...
    - bonzini 8y ago
      You should compare VirtualBox to virt-manager or GNOME Boxes, not QEMU, just like you shouldn't compare Visual Studio to gcc.
      - oblio 8y ago
        Yeah, but VirtualBox & co. come as a package. virt-manager and Gnome Boxes, as you can see from the other comments, have been missed by various people trying to use these tools :)
      - nurettin 8y ago
        I am on my third confusing error using virt-manager to create an armv7 build environment. First one was some ca-cert error, which meant I forgot to type a slash somewhere in some URL, second was something about not being able to enable firewall, which required firewalld and dnsmasq to be installed, now I'm having an error adding a qcow2 image to my vm setup, it probably doesn't like my mounted storage. We can compare this experience to terrible vendor lockins.
deleted 8y ago
[deleted]
simula67 8y ago
I have not used it, but Drew Devault had a post explaining how to get started with Qemu : https://drewdevault.com/2018/09/10/Getting-started-with-qemu.html https://drewdevault.com/2018/09/10/Getting-started-with-qemu...
- aorth 8y ago
  After a decade of only using qemu via libvirt on the server, and lazily using VirtualBox on the desktop, this post convinced me to start using qemu locally. I run unprivileged guests for development and testing and it fits my workflow perfectly for about six months now.
- AceJohnny2 8y ago
  > There’s really no excuse to be using any other hypervisor [1]. They’re all dogshit compared to qemu. > [1] Especially VirtualBox. If you use VirtualBox after reading this article you make poor life choices and are an embarrassment to us all. Well I'm convinced! What a well-presented article! Appropriate Futurama quote: "Remember when I told you about always ending your stories a sentence earlier?". Drew was doing well until he ended with this unnecessary condescending and hostile conclusion (and if you think it's funny, note that "the failure mode of smartass is asshole"). It's a trend of his. He's got interesting and informed things to say, but it's hard to really take him seriously because of this.
emmelaich 8y ago
The Gnome Boxes app is excellent https://wiki.gnome.org/action/show/Apps/Boxes https://wiki.gnome.org/action/show/Apps/Boxes
pferde 8y ago
There are several higher level frameworks that wrap QEMU-KVM and make it easier to use. oVirt, libvirt and proxmox are among my favourites, with the latter being quite polished and with advanced features for HA use.
chousuke 8y ago
Using QEMU directly is a bit tricky and it's not really intended to be used in that fashion by most users, unlike VirtualBox. Both gnome-boxes and virt-manager use QEMU under the hood though, and they provide a point-and-click experience. I use virt-manager myself and I do not think it's any more difficult than VirtualBox is.
- vero2 8y ago
  We have several windows vms running in virtualbox. All have rdp/vnc for specialised programs. (These are offline). Everytime I tried to migrate to qemu or virt-manager or gnome-boxes it was a real nightmare especially installing SPICE+virtio drivers+ networking. I read in Phoronix the performance of virtualbox is subpar and all KVM/qemu is the way to go. Please, could the OSS community build a easy to use GUI that SPICE+virtio drivers+ networking is made simple. Any one from qemu, I really appreciate the work but still very challenging for sysadmins.
  - chousuke 8y ago
    I confess that I haven't run Windows with virt-manager on my own desktop in a long while, but it worked just fine after installing the required drivers at install time. IIRC Redhat distributes signed drivers for Windows for free. The problems I did have were with VMs I tried to convert from VMware to KVM, and those stemmed from Windows' inability (version 7) to properly handle changing hardware. Perhaps it's better at nowadays.
    - naasking 8y ago
      Did this recently. It runs ok, but input and responsiveness latency is very noticeable.
  - aasasd 8y ago
    Virt-manager seems to be written in Python. So if libvirt supports the bits you want then with some tinkering in GTK you might be able to bolt the controls onto Virt-manager, at least to get things started.
  - 8y ago
BossingAround 8y ago
Rather than using QEMU by itself, it might be advisable to use libvirt..? As somewhat of a newbie to virtualization, that's how I started - googling how to use libvirt and virsh, and I found a lot of resources that helped me to deploy what I needed... What are your painpoints in particular? I.e. which topics would you like to explore deeper and can't due to poor documentation/resources? We could work on improving the resources.
- babuskov 8y ago
  > What are your painpoints in particular? I believe his pain point is that with Virtual Box you get a very simple GUI where you can get VM up and running very quickly without having to use Google or read any guides at all.
  - BossingAround 8y ago
    For that, I use 'Virtual Machine Manager' [1], which works for me extremely well, and is available for major distros. [1] https://virt-manager.org/ https://virt-manager.org/
  - philpem 8y ago
    Well, qemu + libvirt + virt-manager works pretty well. Virt-manager adds a very nice VirtualBox-style GUI front-end to libvirt.
  - carlavilla 8y ago
    You can use QtEmu (https://gitlab.com/qtemu https://gitlab.com/qtemu)
- qalmakka 8y ago
  I personally dislike libvirt a lot, and virsh in particular. I think that it's pointlessly complicated and burdensome to use, and that in general the hypervisor-agnosticabstractions provided by libvirt are nullified by the fact that no backend except Qemu really works well, this making them a pointless exercise in futility.
pm215 8y ago
My recommendation for that sort of use would be to look at using a 'management layer' app that sits above QEMU and handles all the fiddly command line details for you -- https://virt-manager.org/ https://virt-manager.org/ (which uses/is part of libvirt) is a popular one, I think. The QEMU project has kind of settled on the philosophy of being the low-level tool which provides you "full manual control" and interfaces for a higher-level UI to make use of. User-friendliness is not really our forte and so we let other projects provide that part.
- idoubtit 8y ago
  This is the context I was missing when I switched to QEMU. I had not heard about libvirt. A simple "apt search qemu" had shown a few unsatisfying GUIs, but nothing about other layers above qemu. I suggest QEMU's documentation could mention libvirt and its user-friendly interfaces.
  - kashyapc 8y ago
    FWIW, QEMU's upstream README does mention libvirt and higher-level managment tools: "QEMU aims to fit into a variety of use cases. It can be invoked directly by users wishing to have full control over its behaviour and settings. It also aims to facilitate integration into higher level management layers, by providing a stable command line interface and monitor API. It is commonly invoked indirectly via the libvirt library when using open source applications such as oVirt, OpenStack and virt-manager." But I get it, not everyone goes in search of a README buried in a Git repository. I myself am not sure if this REAMDE is rendered in an HTML format somewhere. [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=README https://git.qemu.org/?p=qemu.git;a=blob_plain;f=README
  - carlavilla 8y ago
    Modern GUI for QEMU: https://gitlab.com/qtemu https://gitlab.com/qtemu
  - LukeShu 8y ago
    Note that libvirt isn't Qemu-specific; it supports multiple backends, including VirtualBox and Xen.
revskill 8y ago
This reminds me of: One clear documentation page is worth thousands hours of man work.
signa11 8y ago
> I lack the competence to seriously contribute. actually, having gone through the process of successfully setting up the vm’s make you eminently suitable to contribute documentation knowledge that is sorely lacking in the online resources you have pointed out. i am sure it will be greatly appreciated and will get improved upon areas that are missing etc.
kashyapc 8y ago
Launching QEMU directly is not merely "tricky", but incredibly inefficient and ineffective once you need something that's beyond bare minimal. E.g. here is a minimal QEMU command-line that gives you access to a serial console in the guest: `qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio` Simple, yeah? Now here (it's too long to post in this comment) is a real world QEMU command-line (as launched by the libvirt[1]): htttps://kashyapc.fedorapeople.org/Fedora-28-QEMU-command-line-by-libvirt.txt Compare and contrast the number of devices that libvirt adds. FWIW, I strongly recommend to use libvirt to launch your QEMU-based guests. It is far more effective, and more importantly, provides security infrastructure like running the QEMU process as an unprivileged user, protecting the QEMU process (and its associated disk iamges) via the SELinux-based sVirt mechanism, and so forth. Check out these[2] slides (recording is online, too) on "Security in QEMU" from one of the QEMU maintainers, Stefan Hajnoczi, at last year's KVM Forum in Edinburgh. [1] http://libvirt.org/ http://libvirt.org/ [2] https://vmsplice.net/~stefan/stefanha-kvm-forum-2018.pdf https://vmsplice.net/~stefan/stefanha-kvm-forum-2018.pdf
- antocv 8y ago
  > like running the QEMU process as an unprivileged user, Anyone can run your example "qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio" from the command line/bash, as non-root user, yet libvirt which requires root to install, and is expecting root to configure it, is better? How? Libvirt is one of those softwares which ads a layer of complexity for the sake of complexity, yet offers nothing of value in return.
  - ori_b 8y ago
    Indeed. Instead of contributing a patch to qemu with better defaults, or even support for config files, someone wrote tens of thousands of lines of complex code. People forget that software is malleable. Bend it to your needs, instead of adding fragile wrapper layers.
bloopernova 8y ago
When I was running a Fedora desktop and was doing a lot of dev/test work on VMs+Foreman+Puppet, I used Libvirt+QEMU as the VM hypervisor. I ran Foreman inside a CentOS VM, which talked to my desktop's Libvirtd to provision new VMs, hand out IPs via DHCP, boot via TFTP, then kickstart. It worked pretty well, and helped me to spin up a vcenter foreman controlled dev environment for a few hundred users in just a few days. My recommendation would be Vagrant + Libvirt.
chapium 8y ago
You can always document your experience, what worked, what didn't work, things you are still unsure about.

tobbyb 8y ago

We need to spin up a lot of VMs of various Linux distributions for testing Flockport and we found the existing tooling too heavy and involved. In the end we rolled our own solution that is much more lightweight, uses QEMU directly and provides better visibility, ease of use and even some potentially useful networking features. Here are a few screencasts [1]. This is mainly designed as a module for Flockport and we will preview this soon. [1] https://www.youtube.com/watch?v=uEnk9m-egeA https://www.youtube.com/watch?v=uEnk9m-egeA

Double_a_92 8y ago

Welcome to the Open Source experience... sadly.

darkwater 8y ago
https://www.virtualbox.org/wiki/Licensing_FAQ https://www.virtualbox.org/wiki/Licensing_FAQ
Retra 8y ago
As opposed to what? Most proprietary tools don't have good documentation either. And then you may not even be able to look at the code to see what it does.
- oblio 8y ago
  I don't get why he's downvoted. Virtualbox (which is not proprietary), VMWare, Hyper-V are way easier to use. That's a fact. And not many people need to look at the code in real life...
  - bonzini 8y ago
    You should compare VirtualBox to virt-manager or GNOME Boxes, not QEMU, just like you shouldn't compare Visual Studio to gcc.
    - oblio 8y ago
      Yeah, but VirtualBox & co. come as a package. virt-manager and Gnome Boxes, as you can see from the other comments, have been missed by various people trying to use these tools :)
    - nurettin 8y ago
      I am on my third confusing error using virt-manager to create an armv7 build environment. First one was some ca-cert error, which meant I forgot to type a slash somewhere in some URL, second was something about not being able to enable firewall, which required firewalld and dnsmasq to be installed, now I'm having an error adding a qcow2 image to my vm setup, it probably doesn't like my mounted storage. We can compare this experience to terrible vendor lockins.

deleted 8y ago

[deleted]

simula67 8y ago

I have not used it, but Drew Devault had a post explaining how to get started with Qemu : https://drewdevault.com/2018/09/10/Getting-started-with-qemu.html https://drewdevault.com/2018/09/10/Getting-started-with-qemu...

aorth 8y ago
After a decade of only using qemu via libvirt on the server, and lazily using VirtualBox on the desktop, this post convinced me to start using qemu locally. I run unprivileged guests for development and testing and it fits my workflow perfectly for about six months now.
AceJohnny2 8y ago
> There’s really no excuse to be using any other hypervisor [1]. They’re all dogshit compared to qemu. > [1] Especially VirtualBox. If you use VirtualBox after reading this article you make poor life choices and are an embarrassment to us all. Well I'm convinced! What a well-presented article! Appropriate Futurama quote: "Remember when I told you about always ending your stories a sentence earlier?". Drew was doing well until he ended with this unnecessary condescending and hostile conclusion (and if you think it's funny, note that "the failure mode of smartass is asshole"). It's a trend of his. He's got interesting and informed things to say, but it's hard to really take him seriously because of this.

emmelaich 8y ago

The Gnome Boxes app is excellent https://wiki.gnome.org/action/show/Apps/Boxes https://wiki.gnome.org/action/show/Apps/Boxes

pferde 8y ago

There are several higher level frameworks that wrap QEMU-KVM and make it easier to use. oVirt, libvirt and proxmox are among my favourites, with the latter being quite polished and with advanced features for HA use.

chousuke 8y ago

Using QEMU directly is a bit tricky and it's not really intended to be used in that fashion by most users, unlike VirtualBox. Both gnome-boxes and virt-manager use QEMU under the hood though, and they provide a point-and-click experience. I use virt-manager myself and I do not think it's any more difficult than VirtualBox is.

vero2 8y ago
We have several windows vms running in virtualbox. All have rdp/vnc for specialised programs. (These are offline). Everytime I tried to migrate to qemu or virt-manager or gnome-boxes it was a real nightmare especially installing SPICE+virtio drivers+ networking. I read in Phoronix the performance of virtualbox is subpar and all KVM/qemu is the way to go. Please, could the OSS community build a easy to use GUI that SPICE+virtio drivers+ networking is made simple. Any one from qemu, I really appreciate the work but still very challenging for sysadmins.
- chousuke 8y ago
  I confess that I haven't run Windows with virt-manager on my own desktop in a long while, but it worked just fine after installing the required drivers at install time. IIRC Redhat distributes signed drivers for Windows for free. The problems I did have were with VMs I tried to convert from VMware to KVM, and those stemmed from Windows' inability (version 7) to properly handle changing hardware. Perhaps it's better at nowadays.
  - naasking 8y ago
    Did this recently. It runs ok, but input and responsiveness latency is very noticeable.
- aasasd 8y ago
  Virt-manager seems to be written in Python. So if libvirt supports the bits you want then with some tinkering in GTK you might be able to bolt the controls onto Virt-manager, at least to get things started.
- 8y ago

BossingAround 8y ago

Rather than using QEMU by itself, it might be advisable to use libvirt..? As somewhat of a newbie to virtualization, that's how I started - googling how to use libvirt and virsh, and I found a lot of resources that helped me to deploy what I needed... What are your painpoints in particular? I.e. which topics would you like to explore deeper and can't due to poor documentation/resources? We could work on improving the resources.

babuskov 8y ago
> What are your painpoints in particular? I believe his pain point is that with Virtual Box you get a very simple GUI where you can get VM up and running very quickly without having to use Google or read any guides at all.
- BossingAround 8y ago
  For that, I use 'Virtual Machine Manager' [1], which works for me extremely well, and is available for major distros. [1] https://virt-manager.org/ https://virt-manager.org/
- philpem 8y ago
  Well, qemu + libvirt + virt-manager works pretty well. Virt-manager adds a very nice VirtualBox-style GUI front-end to libvirt.
- carlavilla 8y ago
  You can use QtEmu (https://gitlab.com/qtemu https://gitlab.com/qtemu)
qalmakka 8y ago
I personally dislike libvirt a lot, and virsh in particular. I think that it's pointlessly complicated and burdensome to use, and that in general the hypervisor-agnosticabstractions provided by libvirt are nullified by the fact that no backend except Qemu really works well, this making them a pointless exercise in futility.

pm215 8y ago

My recommendation for that sort of use would be to look at using a 'management layer' app that sits above QEMU and handles all the fiddly command line details for you -- https://virt-manager.org/ https://virt-manager.org/ (which uses/is part of libvirt) is a popular one, I think. The QEMU project has kind of settled on the philosophy of being the low-level tool which provides you "full manual control" and interfaces for a higher-level UI to make use of. User-friendliness is not really our forte and so we let other projects provide that part.

idoubtit 8y ago
This is the context I was missing when I switched to QEMU. I had not heard about libvirt. A simple "apt search qemu" had shown a few unsatisfying GUIs, but nothing about other layers above qemu. I suggest QEMU's documentation could mention libvirt and its user-friendly interfaces.
- kashyapc 8y ago
  FWIW, QEMU's upstream README does mention libvirt and higher-level managment tools: "QEMU aims to fit into a variety of use cases. It can be invoked directly by users wishing to have full control over its behaviour and settings. It also aims to facilitate integration into higher level management layers, by providing a stable command line interface and monitor API. It is commonly invoked indirectly via the libvirt library when using open source applications such as oVirt, OpenStack and virt-manager." But I get it, not everyone goes in search of a README buried in a Git repository. I myself am not sure if this REAMDE is rendered in an HTML format somewhere. [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=README https://git.qemu.org/?p=qemu.git;a=blob_plain;f=README
- carlavilla 8y ago
  Modern GUI for QEMU: https://gitlab.com/qtemu https://gitlab.com/qtemu
- LukeShu 8y ago
  Note that libvirt isn't Qemu-specific; it supports multiple backends, including VirtualBox and Xen.

revskill 8y ago

This reminds me of: One clear documentation page is worth thousands hours of man work.

signa11 8y ago

> I lack the competence to seriously contribute. actually, having gone through the process of successfully setting up the vm’s make you eminently suitable to contribute documentation knowledge that is sorely lacking in the online resources you have pointed out. i am sure it will be greatly appreciated and will get improved upon areas that are missing etc.

kashyapc 8y ago

Launching QEMU directly is not merely "tricky", but incredibly inefficient and ineffective once you need something that's beyond bare minimal. E.g. here is a minimal QEMU command-line that gives you access to a serial console in the guest: `qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio` Simple, yeah? Now here (it's too long to post in this comment) is a real world QEMU command-line (as launched by the libvirt[1]): htttps://kashyapc.fedorapeople.org/Fedora-28-QEMU-command-line-by-libvirt.txt Compare and contrast the number of devices that libvirt adds. FWIW, I strongly recommend to use libvirt to launch your QEMU-based guests. It is far more effective, and more importantly, provides security infrastructure like running the QEMU process as an unprivileged user, protecting the QEMU process (and its associated disk iamges) via the SELinux-based sVirt mechanism, and so forth. Check out these[2] slides (recording is online, too) on "Security in QEMU" from one of the QEMU maintainers, Stefan Hajnoczi, at last year's KVM Forum in Edinburgh. [1] http://libvirt.org/ http://libvirt.org/ [2] https://vmsplice.net/~stefan/stefanha-kvm-forum-2018.pdf https://vmsplice.net/~stefan/stefanha-kvm-forum-2018.pdf

antocv 8y ago
> like running the QEMU process as an unprivileged user, Anyone can run your example "qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio" from the command line/bash, as non-root user, yet libvirt which requires root to install, and is expecting root to configure it, is better? How? Libvirt is one of those softwares which ads a layer of complexity for the sake of complexity, yet offers nothing of value in return.
- ori_b 8y ago
  Indeed. Instead of contributing a patch to qemu with better defaults, or even support for config files, someone wrote tens of thousands of lines of complex code. People forget that software is malleable. Bend it to your needs, instead of adding fragile wrapper layers.

bloopernova 8y ago

When I was running a Fedora desktop and was doing a lot of dev/test work on VMs+Foreman+Puppet, I used Libvirt+QEMU as the VM hypervisor. I ran Foreman inside a CentOS VM, which talked to my desktop's Libvirtd to provision new VMs, hand out IPs via DHCP, boot via TFTP, then kickstart. It worked pretty well, and helped me to spin up a vcenter foreman controlled dev environment for a few hundred users in just a few days. My recommendation would be Vagrant + Libvirt.

chapium 8y ago

You can always document your experience, what worked, what didn't work, things you are still unsure about.