9 ms·
Linux Hypervisor Setup (Libvirt/QEMU/KVM)
- sneak 6y agoWhat do you all use for quick one-command (non gui) new vm bringup? I haven't had much luck with terraform with libvirt, and I miss Vagrant but don't want to lock myself into its syntax. Maybe I should just start using it again but I feel like there has got to be a good libvirt-based way. Provisioning new test vms with userdata embedded into an kickstart-or-whatever iso that is provided to libvirt also seems clunky. docker-machine is no longer maintained so I can't use it as a vagrant replacement to just bring up sshable hosts. Multipass? Curious as to how others have solved this.
- bonzini 6y agovirt-install or virt-builder.
- sneak 6y agoI have some extremely clunky 50 line shell scripts around virt-install presently but was hoping to find something a little more streamlined (like the usual Vagrant or docker-machine workflow). Do you just manually run a huge virt-install command each time?
- bonzini 6y agoI usually use virt-builder actually. But it's like 3-4 command line options for virt-install, not 50.
- ohyeshedid 6y agoHave you looked into Weaveworks Ignite? https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite
- vegardx 6y agoI was playing around with this a while ago, using cloud-init and virt-install. But I found it to be quite cumbersome if you don't have some kind of metadata service that can provide cloud-init with the user-data, ie. not using the nocloud provider. I started looking at this mock EC2 metadata service, but never did anything with it: https://github.com/sjjf/md_server https://github.com/sjjf/md_server Ninja-edit: It seems like virt-install rencently abstracted the whole NoCloud-provider, this changes a lot of things: https://blog.wikichoon.com/2020/09/virt-install-cloud-init.html https://blog.wikichoon.com/2020/09/virt-install-cloud-init.h...
- SteveNuts 6y agoWe implemented our own mock ec2 metadata service for KVM cloud-init, probably the most fun project I've worked on. The tricky part is intercepting the requests and redirecting them to your service
- madjam002 6y agoYou mention Terraform, I'm using https://github.com/dmacvicar/terraform-provider-libvirt https://github.com/dmacvicar/terraform-provider-libvirt quite heavily and it works a treat.
- sneak 6y agoThat's the one I've had suboptimal experience with. Maybe I need to learn it better. What kind of storage are you using it with? I ran into issues using file-backed trying to keep everything in tf.
- madjam002 6y agoI’m copying qcow2 volumes with it to file based storage and some Ceph RBD as well, I mean AFAIK it just talks to the standard libvirt volumes interface so anything you can do with that you can do with the Terraform provider.
- mazatta 6y agoI've had luck with Proxmox and this Terraform provider: https://registry.terraform.io/providers/Telmate/proxmox/latest/docs https://registry.terraform.io/providers/Telmate/proxmox/late... Other than a working Proxmox install, you'll need to create a cloud-init template, which is documented here: https://pve.proxmox.com/wiki/Cloud-Init_Support https://pve.proxmox.com/wiki/Cloud-Init_Support
- lioeters 6y agoJust wanted to chime in and say that, for my purposes, Multipass has been perfect for work and personal use. https://multipass.run/ https://multipass.run/ (For others, it's a simple-to-use CLI that provisions Ubuntu instances.)
- ofrzeta 6y agovirt-install with kickstart or preseed files and VMs attached to the serial console, like virt-install -n centos -r 1024 --cpu=host -l http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ --disk path=/var/lib/libvirt/images/centos.img,size=10,bus=virtio --boot kernel_args="console=/dev/ttyS0" -w network=internal,model=virtio --initrd-inject=centos.ks -x "ks=file:/centos.ks console=ttyS0" --graphics none Setting up the serial console requires some configuration on the host part, too, unfortunately, depending on the OS.
- abbbi 6y agovagrant with the vagrant-libvirt plugin. The roboxes project provides pre-installed and configured vagrant boxes for allmost all distributions. Its really just a matter of seconds to bring up whatever distribution box you like. Even pre-configured and sysprepped windows images exist.
- hyperpl 6y agoI use qemu and never really understood the need for any further abstraction such as libvirt over it's cli usage. Can anyone tell me what there is to be gained other than needing further setup, packages and systemd services?
- dijit 6y agolibvirt is mainly a consistency wrapper. So you can use kvm or lxc. As for what it “gives”, I suppose I’ve never migrated a VM outside of libvirt, though I’m not 100% sure if that’s not possible with plain old qemu+kvm
- kbumsik 6y agoI am not an expert but libvirt is not only for qemu but it supports various virtualization technologies such as Xen, LXC, VMware. So if you decide to stick with QEMU then I guess you don't need to use libvirt.
- bonzini 6y agoThis is a blog post I wrote about the KVM userspace architecture, it goes into why Red Hat uses Libvirt in its virtualization products. https://www.redhat.com/en/blog/all-you-need-know-about-kvm-userspace https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u... (fixed now)
- dooglius 6y agoGetting a 404
- curiousgal 6y agohttps://www.redhat.com/en/blog/all-you-need-know-about-kvm-userspace https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...
- teilo 6y agoIt's not just about abstraction. VirtIO is significantly more efficient than SATA emulation, and as it is built into the Linux kernel it just works. There is also a Windows driver package that adds VirtIO support, but it's a bit tricky to get it to work when porting in an existing Windows VM. VirtIO also makes it possible to do USB relay.
- HumblyTossed 6y agoProxmox exists.
- invokestatic 6y agoProbably my favorite feature of libvirt is the security and isolation features provided by sVirt. It applies a security policy via SELinux (or AppArmor) that ensures that in the event of a VM breakout exploit, the attacker can only access resources allocated to that VM. So it isolates VMs from the host and from each other. Really cool!
- qwertox 6y agoHow annoying that this page uses a non-standard port for the WebSocket "livereload" feature (1313). It makes my firewall pop up constantly.
- deleted 6y ago[deleted]
- cyberlab 6y agoBrowse with JS disabled by default. Then in uBlock Origin you can temporarily whitelist the page and browse it with JS enabled if it really requires it.
- synergy20 6y agoOne thing is missing comparing to virtualbox, is the bridge-mode especially when host uses wireless which is very common these days. virtualbox hacked bridge-mode and made it working well all the time, with libvirt/qemu/kvm you will need write a lot of scripts and click around to set this up. This is the sole reason I had to stick with virtualbox these years.
- MuffinFlavored 6y agoI'm a big VirtualBox fan as well and I'm sad to see that Apple M1 (ARM) support isn't really a priority.
- numbsafari 6y agoI get the impression Oracle is killing VirtualBox through neglect.
- 2ion 6y agoThen you need to compare to the (for non-commercial use free ) vmware workstation (player) as well. - Simple OpenGL accelerated guest graphics (alternatives: intel gvt-g, card passthrough, virgl in qemu (not generally available), SPICE+qxl in qemu (not nearly as performant)) - bridged networking "just works" - open-vm-tools enable quite an interesting array of comfortable guest integration - disk image format can be handled by qemu-img for conversion purposes, so it's easy to migrate vmware <> virtualbox <> qemu+kvm. The graphics part is why for desktop VMs, I prefer the vmware solution at the moment.
- neurostimulant 6y agoThis is the main virtualbox feature I sorely missed running kvm. Is there any easy way to configure bridge mode on kvm? I didn't dig too much because I also run docker and kubernetes on the same machine and I don't want to break their network config accidentally.
- VMtest 6y agoIf anyone knows how to setup the bridge with wireless in libvirt/qemu/kvm following the steps with/without ebtables listed in Debian and Arch wiki, please do tell 1. https://wiki.archlinux.org/index.php/QEMU#Network_sharing_between_physical_device_and_a_Tap_device_through_iptables https://wiki.archlinux.org/index.php/QEMU#Network_sharing_be... 2. https://wiki.archlinux.org/index.php/Network_bridge#Wireless_interface_on_a_bridge https://wiki.archlinux.org/index.php/Network_bridge#Wireless... 3. https://wiki.debian.org/BridgeNetworkConnections#Bridging_with_a_wireless_NIC https://wiki.debian.org/BridgeNetworkConnections#Bridging_wi...
- psanford 6y agoI've been quite happy with LXD's support for VMs. Its really easy to pull an image an have a running VM with a single command.
- inshadows 6y agoLXD does virtual machines now? I though it does only "containers" (running processes namespaces, like Docker).
- esjeon 6y agoWow, so LXD added VM support! But it seems like still experimental. [1] [1]: https://lxd.readthedocs.io/en/latest/virtual-machines/ https://lxd.readthedocs.io/en/latest/virtual-machines/
- egberts1 6y agoI’ve been using Proxmox community edition for my home datacenter. It’s good to see what this article details into it and how some can pull all this together … for free (or for a small sum). URL: https://proxmox.com/en/proxmox-ve/features https://proxmox.com/en/proxmox-ve/features
- linsomniac 6y agoI was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive dedicated to putting ISOs on, in addition to the OS install and the virtual machine storage. I was trying to set it up on a SFF machine and was limited in what I could put into it and I just kind of noped out of it. I couldn't install any VMs until I did this. I don't recall this being an issue before. I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up. In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.
- ylk 6y agoWhat made you think that you need a separate disk just for ISOs? They can be uploaded to and stored on the default directory storage.
- egberts1 6y agoI snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.) Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup. 8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox . runs very quiet. so now it’s my home datacenter.
- 6y ago
- sascha_sl 6y agolibvirt has too many gotchas to be as easy to use as virtualbox, vmware or proxmox. Proxmox also uses qemu+kvm, but is opinionated enough to pick the most appropriate option by default. Example? For Networking, you probably want the virtio driver for maximum throughput if it is supported on the guest (even Windows has drivers now), using the emulated realtek device/e1000 is very slow. That means you should probably also use the virtio block devices, right? Nope, wrong, work on that was abandoned years ago, use SATA!
- nzmsv 6y agoThe point about storage doesn't seem to be true: https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuration/ https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuratio... However, I am not an expert and would love to learn more. Intuitively though I wouldn't expect emulated SATA to be better than paravirt of any kind.
- zaat 6y agoOne annoyance I had with the defaults are the short dhcp lease time configured for dnsmasq, causing repeated dhcp lease logging in the logs. Turns out you can configure the lease time using virsh net-edit. The libvrit docs contain additional useful configuration options. Another small thing but big quality of life improvement is exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that everything you execute uses the correct connection.
- guerby 6y agoCockpit is a nice web UI for libvirt: https://cockpit-project.org/ https://cockpit-project.org/ On debian add backports then it's as simple as "apt-get install cockpit-machines" then point your browser to http://localhost:9090 http://localhost:9090
- 7kay 6y agoDo I guess correctly that you have cockpit running on the host and manage guests via the cockpit-machines plugin? If so, do you have any hints on where to find documentation about that? A cursory glance at the repo didn't produce much information.
- ffeiek 6y agoQubes OS (Libvirt/QEMU/Xen) would also be good
- xrd 6y agoI've been struggling to run Windows VMs on Linux for a while. Are any of these tools a good option for that? I've installed a Windows VM more times than I can count and it hangs after 20-30 minutes of waiting. If anyone has a suggestion for an all in one tool that can help with this, I'm all ears.
- xrd 6y agocockpit worked great!
- ofrzeta 6y agoAlso useful: - using Vagrant with the Libvirt plugin. Then there's vagrant-mutate for converting Virtualbox boxes to KVM images. - VM snapshots with virsh