7 ms·
Linux Hypervisor Setup (Libvirt/QEMU/KVM)
- sneak 5y agoWhat do you all use for quick one-command (non gui) new vm bringup? I haven't had much luck with terraform with libvirt, and I miss Vagrant but don't want to lock myself into its syntax. Maybe I should just start using it again but I feel like there has got to be a good libvirt-based way. Provisioning new test vms with userdata embedded into an kickstart-or-whatever iso that is provided to libvirt also seems clunky. docker-machine is no longer maintained so I can't use it as a vagrant replacement to just bring up sshable hosts. Multipass? Curious as to how others have solved this.
- bonzini 5y agovirt-install or virt-builder.
- sneak 5y agoI have some extremely clunky 50 line shell scripts around virt-install presently but was hoping to find something a little more streamlined (like the usual Vagrant or docker-machine workflow). Do you just manually run a huge virt-install command each time?
- bonzini 5y agoI usually use virt-builder actually. But it's like 3-4 command line options for virt-install, not 50.
- ohyeshedid 5y agoHave you looked into Weaveworks Ignite? https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite
- vegardx 5y agoI was playing around with this a while ago, using cloud-init and virt-install. But I found it to be quite cumbersome if you don't have some kind of metadata service that can provide cloud-init with the user-data, ie. not using the nocloud provider. I started looking at this mock EC2 metadata service, but never did anything with it: https://github.com/sjjf/md_server https://github.com/sjjf/md_server Ninja-edit: It seems like virt-install rencently abstracted the whole NoCloud-provider, this changes a lot of things: https://blog.wikichoon.com/2020/09/virt-install-cloud-init.html https://blog.wikichoon.com/2020/09/virt-install-cloud-init.h...
- SteveNuts 5y agoWe implemented our own mock ec2 metadata service for KVM cloud-init, probably the most fun project I've worked on. The tricky part is intercepting the requests and redirecting them to your service
- madjam002 5y agoYou mention Terraform, I'm using https://github.com/dmacvicar/terraform-provider-libvirt https://github.com/dmacvicar/terraform-provider-libvirt quite heavily and it works a treat.
- sneak 5y agoThat's the one I've had suboptimal experience with. Maybe I need to learn it better. What kind of storage are you using it with? I ran into issues using file-backed trying to keep everything in tf.
- madjam002 5y agoI’m copying qcow2 volumes with it to file based storage and some Ceph RBD as well, I mean AFAIK it just talks to the standard libvirt volumes interface so anything you can do with that you can do with the Terraform provider.
- mazatta 5y agoI've had luck with Proxmox and this Terraform provider: https://registry.terraform.io/providers/Telmate/proxmox/latest/docs https://registry.terraform.io/providers/Telmate/proxmox/late... Other than a working Proxmox install, you'll need to create a cloud-init template, which is documented here: https://pve.proxmox.com/wiki/Cloud-Init_Support https://pve.proxmox.com/wiki/Cloud-Init_Support
- lioeters 5y agoJust wanted to chime in and say that, for my purposes, Multipass has been perfect for work and personal use. https://multipass.run/ https://multipass.run/ (For others, it's a simple-to-use CLI that provisions Ubuntu instances.)
- ofrzeta 5y agovirt-install with kickstart or preseed files and VMs attached to the serial console, like virt-install -n centos -r 1024 --cpu=host -l http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ --disk path=/var/lib/libvirt/images/centos.img,size=10,bus=virtio --boot kernel_args="console=/dev/ttyS0" -w network=internal,model=virtio --initrd-inject=centos.ks -x "ks=file:/centos.ks console=ttyS0" --graphics none Setting up the serial console requires some configuration on the host part, too, unfortunately, depending on the OS.
- abbbi 5y agovagrant with the vagrant-libvirt plugin. The roboxes project provides pre-installed and configured vagrant boxes for allmost all distributions. Its really just a matter of seconds to bring up whatever distribution box you like. Even pre-configured and sysprepped windows images exist.
- hyperpl 5y agoI use qemu and never really understood the need for any further abstraction such as libvirt over it's cli usage. Can anyone tell me what there is to be gained other than needing further setup, packages and systemd services?
- dijit 5y agolibvirt is mainly a consistency wrapper. So you can use kvm or lxc. As for what it “gives”, I suppose I’ve never migrated a VM outside of libvirt, though I’m not 100% sure if that’s not possible with plain old qemu+kvm
- kbumsik 5y agoI am not an expert but libvirt is not only for qemu but it supports various virtualization technologies such as Xen, LXC, VMware. So if you decide to stick with QEMU then I guess you don't need to use libvirt.
- bonzini 5y agoThis is a blog post I wrote about the KVM userspace architecture, it goes into why Red Hat uses Libvirt in its virtualization products. https://www.redhat.com/en/blog/all-you-need-know-about-kvm-userspace https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u... (fixed now)
- dooglius 5y agoGetting a 404
- curiousgal 5y agohttps://www.redhat.com/en/blog/all-you-need-know-about-kvm-userspace https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...
- teilo 5y agoIt's not just about abstraction. VirtIO is significantly more efficient than SATA emulation, and as it is built into the Linux kernel it just works. There is also a Windows driver package that adds VirtIO support, but it's a bit tricky to get it to work when porting in an existing Windows VM. VirtIO also makes it possible to do USB relay.
- HumblyTossed 5y agoProxmox exists.
- invokestatic 5y agoProbably my favorite feature of libvirt is the security and isolation features provided by sVirt. It applies a security policy via SELinux (or AppArmor) that ensures that in the event of a VM breakout exploit, the attacker can only access resources allocated to that VM. So it isolates VMs from the host and from each other. Really cool!
- qwertox 5y agoHow annoying that this page uses a non-standard port for the WebSocket "livereload" feature (1313). It makes my firewall pop up constantly.
- deleted 5y ago[deleted]
- cyberlab 5y agoBrowse with JS disabled by default. Then in uBlock Origin you can temporarily whitelist the page and browse it with JS enabled if it really requires it.
- synergy20 5y agoOne thing is missing comparing to virtualbox, is the bridge-mode especially when host uses wireless which is very common these days. virtualbox hacked bridge-mode and made it working well all the time, with libvirt/qemu/kvm you will need write a lot of scripts and click around to set this up. This is the sole reason I had to stick with virtualbox these years.
- MuffinFlavored 5y agoI'm a big VirtualBox fan as well and I'm sad to see that Apple M1 (ARM) support isn't really a priority.
- numbsafari 5y agoI get the impression Oracle is killing VirtualBox through neglect.
- 2ion 5y agoThen you need to compare to the (for non-commercial use free ) vmware workstation (player) as well. - Simple OpenGL accelerated guest graphics (alternatives: intel gvt-g, card passthrough, virgl in qemu (not generally available), SPICE+qxl in qemu (not nearly as performant)) - bridged networking "just works" - open-vm-tools enable quite an interesting array of comfortable guest integration - disk image format can be handled by qemu-img for conversion purposes, so it's easy to migrate vmware <> virtualbox <> qemu+kvm. The graphics part is why for desktop VMs, I prefer the vmware solution at the moment.
- neurostimulant 5y agoThis is the main virtualbox feature I sorely missed running kvm. Is there any easy way to configure bridge mode on kvm? I didn't dig too much because I also run docker and kubernetes on the same machine and I don't want to break their network config accidentally.
- VMtest 5y agoIf anyone knows how to setup the bridge with wireless in libvirt/qemu/kvm following the steps with/without ebtables listed in Debian and Arch wiki, please do tell 1. https://wiki.archlinux.org/index.php/QEMU#Network_sharing_between_physical_device_and_a_Tap_device_through_iptables https://wiki.archlinux.org/index.php/QEMU#Network_sharing_be... 2. https://wiki.archlinux.org/index.php/Network_bridge#Wireless_interface_on_a_bridge https://wiki.archlinux.org/index.php/Network_bridge#Wireless... 3. https://wiki.debian.org/BridgeNetworkConnections#Bridging_with_a_wireless_NIC https://wiki.debian.org/BridgeNetworkConnections#Bridging_wi...
- psanford 5y agoI've been quite happy with LXD's support for VMs. Its really easy to pull an image an have a running VM with a single command.
- inshadows 5y agoLXD does virtual machines now? I though it does only "containers" (running processes namespaces, like Docker).
- esjeon 5y agoWow, so LXD added VM support! But it seems like still experimental. [1] [1]: https://lxd.readthedocs.io/en/latest/virtual-machines/ https://lxd.readthedocs.io/en/latest/virtual-machines/
- egberts1 5y agoI’ve been using Proxmox community edition for my home datacenter. It’s good to see what this article details into it and how some can pull all this together … for free (or for a small sum). URL: https://proxmox.com/en/proxmox-ve/features https://proxmox.com/en/proxmox-ve/features
- linsomniac 5y agoI was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive dedicated to putting ISOs on, in addition to the OS install and the virtual machine storage. I was trying to set it up on a SFF machine and was limited in what I could put into it and I just kind of noped out of it. I couldn't install any VMs until I did this. I don't recall this being an issue before. I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up. In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.
- ylk 5y agoWhat made you think that you need a separate disk just for ISOs? They can be uploaded to and stored on the default directory storage.
- egberts1 5y agoI snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.) Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup. 8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox . runs very quiet. so now it’s my home datacenter.
- 5y ago
- sascha_sl 5y agolibvirt has too many gotchas to be as easy to use as virtualbox, vmware or proxmox. Proxmox also uses qemu+kvm, but is opinionated enough to pick the most appropriate option by default. Example? For Networking, you probably want the virtio driver for maximum throughput if it is supported on the guest (even Windows has drivers now), using the emulated realtek device/e1000 is very slow. That means you should probably also use the virtio block devices, right? Nope, wrong, work on that was abandoned years ago, use SATA!
- nzmsv 5y agoThe point about storage doesn't seem to be true: https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuration/ https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuratio... However, I am not an expert and would love to learn more. Intuitively though I wouldn't expect emulated SATA to be better than paravirt of any kind.
- zaat 5y agoOne annoyance I had with the defaults are the short dhcp lease time configured for dnsmasq, causing repeated dhcp lease logging in the logs. Turns out you can configure the lease time using virsh net-edit. The libvrit docs contain additional useful configuration options. Another small thing but big quality of life improvement is exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that everything you execute uses the correct connection.
- guerby 5y agoCockpit is a nice web UI for libvirt: https://cockpit-project.org/ https://cockpit-project.org/ On debian add backports then it's as simple as "apt-get install cockpit-machines" then point your browser to http://localhost:9090 http://localhost:9090
- 7kay 5y agoDo I guess correctly that you have cockpit running on the host and manage guests via the cockpit-machines plugin? If so, do you have any hints on where to find documentation about that? A cursory glance at the repo didn't produce much information.
- ffeiek 5y agoQubes OS (Libvirt/QEMU/Xen) would also be good
- xrd 5y agoI've been struggling to run Windows VMs on Linux for a while. Are any of these tools a good option for that? I've installed a Windows VM more times than I can count and it hangs after 20-30 minutes of waiting. If anyone has a suggestion for an all in one tool that can help with this, I'm all ears.
- xrd 5y agocockpit worked great!
- ofrzeta 5y agoAlso useful: - using Vagrant with the Libvirt plugin. Then there's vagrant-mutate for converting Virtualbox boxes to KVM images. - VM snapshots with virsh