4 ms·
If I'm not wrong, the pre-allocation of I/O ranges in PCIe bridges is needed only if you intend to hot-plug devices that were not present in the first enumerati
by EmilioPeJu 3y ago
If I'm not wrong, the pre-allocation of I/O ranges in PCIe bridges is needed only if you intend to hot-plug devices that were not present in the first enumeration.. but in VMs the hardware is known from the start and the PCIe enumeration can assign I/O ranges only if devices underneath actually needs them... is there a reason why hot-plugging is needed in VMs?
- xyzzyz 3y agoCloud customers love it when they can just attach stuff to their VMs without having to recreate them or even reboot them.
- josephcsible 3y agoIsn't the cloud notoriously worse about hotplugging anything than on-prem systems are? For example, vSphere supports hot adding CPUs and RAM to VMs, but Azure doesn't.
- tyingq 3y agoSeems unsurprising. On Azure, if it goes wrong, the various tenants aren't all working at the same company.
- mgerdts 3y ago> is needed only if you intend to hot-plug devices that were not present in the first enumeration Correct. I regularly use VMs with more that 14 statically configured PCI devices using QEMU with libvirt without having to resort to qemu:cmdline.
- dottedmag 3y agoAuthor here. Have you got it working with PCI or PCIe? PCI devices attached to the top-level bus do not request I/O ports unless they need to, and if they do, they request only small slice. QEMU also allows one to put 8 static PCIe devices into a single "multifunction PCIe device", so it requests 4K I/O ports per 8 devices, giving a bigger headroom. The downside, of course, is that all these 8 devices lose individual hotpluggability, and can only be added/removed en masse. The biggest problem is hotplug slots, each taking 4K I/O ports unless told otherwise in a way libvirt does not support as I described in the article.
- mgerdts 3y agoThere are 14 hardware based PCIe devices (mixture of NVME drives and NIC VFs) along with various other emulated PCI devices (virtio block, serial, etc.). I have not tried to hot [un]plug devices with this configuration. It looks as though I’m likely to be disappointed if I try. Thanks for the explanation.
- jeffreygoesto 3y agoDevices passed from the host to the guest?
- mgerdts 3y agoHot plugging refers to adding devices to the VM while the VM is running. Passing host devices through is commonly accomplished without hotplug.
- jeffreygoesto 3y agoRight. I should have been more clear that it can be you hotplug a host device and pass it in. Admitted, this is typically USB and not PCIe. And the PCMCIA days are over...
- Nextgrid 3y agoIt’s not a common use-case but I could see it being useful for sharing hardware that requires exclusive access like GPUs/ML accelerators. Currently if you need GPUs they come with the instance itself meaning you need to boot your VM from scratch, do the work and then shut it down to relinquish the GPU. With hot-plug you could have continuously running VMs that only attach/detach GPUs as needed, no longer taking the overhead of a full cold boot/shutdown every time.
- adql 3y agoadding NVMe emulated storage would be one
- dottedmag 3y agoAuthor here. As correctly guessed in other comments: cloud infrastructure. To make public IPs and volumes hotpluggable without a guest agent running inside every VM one has to manage them in a way guest OS will handle hotplug using regular mechanisms. For volumes it's PCIe storage hotplug, for public IPs it's PCIe network card hotplug. If a VM is used as a Kubernetes worker, couple of dozen volumes and public IPs attached is not an unlikely situation.