Latency and the ability to switch between the VM and the host. I have a cheap USB input switcher which I've owned for ages for this purpose. IIUC unless you're

Latency and the ability to switch between the VM and the host. I have a cheap USB input switcher which I've owned for ages for this purpose. IIUC unless you're running Looking Glass or some other alternative method, passing through each device individually locks them while the VM is running.