3 ms·

What would you recommend for debugging nested virt, i.e. with Xen or KVM as the 'guest'? https://github.com/Wenzel/pyvmidbg https://github.com/Wenzel/pyvmidbg

What would you recommend for debugging nested virt, i.e. with Xen or KVM as the 'guest'? https://github.com/Wenzel/pyvmidbg https://github.com/Wenzel/pyvmidbg is also using libvmi with Xen.

smichaels 8y ago
I can only speak for Xen, since I don't have any experience with KVM at the moment. That aside, I've never tried to debug nested VMs, as my research never required doing so, but in theory xendbg should be able to do it at any level as long as your inner Xen VMs can run a separate process for the debug stub. Let's say you have a setup like so. Base OS with Xen \-> Outer VM with Xen \-> Inner VM, e.g. a unikernel Xendbg only requires that you run the debug stub in ring 0 on the machine that is running Xen, whose VMs you want to debug. If the base substrate is a general-purpose OS, you can of course run xendbg on it, in which case you'd be able to inspect the first-level VM as it runs, which would allow you to debug Xen itself as it runs on that VM. Alternately, you can put the debug stub directly on the outer VM and then expose the debug bridge to the top-level OS, and thereby debug the inner VM from the base OS. Xendbg doesn't use libvmi. In fact, if libvmi had been sufficient for my original uniknernel debugging use case, I probably wouldn't have ended up writing xendbg. The main issue is that libvmi makes the assumption that the guest is either Linux or Windows, and doesn't fully support PV guests --- e.g. its Xen driver creates breakpoints using HVM-specific monitoring APIs. The unikernels I was looking at (Rumprun, IncludeOS, Mirage) at are all PV-only. Speaking of pyvmidbg, I have recently been talking with the author of pyvmidbg (Matthieu Tarral) about collaborating on a new libvmi alternative. We're just in the early stages of thinking about it, though, so I don't want to make any promises about features or release timelines…

What would you recommend for debugging nested virt, i.e. with Xen or KVM as the 'guest'? https://github.com/Wenzel/pyvmidbg https://github.com/Wenzel/pyvmidbg is also using libvmi with Xen.

smichaels 8y ago

I can only speak for Xen, since I don't have any experience with KVM at the moment. That aside, I've never tried to debug nested VMs, as my research never required doing so, but in theory xendbg should be able to do it at any level as long as your inner Xen VMs can run a separate process for the debug stub. Let's say you have a setup like so. Base OS with Xen \-> Outer VM with Xen \-> Inner VM, e.g. a unikernel Xendbg only requires that you run the debug stub in ring 0 on the machine that is running Xen, whose VMs you want to debug. If the base substrate is a general-purpose OS, you can of course run xendbg on it, in which case you'd be able to inspect the first-level VM as it runs, which would allow you to debug Xen itself as it runs on that VM. Alternately, you can put the debug stub directly on the outer VM and then expose the debug bridge to the top-level OS, and thereby debug the inner VM from the base OS. Xendbg doesn't use libvmi. In fact, if libvmi had been sufficient for my original uniknernel debugging use case, I probably wouldn't have ended up writing xendbg. The main issue is that libvmi makes the assumption that the guest is either Linux or Windows, and doesn't fully support PV guests --- e.g. its Xen driver creates breakpoints using HVM-specific monitoring APIs. The unikernels I was looking at (Rumprun, IncludeOS, Mirage) at are all PV-only. Speaking of pyvmidbg, I have recently been talking with the author of pyvmidbg (Matthieu Tarral) about collaborating on a new libvmi alternative. We're just in the early stages of thinking about it, though, so I don't want to make any promises about features or release timelines…