4 ms·

Nested virt on x86 is curiously painful; you'd kind of think each layer would be isolated, so that the L0 (hardware) would only have to worry about it's VM (L1)

Nested virt on x86 is curiously painful; you'd kind of think each layer would be isolated, so that the L0 (hardware) would only have to worry about it's VM (L1), and L1 would have to worry about it's VM (L2); but nope - the L0 top level hypervisor sees faults from the L2 and has to figure out that they are actually L2 not L0. IMHO the extra complexity (and historical flakiness of it) - makes me say that enabling nesting is a bad idea for public VM hosts.

deleted 3mo ago
[deleted]
iririririr 3mo ago
but that's the entire point of kvm. that leak is the feature! that's how you get the performance boost.
stinkbeetle 3mo ago
The alternatives are the L0 emulates a hypervisor-privileged mode to the L1 which would be a performance cost (and does not really avoid the problem since the L0 would have to emulate vmenter/vmrun and therefore be aware of the L2 anyway, or the hardware would have to provide some nested virtualization facility which seems like it would be complicated and expensive for little benefit, though I could be mistaken. Does anything do a "real" nested virtualization in hardware? s390 might but I know ~nothing about it and it probably does not expose its bare metal hardware layers to Linux/KVM anyway.
- nijave 3mo ago
  IBM mainframes might but I have a very shallow understanding. Found this https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar
  - bonzini 3mo ago
    That's exactly the same as x86. Nested virtualization support is almost entirely in the hypervisor.
    - nijave 3mo ago
      x86 doesn't have a firmware hypervisor that allows splitting into LPARs. That's showing 3 layers of nesting: LPAR, hypervisor, nested hypervisor
      - bonzini 3mo ago
        On POWER, LPARs and the first level of hypervisor do not use any LPAR-specific nesting support in the processor. It's all handled by the firmware, not the hardware.
- bonzini 3mo ago
  No, even s390 needs shadow paging. Intel and AMD both have some small amount of acceleration of nested virtualization, respectively with shadow VMCS and virtualized VMLOAD/VMSAVE.

deleted 3mo ago

[deleted]

iririririr 3mo ago

but that's the entire point of kvm. that leak is the feature! that's how you get the performance boost.

stinkbeetle 3mo ago

The alternatives are the L0 emulates a hypervisor-privileged mode to the L1 which would be a performance cost (and does not really avoid the problem since the L0 would have to emulate vmenter/vmrun and therefore be aware of the L2 anyway, or the hardware would have to provide some nested virtualization facility which seems like it would be complicated and expensive for little benefit, though I could be mistaken. Does anything do a "real" nested virtualization in hardware? s390 might but I know ~nothing about it and it probably does not expose its bare metal hardware layers to Linux/KVM anyway.

nijave 3mo ago
IBM mainframes might but I have a very shallow understanding. Found this https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar
- bonzini 3mo ago
  That's exactly the same as x86. Nested virtualization support is almost entirely in the hypervisor.
  - nijave 3mo ago
    x86 doesn't have a firmware hypervisor that allows splitting into LPARs. That's showing 3 layers of nesting: LPAR, hypervisor, nested hypervisor
    - bonzini 3mo ago
      On POWER, LPARs and the first level of hypervisor do not use any LPAR-specific nesting support in the processor. It's all handled by the firmware, not the hardware.
bonzini 3mo ago
No, even s390 needs shadow paging. Intel and AMD both have some small amount of acceleration of nested virtualization, respectively with shadow VMCS and virtualized VMLOAD/VMSAVE.