5 ms·

Are you saying that Amazon does not support nested virtulization because it is too broken and is unusable technology. I've never heard that before. Can you re

Are you saying that Amazon does not support nested virtulization because it is too broken and is unusable technology. I've never heard that before. Can you reference anything tangible to say that nested virtualization is too buggy to use in production?

tyingq 8y ago
Redhat seems wary. https://www.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests https://www.redhat.com/en/blog/inception-how-usable-are-nest... "While Red Hat is now doing some level of QA for cascaded KVM, we are not supporting it - so it is clearly not meant for production use."
- techntoke 8y ago
  Which means absolutely nothing. Not for production use to them likely means they don't have a functional product to support it, not that it doesn't work.
  - Thaxll 8y ago
    Well given the expertise that RedHat has with KVM, it's a big warning "don't run nested kvm for now", it's pretty clear, what more do you need? https://www.linux-kvm.org/page/Nested_Guests https://www.linux-kvm.org/page/Nested_Guests "As of Feb 2018 this feature is considered working but experimental, and some limitations apply."
    - techntoke 8y ago
      It is enabled by default in the latest stable kernel: https://www.phoronix.com/scan.php?page=news_item&px=KVM-Linux-4.20-Features https://www.phoronix.com/scan.php?page=news_item&px=KVM-Linu...
vishvananda 8y ago
My evidence is mostly anecdotal, unfortunately. Most of my experimentation was KVM in KVM from about 2012 and i saw frequent long pauses in the kernel, lockups, and kernel panics. At the time, the attitude from the qemu-kvm community was that it wasn't a high priority and that it may or may not work. It is possible that it is better now, but redhat's opinion[1] as of last year was still that it wasn't supported in production. I don't know if it is any better with other hypervisors. Last week, for example, I ran into an issue with nested virt in vmware fusion[2] that prevented me from running kvm in a guest. I would need a bunch of testing before I trusted it. It looks like xen has spotty support[3] as well. Amazon uses a heavily forked version of xen, so I suspect support for it is even worse on their version. (EDIT: added info on xen) [1]: https://www.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests https://www.redhat.com/en/blog/inception-how-usable-are-nest... [2]: https://bugs.launchpad.net/qemu/+bug/1661386 https://bugs.launchpad.net/qemu/+bug/1661386 [3]: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe...
abdulla 8y ago
Working on a hypervisor, I see this. It mostly crops up as things like IPIs being more costly when nested.
rwmj 8y ago
Nested KVM has been buggy since forever. Although a wonderful thing when it's working I'm not surprised Amazon doesn't allow it on their production systems. It has a nasty combination of: ability to crash the L0 hypervisor, different behaviour depending on features and family of the processor, and requiring very specialized skills to understand and debug. Edit: If you do want to try testing nested KVM, then try my supernested project: http://git.annexia.org/?p=supernested.git;a=summary http://git.annexia.org/?p=supernested.git;a=summary and Day 13 of the 2016 QEMU Advent Calendar https://www.qemu-advent-calendar.org/2016/ https://www.qemu-advent-calendar.org/2016/ Edit 2: Don't confused nested KVM with nesting TCG. That's fine, albeit rather slow.
- andrewstuart 8y ago
  >> ability to crash the L0 hypervisor Prove it and you'll get a bug bounty from Doogle and Microsoft won't you?
  - rwmj 8y ago
    Why would Microsoft be interested? (no idea what "Doogle" is - do you mean Google? Why would they care about nested KVM?) Anyway when I find bugs in nested KVM I have reported them to the kernel developers or QEMU in the usual way and help any way I can to get them fixed.

tyingq 8y ago

Redhat seems wary. https://www.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests https://www.redhat.com/en/blog/inception-how-usable-are-nest... "While Red Hat is now doing some level of QA for cascaded KVM, we are not supporting it - so it is clearly not meant for production use."

techntoke 8y ago
Which means absolutely nothing. Not for production use to them likely means they don't have a functional product to support it, not that it doesn't work.
- Thaxll 8y ago
  Well given the expertise that RedHat has with KVM, it's a big warning "don't run nested kvm for now", it's pretty clear, what more do you need? https://www.linux-kvm.org/page/Nested_Guests https://www.linux-kvm.org/page/Nested_Guests "As of Feb 2018 this feature is considered working but experimental, and some limitations apply."
  - techntoke 8y ago
    It is enabled by default in the latest stable kernel: https://www.phoronix.com/scan.php?page=news_item&px=KVM-Linux-4.20-Features https://www.phoronix.com/scan.php?page=news_item&px=KVM-Linu...

vishvananda 8y ago

My evidence is mostly anecdotal, unfortunately. Most of my experimentation was KVM in KVM from about 2012 and i saw frequent long pauses in the kernel, lockups, and kernel panics. At the time, the attitude from the qemu-kvm community was that it wasn't a high priority and that it may or may not work. It is possible that it is better now, but redhat's opinion[1] as of last year was still that it wasn't supported in production. I don't know if it is any better with other hypervisors. Last week, for example, I ran into an issue with nested virt in vmware fusion[2] that prevented me from running kvm in a guest. I would need a bunch of testing before I trusted it. It looks like xen has spotty support[3] as well. Amazon uses a heavily forked version of xen, so I suspect support for it is even worse on their version. (EDIT: added info on xen) [1]: https://www.redhat.com/en/blog/inception-how-usable-are-nested-kvm-guests https://www.redhat.com/en/blog/inception-how-usable-are-nest... [2]: https://bugs.launchpad.net/qemu/+bug/1661386 https://bugs.launchpad.net/qemu/+bug/1661386 [3]: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe...

abdulla 8y ago

Working on a hypervisor, I see this. It mostly crops up as things like IPIs being more costly when nested.

rwmj 8y ago

Nested KVM has been buggy since forever. Although a wonderful thing when it's working I'm not surprised Amazon doesn't allow it on their production systems. It has a nasty combination of: ability to crash the L0 hypervisor, different behaviour depending on features and family of the processor, and requiring very specialized skills to understand and debug. Edit: If you do want to try testing nested KVM, then try my supernested project: http://git.annexia.org/?p=supernested.git;a=summary http://git.annexia.org/?p=supernested.git;a=summary and Day 13 of the 2016 QEMU Advent Calendar https://www.qemu-advent-calendar.org/2016/ https://www.qemu-advent-calendar.org/2016/ Edit 2: Don't confused nested KVM with nesting TCG. That's fine, albeit rather slow.

andrewstuart 8y ago
>> ability to crash the L0 hypervisor Prove it and you'll get a bug bounty from Doogle and Microsoft won't you?
- rwmj 8y ago
  Why would Microsoft be interested? (no idea what "Doogle" is - do you mean Google? Why would they care about nested KVM?) Anyway when I find bugs in nested KVM I have reported them to the kernel developers or QEMU in the usual way and help any way I can to get them fixed.