5 ms·

I really like QubesOS, but you cannot run VMs inside a qube, or other things that require VMs like Docker Desktop for Linux, because the xen hypervisor does not

I really like QubesOS, but you cannot run VMs inside a qube, or other things that require VMs like Docker Desktop for Linux, because the xen hypervisor does not support nested virtualization.

frugalmail 3y ago
Double whammy, because sadly you can't run it in a VM either. At least that's my experience.
- Syonyk 3y ago
  If your hypervisor supports nested virtualization, it should work. It'll probably grumble about the lack of IOMMU and various other things, but you should be able to run it in VMWare or KVM, if you're willing to jump through some hoops. Not sure why you'd bother, though. If you're already spinning up VMs, just use that capability. Qubes makes "spinning up a ton of VMs on the iron" a lot easier and more usable.
  - flashback2199 3y ago
    > Not sure why you'd bother Because some software expects to run a VM itself such as Docker Desktop which I said in my original comment.
    - Syonyk 3y ago
      And if you'll note, under your original comment I added some detail on how to enable nested virtualization in Qubes.
      - flashback2199 3y ago
        Replied
  - kj4ips 3y ago
    My understanding is that this is how Qubes itself is tested.
legrande 3y ago
> nested virtualization Such abstraction is very unstable. You can always VNC into a machine from a browser though, so 'vmception' can be achieved.
- moondev 3y ago
  As someone who often runs multiple entire virtualized datacenters on top of ESXi hosts which are actually virtual machines themselves running in ESXi, I highly disagree. Nested virtualization is a game changer in any kind of lab scenario. It also runs great on Chromebooks. The ChromeOS Linux environment is actually a KVM, so launching virtual machines from there is nested but you would never know it (based on my experience with the (framework Chromebook edition). Having 64GB of memory doesn't hurt either of course.
hiatus 3y ago
A qube _is_ a VM though, no? So if you wanted to create a VM, you can create a QubesOS template and instantiate that?
- flashback2199 3y ago
  Let me know how it works for you
  - haswell 3y ago
    Not a Qubes user, just a curious reader; what is the issue with this? It sounds like you’re implying that what parent said isn’t practical. I’m curious why?
    - flashback2199 3y ago
      The issue with trying to hack Docker Desktop and similar tools to boot a qube instead of a VM and somehow hack QubesOS to not isolate those qubes from each other seems self-evident to me and I don't care to explain further.
      - ThePowerOfFuet 3y ago
        Technically correct, but kind of a shitty answer, no?
        flashback2199 3y ago
        Do you want an answer to that question or do you want an argument?
Syonyk 3y ago
You can. It's just neither recommended nor enabled by default. https://forum.qubes-os.org/t/nested-virtualization/14790 https://forum.qubes-os.org/t/nested-virtualization/14790 Poke around /etc/libvirt/libxl and your particular VM's config file. You'll find some lines like: <feature name='vmx' policy='disable'/> <feature name='svm' policy='disable'/> Enable it, and you should have working nested virtualization.
- flashback2199 3y ago
  I did that very thing about a year ago when I still had QubesOS installed, and it did not work. There seems to be a lot of misinformation about this swirling around the web. It simply does not work. There is a post somewhere that confirms it but I don't have the link. Unless the QubesOS devs/maintainers made a 180 degree turn since I tried it and decided to start compiling QubesOS with xen nested virtualization enabled, but I doubt it because their reason was that xen's nested virtualization feature is basically broken anyway.
- flashback2199 3y ago
  Shoot, as soon as I hit reply some neurons lit up and now I remember I was actually able to enable nested virtualization in QubesOS, and the relevant options in the VirtualBox preferences inside a qube became enabled once I did that, but whenever I tried booting any VM the whole system hanged. The same system and BIOS settings worked in Ubuntu to boot a nested VM in VirtualBox, so I think I had the BIOS settings correct. Anyhow, it seemed like a dead-end, so I abandoned it.
  - Syonyk 3y ago
    I'll have to look at it more. I mostly use AMD systems these days, which don't support nested virt in Xen, as I understand it, but it looks like it should work on Intel.
    - flashback2199 3y ago
      I was on Intel when I tried. No worries though, not really planning on trying it again.
yankput 3y ago
Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux? And it should be quite lightweight as it’s just a container… It’s not that I don’t believe you but I don’t understand it… why would you need VM on Linux for Docker? edit: huh https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Docker%20Desktop%20for%20Linux%20runs,a%20consistent%20experience%20across%20platforms https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Dock.... that’s… a bit stupid in my opinion. But you can always just use the default daemon so, eh. whatever. maybe I’m wrong. there are reasons I guess
- flashback2199 3y ago
  It's a good question - docker doesn't require a VM on Linux, but Docker Desktop does. I assume it's to make it basically the same experience as on Docker Desktop on Windows and macOS, but I'm not totally sure. You can install docker the same way one would on a server in a qube in QubesOS and it works fine, I think I tried that once just to be sure, I just wanted to be able to have Docker Desktop. I also didn't want to paint myself into a corner in case I should need to run something else that also expects to be able to run a VM.
- dathinab 3y ago
  > Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux? docker has been playing rather loose with security for quite a long time in the past, so I per-se wouldn't trust it to be secure additionally there are (oversimplified) 3 ways to run it: 1. a "root" deamon, the default docker daemon setup in most distros and the way docker originally was designed to be used. This is needed for some advanced (non secure) features (which IMHO should not exist) but has pretty serve security issues IMHO, like either being in the docker group being a de-facto root/sudo or it forcing you to use sudo all the time and bugs in the docker deamon potentially leading to catastrophic exploits. 2. as a unprivileged user using cgroupsv2 like podman does by default, this has some limitations but works really well and as long as "linux containers" are secure it should be secure (oversimplified). Still the security of containers isn't perfect as they share the kernel. As a rule of thump only use if you get reliable kernel security updates. 3. starting VMs, much more overhead in many ways and needing virtualization, but works with old "outdated" kernels or hardened kernels(1) not allowing unprivileged cgroups The 3rd method is the most secure and works across most dirstros/linux setups even if unusual setup and is the same kind of path docker takes on windows/mac. So no wonder a product like Docker Desktop uses it. Also the overhead of virtualization is today not "that" bad. The the first method is still the default method for docker CLI on most linux distros is honestly far beyond my understanding. In the past pre reliable cgroups v2, maybe, but today omg no. It has some valid use cases, but the default should be VMs or unprivileged containers especially on desktop systems (whatever distro maintainers prefer). ---- (1): On instresting conflict is security hardening vs. unprivileged cgroups/containers. On one side unprivileged cgroups allow a lot of additional security. For example they allow running various sandboxes without needing a suid binary or using sudo for setting up the sandbox. Which is really grate. At the same time (oversimplified) it increases the security sensitive kernel interface by a non negligible margin, which isn't grate and hence why hardened linux often disables it by default. EDIT: Sorry for the bad spelling, I need to go to bed.
deleted 3y ago
[deleted]

I really like QubesOS, but you cannot run VMs inside a qube, or other things that require VMs like Docker Desktop for Linux, because the xen hypervisor does not support nested virtualization.

frugalmail 3y ago

Double whammy, because sadly you can't run it in a VM either. At least that's my experience.

Syonyk 3y ago
If your hypervisor supports nested virtualization, it should work. It'll probably grumble about the lack of IOMMU and various other things, but you should be able to run it in VMWare or KVM, if you're willing to jump through some hoops. Not sure why you'd bother, though. If you're already spinning up VMs, just use that capability. Qubes makes "spinning up a ton of VMs on the iron" a lot easier and more usable.
- flashback2199 3y ago
  > Not sure why you'd bother Because some software expects to run a VM itself such as Docker Desktop which I said in my original comment.
  - Syonyk 3y ago
    And if you'll note, under your original comment I added some detail on how to enable nested virtualization in Qubes.
    - flashback2199 3y ago
      Replied
- kj4ips 3y ago
  My understanding is that this is how Qubes itself is tested.

legrande 3y ago

> nested virtualization Such abstraction is very unstable. You can always VNC into a machine from a browser though, so 'vmception' can be achieved.

moondev 3y ago
As someone who often runs multiple entire virtualized datacenters on top of ESXi hosts which are actually virtual machines themselves running in ESXi, I highly disagree. Nested virtualization is a game changer in any kind of lab scenario. It also runs great on Chromebooks. The ChromeOS Linux environment is actually a KVM, so launching virtual machines from there is nested but you would never know it (based on my experience with the (framework Chromebook edition). Having 64GB of memory doesn't hurt either of course.

hiatus 3y ago

A qube _is_ a VM though, no? So if you wanted to create a VM, you can create a QubesOS template and instantiate that?

flashback2199 3y ago
Let me know how it works for you
- haswell 3y ago
  Not a Qubes user, just a curious reader; what is the issue with this? It sounds like you’re implying that what parent said isn’t practical. I’m curious why?
  - flashback2199 3y ago
    The issue with trying to hack Docker Desktop and similar tools to boot a qube instead of a VM and somehow hack QubesOS to not isolate those qubes from each other seems self-evident to me and I don't care to explain further.
    - ThePowerOfFuet 3y ago
      Technically correct, but kind of a shitty answer, no?
      - flashback2199 3y ago
        Do you want an answer to that question or do you want an argument?

Syonyk 3y ago

You can. It's just neither recommended nor enabled by default. https://forum.qubes-os.org/t/nested-virtualization/14790 https://forum.qubes-os.org/t/nested-virtualization/14790 Poke around /etc/libvirt/libxl and your particular VM's config file. You'll find some lines like: <feature name='vmx' policy='disable'/> <feature name='svm' policy='disable'/> Enable it, and you should have working nested virtualization.

flashback2199 3y ago
I did that very thing about a year ago when I still had QubesOS installed, and it did not work. There seems to be a lot of misinformation about this swirling around the web. It simply does not work. There is a post somewhere that confirms it but I don't have the link. Unless the QubesOS devs/maintainers made a 180 degree turn since I tried it and decided to start compiling QubesOS with xen nested virtualization enabled, but I doubt it because their reason was that xen's nested virtualization feature is basically broken anyway.
flashback2199 3y ago
Shoot, as soon as I hit reply some neurons lit up and now I remember I was actually able to enable nested virtualization in QubesOS, and the relevant options in the VirtualBox preferences inside a qube became enabled once I did that, but whenever I tried booting any VM the whole system hanged. The same system and BIOS settings worked in Ubuntu to boot a nested VM in VirtualBox, so I think I had the BIOS settings correct. Anyhow, it seemed like a dead-end, so I abandoned it.
- Syonyk 3y ago
  I'll have to look at it more. I mostly use AMD systems these days, which don't support nested virt in Xen, as I understand it, but it looks like it should work on Intel.
  - flashback2199 3y ago
    I was on Intel when I tried. No worries though, not really planning on trying it again.

yankput 3y ago

Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux? And it should be quite lightweight as it’s just a container… It’s not that I don’t believe you but I don’t understand it… why would you need VM on Linux for Docker? edit: huh https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Docker%20Desktop%20for%20Linux%20runs,a%20consistent%20experience%20across%20platforms https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Dock.... that’s… a bit stupid in my opinion. But you can always just use the default daemon so, eh. whatever. maybe I’m wrong. there are reasons I guess

flashback2199 3y ago
It's a good question - docker doesn't require a VM on Linux, but Docker Desktop does. I assume it's to make it basically the same experience as on Docker Desktop on Windows and macOS, but I'm not totally sure. You can install docker the same way one would on a server in a qube in QubesOS and it works fine, I think I tried that once just to be sure, I just wanted to be able to have Docker Desktop. I also didn't want to paint myself into a corner in case I should need to run something else that also expects to be able to run a VM.
dathinab 3y ago
> Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux? docker has been playing rather loose with security for quite a long time in the past, so I per-se wouldn't trust it to be secure additionally there are (oversimplified) 3 ways to run it: 1. a "root" deamon, the default docker daemon setup in most distros and the way docker originally was designed to be used. This is needed for some advanced (non secure) features (which IMHO should not exist) but has pretty serve security issues IMHO, like either being in the docker group being a de-facto root/sudo or it forcing you to use sudo all the time and bugs in the docker deamon potentially leading to catastrophic exploits. 2. as a unprivileged user using cgroupsv2 like podman does by default, this has some limitations but works really well and as long as "linux containers" are secure it should be secure (oversimplified). Still the security of containers isn't perfect as they share the kernel. As a rule of thump only use if you get reliable kernel security updates. 3. starting VMs, much more overhead in many ways and needing virtualization, but works with old "outdated" kernels or hardened kernels(1) not allowing unprivileged cgroups The 3rd method is the most secure and works across most dirstros/linux setups even if unusual setup and is the same kind of path docker takes on windows/mac. So no wonder a product like Docker Desktop uses it. Also the overhead of virtualization is today not "that" bad. The the first method is still the default method for docker CLI on most linux distros is honestly far beyond my understanding. In the past pre reliable cgroups v2, maybe, but today omg no. It has some valid use cases, but the default should be VMs or unprivileged containers especially on desktop systems (whatever distro maintainers prefer). ---- (1): On instresting conflict is security hardening vs. unprivileged cgroups/containers. On one side unprivileged cgroups allow a lot of additional security. For example they allow running various sandboxes without needing a suid binary or using sudo for setting up the sandbox. Which is really grate. At the same time (oversimplified) it increases the security sensitive kernel interface by a non negligible margin, which isn't grate and hence why hardened linux often disables it by default. EDIT: Sorry for the bad spelling, I need to go to bed.

deleted 3y ago

[deleted]