7 ms·

VM Escape: QEMU Case Study

100 points by 0xFFC 9y ago

j_s 9y ago
If for some reason using QEMU with potentially hostile code (not recommended!), disable all the hardware emulation you can!
- rwmj 9y ago
  You should keep qemu up to date of course, and remove devices you don't need. But more importantly ensure you are using it through libvirt. Libvirt adds an SELinux or AppArmor layer, and a container, and enables seccomp for the qemu process. So even if the qemu process is compromised, it is very limited in what it can do to the host.
  - sdeziel 9y ago
    Unfortunately on Ubuntu, seccomp is opt-in only. Works well when enabled though.
  - aseipp 9y ago
    FWIW I've never really been impressed with QEMU's seccomp sandboxing, because it's almost entirely bolted on, and in the grand scheme, I think, does little. Does libvirt do anything substantially different or just "turn it on"? If you look at the list of syscalls whitelisted[1], it's almost a joke -- you could literally do just about anything given those syscalls if you actually had code execution... And every one of those syscalls is a direct interface to the kernel, which becomes the weakest link in a container/LSM environment where your code will be running, presumably. How is seccomp going to stop my exploit when almost everything my exploit could ever need is whitelisted, including hundreds of syscalls that directly talk to the kernel? I'm glad QEMU is improving their security footprint, but I honestly think its seccomp support is way overblown even though people like to bring it up... When your seccomp policy looks like the one from QEMU -- it just bans a few things they've never used, rather than only allowing things they should use. Taking advantage of it to the level of something like Chrome does (which truly mitigates attack surface with process isolation) would require almost an entire redesign of the whole thing, I think. [1] https://github.com/qemu/qemu/blob/master/qemu-seccomp.c https://github.com/qemu/qemu/blob/master/qemu-seccomp.c
    - rwmj 9y ago
      Sure, I agree. It's the SELinux policy which really confines QEMU. For example the compromised QEMU will only be able to open exactly the files containing the guest's drives (not even the drives of other guests on the same host).
      - secur101 9y ago
        @rwmj can you please point to the list of white-listed QEMU-KVM devices used in RHEL?
        rwmj 9y ago
        The qemu source RPMs are now hosted here: https://git.centos.org/project/rpms https://git.centos.org/project/rpms You have to type "qemu" in the box at top right. The qemu-kvm-rhev source RPM is here: http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/ http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en...
        secur101 9y ago
        Thanks for your reply. I made some suggestions for KVM and QEMU below. Also I was wondering what your POV is on my thoughts: https://news.ycombinator.com/item?id=14227605 https://news.ycombinator.com/item?id=14227605 https://news.ycombinator.com/item?id=14228563 https://news.ycombinator.com/item?id=14228563
- bonzini 9y ago
  Pretty much all cloud providers that use KVM (where "pretty much all" probably means all except Google) are using QEMU and potentially can be running hostile code, so the "not recommended" remark is perhaps a bit exaggerated.
  - 0xFFC 9y ago
    What do they use at Google?
    - rwmj 9y ago
      Some proprietary code replacing qemu (but still using KVM). Since they still have to emulate PC devices, this just means they have a different set of security holes and fewer people reviewing the code.
      - sitkack 9y ago
        Given QEMUs track record, I'd wager that the goog code gets more reviewers and more testing. QEMU is literally swiss cheese, or it emulates real swiss cheese, poorly.
        bonzini 9y ago
        Any proof of what you are saying, or is it just FUD? Most QEMU CVEs are related to devices that should never be used in cloud provider scenarios (you'll often find that they are disabled in RHEL for this exact reason). If anything, prompt handling of vulnerabilities in those devices is a sign of taking security seriously...
        j_s 9y ago
        Not sure about OP; it is more of an appeal to authority here: https://news.ycombinator.com/item?id=13484146 https://news.ycombinator.com/item?id=13484146 No doubt in my mind Google has the top-tier working on this, at least now that GCE is public-facing. I was impressed to read they actively monitor/mitigate Rowhammer, something I've not seen mentioned anywhere else (could just be my ignorance).
throw2016 9y ago
I think there is too much cruft in Qemu and a simple install can pull in hundreds of dependencies. For instance if you just want to run x86_64 VMs there is no point getting 20 different platforms. For the standard run of the mill VM there is also rarely any need for the multitude of devices. The more stuff you put in the more surface area for exploits. The Qemu install candidate needs to be seriously redesigned and pruned down by package maintainers. Intel Clear Linux tried running VMs without qemu altogether with Kvmtool but are now using 'Qemu lite'. But there are no packages, instructions or even a readme on what exactly they have done to make 'Qemu lite'. Kvmtool is one of those packages with zero documentation. Libvirt is needlessly convoluted and its usage complexity has led to an inferior solution like Virtualbox gaining traction on Linux, inspite of kvm in the kernel. For those looking for a simpler way to use kvm I suggest trying something like Aqemu or even just resorting to the command line. I found it far more lightweight and reasonable than libvirt's heavy handed way of running a simple qemu-kvm command line and adding an XML mess to it.
- bonzini 9y ago
  QEMU-lite is just a set of patches on top of QEMU, which are in the process of being merged.
- kashyapc 9y ago
  Please don't make suggestions like "libvirt is too convoluted, just resort to [QEMU] command-line" -- FWIW, I'm saying this as an every day QEMU command-line user -- without having the complete picture. Yes, there are some valid scenarios where people who know what they're doing can (and they do) directly use the QEMU command-line. But for the most majority, libvirt makes life far simpler, because of the following. Daniel Berrangé (lead maintainer of libvirt) articulates it much more succintly than I could, allow me to quote him: "It is a common attitude among people who look at QEMU and see a deceptively simple command-line syntax for launching a VM and don't realize that you'll enter a world of hurt when you go beyond the initial simple command-line syntax. This[1] was written in 2011, but is pretty valid and probably more besides: https://www.berrange.com/posts/2011/06/07/what-benefits-does-libvirt-offer-to-developers-targetting-qemukvm/ Recently, we had a very nice example of benefit of libvirt to security when the VENOM bug came out. Anyone using libvirt automatically had anti-VENOM available thanks to SELinux/AppArmour which would prevent exploitation in most cases." More on it (from his very detailed response on this thread[+], specifically from the 4th paragraph on): "Also note that this kind of bug [VENOM] in QEMU device emulation is the poster child example for the benefit of having sVirt (either SELinux or AppArmor backends) enabled on your compute hosts.. With sVirt, QEMU is restricted to only access resources that have been explicitly assigned to it. This makes it very difficult (likely/hopefully impossible[1]) for a compromised QEMU to be used to break out to compromise the host as a whole, likewise protect against compromising other QEMU processes on the same host. The common Linux distros like RHEL, Fedora, Debian, Ubuntu, etc all have sVirt feature available and enabled by default and OpenStack doesn't do anything to prevent it from working. Hopefully no one is actively disabling it themselves leaving themselves open to attack... [1] I'll never claim anything is 100% foolproof, but it is intended to be impossible to escape sVirt, so any such viable escape routes would themselves be considered security bugs. " [+] http://lists.openstack.org/pipermail/openstack-operators/2015-May/006947.html http://lists.openstack.org/pipermail/openstack-operators/201...
legulere 9y ago
With all the bugs in hardware emulation, wouldn't it make sense to emulate the linux kernel a la bash for windows instead of running the linux kernel on emulated hardware?
- btbuilder 9y ago
  Sounds like you are describing containers; while not emulation neither is virtualization. There are many more opportunities for escape dealing with Linux containers than virtualization due to the increased complexity of the interface. While I'm impressed by the work Microsoft have done to support the Linux kernel interfaces I would imagine the complexity of the effort to implement correct behavior from Windows kernel primitives would lead to more potential security vulnerabilities. Another comparison might be Linux syscall support within illumos[1] which AFAIK relies on mature Solaris Zones for isolation. [1] https://www.slideshare.net/bcantrill/illumos-lx https://www.slideshare.net/bcantrill/illumos-lx
  - danieldk 9y ago
    Sounds like you are describing containers; while not emulation neither is virtualization. Another possibility would be User-mode Linux (UML), in contrast to containers, it gives each 'virtual machine' its own Linux kernel, where the Linux kernel runs as another Linux program.
- pmiller2 9y ago
  Not always. Suppose you are writing a stand-alone kernel, for instance. That's much easier to debug in a VM than on real hardware.
overgryphon 9y ago
This is a case of legacy code left in an important attack surface. I doubt many people need a virtual floppy drive today.
- 0x0 9y ago
  Nonsense. I use the virtual floppy drive in VMs all the time, because I'm virtualizing legacy systems. But I do agree it could probably be disabled (and thus unexploitable) by default.
- ec109685 9y ago
  This is a different bug.

100 points by 0xFFC 9y ago

j_s 9y ago

If for some reason using QEMU with potentially hostile code (not recommended!), disable all the hardware emulation you can!

rwmj 9y ago
You should keep qemu up to date of course, and remove devices you don't need. But more importantly ensure you are using it through libvirt. Libvirt adds an SELinux or AppArmor layer, and a container, and enables seccomp for the qemu process. So even if the qemu process is compromised, it is very limited in what it can do to the host.
- sdeziel 9y ago
  Unfortunately on Ubuntu, seccomp is opt-in only. Works well when enabled though.
- aseipp 9y ago
  FWIW I've never really been impressed with QEMU's seccomp sandboxing, because it's almost entirely bolted on, and in the grand scheme, I think, does little. Does libvirt do anything substantially different or just "turn it on"? If you look at the list of syscalls whitelisted[1], it's almost a joke -- you could literally do just about anything given those syscalls if you actually had code execution... And every one of those syscalls is a direct interface to the kernel, which becomes the weakest link in a container/LSM environment where your code will be running, presumably. How is seccomp going to stop my exploit when almost everything my exploit could ever need is whitelisted, including hundreds of syscalls that directly talk to the kernel? I'm glad QEMU is improving their security footprint, but I honestly think its seccomp support is way overblown even though people like to bring it up... When your seccomp policy looks like the one from QEMU -- it just bans a few things they've never used, rather than only allowing things they should use. Taking advantage of it to the level of something like Chrome does (which truly mitigates attack surface with process isolation) would require almost an entire redesign of the whole thing, I think. [1] https://github.com/qemu/qemu/blob/master/qemu-seccomp.c https://github.com/qemu/qemu/blob/master/qemu-seccomp.c
  - rwmj 9y ago
    Sure, I agree. It's the SELinux policy which really confines QEMU. For example the compromised QEMU will only be able to open exactly the files containing the guest's drives (not even the drives of other guests on the same host).
    - secur101 9y ago
      @rwmj can you please point to the list of white-listed QEMU-KVM devices used in RHEL?
      - rwmj 9y ago
        The qemu source RPMs are now hosted here: https://git.centos.org/project/rpms https://git.centos.org/project/rpms You have to type "qemu" in the box at top right. The qemu-kvm-rhev source RPM is here: http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/ http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en...
        secur101 9y ago
        Thanks for your reply. I made some suggestions for KVM and QEMU below. Also I was wondering what your POV is on my thoughts: https://news.ycombinator.com/item?id=14227605 https://news.ycombinator.com/item?id=14227605 https://news.ycombinator.com/item?id=14228563 https://news.ycombinator.com/item?id=14228563
bonzini 9y ago
Pretty much all cloud providers that use KVM (where "pretty much all" probably means all except Google) are using QEMU and potentially can be running hostile code, so the "not recommended" remark is perhaps a bit exaggerated.
- 0xFFC 9y ago
  What do they use at Google?
  - rwmj 9y ago
    Some proprietary code replacing qemu (but still using KVM). Since they still have to emulate PC devices, this just means they have a different set of security holes and fewer people reviewing the code.
    - sitkack 9y ago
      Given QEMUs track record, I'd wager that the goog code gets more reviewers and more testing. QEMU is literally swiss cheese, or it emulates real swiss cheese, poorly.
      - bonzini 9y ago
        Any proof of what you are saying, or is it just FUD? Most QEMU CVEs are related to devices that should never be used in cloud provider scenarios (you'll often find that they are disabled in RHEL for this exact reason). If anything, prompt handling of vulnerabilities in those devices is a sign of taking security seriously...
        j_s 9y ago
        Not sure about OP; it is more of an appeal to authority here: https://news.ycombinator.com/item?id=13484146 https://news.ycombinator.com/item?id=13484146 No doubt in my mind Google has the top-tier working on this, at least now that GCE is public-facing. I was impressed to read they actively monitor/mitigate Rowhammer, something I've not seen mentioned anywhere else (could just be my ignorance).

throw2016 9y ago

I think there is too much cruft in Qemu and a simple install can pull in hundreds of dependencies. For instance if you just want to run x86_64 VMs there is no point getting 20 different platforms. For the standard run of the mill VM there is also rarely any need for the multitude of devices. The more stuff you put in the more surface area for exploits. The Qemu install candidate needs to be seriously redesigned and pruned down by package maintainers. Intel Clear Linux tried running VMs without qemu altogether with Kvmtool but are now using 'Qemu lite'. But there are no packages, instructions or even a readme on what exactly they have done to make 'Qemu lite'. Kvmtool is one of those packages with zero documentation. Libvirt is needlessly convoluted and its usage complexity has led to an inferior solution like Virtualbox gaining traction on Linux, inspite of kvm in the kernel. For those looking for a simpler way to use kvm I suggest trying something like Aqemu or even just resorting to the command line. I found it far more lightweight and reasonable than libvirt's heavy handed way of running a simple qemu-kvm command line and adding an XML mess to it.

bonzini 9y ago
QEMU-lite is just a set of patches on top of QEMU, which are in the process of being merged.
kashyapc 9y ago
Please don't make suggestions like "libvirt is too convoluted, just resort to [QEMU] command-line" -- FWIW, I'm saying this as an every day QEMU command-line user -- without having the complete picture. Yes, there are some valid scenarios where people who know what they're doing can (and they do) directly use the QEMU command-line. But for the most majority, libvirt makes life far simpler, because of the following. Daniel Berrangé (lead maintainer of libvirt) articulates it much more succintly than I could, allow me to quote him: "It is a common attitude among people who look at QEMU and see a deceptively simple command-line syntax for launching a VM and don't realize that you'll enter a world of hurt when you go beyond the initial simple command-line syntax. This[1] was written in 2011, but is pretty valid and probably more besides: https://www.berrange.com/posts/2011/06/07/what-benefits-does-libvirt-offer-to-developers-targetting-qemukvm/ Recently, we had a very nice example of benefit of libvirt to security when the VENOM bug came out. Anyone using libvirt automatically had anti-VENOM available thanks to SELinux/AppArmour which would prevent exploitation in most cases." More on it (from his very detailed response on this thread[+], specifically from the 4th paragraph on): "Also note that this kind of bug [VENOM] in QEMU device emulation is the poster child example for the benefit of having sVirt (either SELinux or AppArmor backends) enabled on your compute hosts.. With sVirt, QEMU is restricted to only access resources that have been explicitly assigned to it. This makes it very difficult (likely/hopefully impossible[1]) for a compromised QEMU to be used to break out to compromise the host as a whole, likewise protect against compromising other QEMU processes on the same host. The common Linux distros like RHEL, Fedora, Debian, Ubuntu, etc all have sVirt feature available and enabled by default and OpenStack doesn't do anything to prevent it from working. Hopefully no one is actively disabling it themselves leaving themselves open to attack... [1] I'll never claim anything is 100% foolproof, but it is intended to be impossible to escape sVirt, so any such viable escape routes would themselves be considered security bugs. " [+] http://lists.openstack.org/pipermail/openstack-operators/2015-May/006947.html http://lists.openstack.org/pipermail/openstack-operators/201...

legulere 9y ago

With all the bugs in hardware emulation, wouldn't it make sense to emulate the linux kernel a la bash for windows instead of running the linux kernel on emulated hardware?

btbuilder 9y ago
Sounds like you are describing containers; while not emulation neither is virtualization. There are many more opportunities for escape dealing with Linux containers than virtualization due to the increased complexity of the interface. While I'm impressed by the work Microsoft have done to support the Linux kernel interfaces I would imagine the complexity of the effort to implement correct behavior from Windows kernel primitives would lead to more potential security vulnerabilities. Another comparison might be Linux syscall support within illumos[1] which AFAIK relies on mature Solaris Zones for isolation. [1] https://www.slideshare.net/bcantrill/illumos-lx https://www.slideshare.net/bcantrill/illumos-lx
- danieldk 9y ago
  Sounds like you are describing containers; while not emulation neither is virtualization. Another possibility would be User-mode Linux (UML), in contrast to containers, it gives each 'virtual machine' its own Linux kernel, where the Linux kernel runs as another Linux program.
pmiller2 9y ago
Not always. Suppose you are writing a stand-alone kernel, for instance. That's much easier to debug in a VM than on real hardware.

overgryphon 9y ago

This is a case of legacy code left in an important attack surface. I doubt many people need a virtual floppy drive today.

0x0 9y ago
Nonsense. I use the virtual floppy drive in VMs all the time, because I'm virtualizing legacy systems. But I do agree it could probably be disabled (and thus unexploitable) by default.
ec109685 9y ago
This is a different bug.