5 ms·

Pretty much all cloud providers that use KVM (where "pretty much all" probably means all except Google) are using QEMU and potentially can be running hostile co

by bonzini 10y ago

Pretty much all cloud providers that use KVM (where "pretty much all" probably means all except Google) are using QEMU and potentially can be running hostile code, so the "not recommended" remark is perhaps a bit exaggerated.

0xFFC 10y ago
What do they use at Google?
- rwmj 10y ago
  Some proprietary code replacing qemu (but still using KVM). Since they still have to emulate PC devices, this just means they have a different set of security holes and fewer people reviewing the code.
  - sitkack 10y ago
    Given QEMUs track record, I'd wager that the goog code gets more reviewers and more testing. QEMU is literally swiss cheese, or it emulates real swiss cheese, poorly.
    - bonzini 10y ago
      Any proof of what you are saying, or is it just FUD? Most QEMU CVEs are related to devices that should never be used in cloud provider scenarios (you'll often find that they are disabled in RHEL for this exact reason). If anything, prompt handling of vulnerabilities in those devices is a sign of taking security seriously...
      - j_s 10y ago
        Not sure about OP; it is more of an appeal to authority here: https://news.ycombinator.com/item?id=13484146 https://news.ycombinator.com/item?id=13484146 No doubt in my mind Google has the top-tier working on this, at least now that GCE is public-facing. I was impressed to read they actively monitor/mitigate Rowhammer, something I've not seen mentioned anywhere else (could just be my ignorance).
      - peterwwillis 10y ago
        QEMU is a 14 year old codebase designed for research and cross-platform emulation. It is not developed with a focus toward ongoing security testing/auditing. Xen uses a stripped-down QEMU to boot unpatched guest OSes. However, even Xen doesn't test its qemu-xen components extensively. Writing a new purpose-built emulator (assuming you know what you're doing) is a better idea. edit: Or use PV guests, and skip all potential QEMU flaws.
        rwmj 10y ago
        Sorry, this is completely wrong FUD. Paolo Bonzini and I work at Red Hat and there is constant review of the QEMU codebase, both for code quality and security issues. Furthermore devices in RHEL are whitelisted. TCG ("cross-platform emulation") is not involved at all when you're using KVM. A newly written emulator would just have a new set of vulnerabilities. When you don't know what you're talking about, please just don't.
        peterwwillis 10y ago
        I'm glad to hear that! When did you implement fuzzing coverage? After the 2016 Qemu mailing list article I read where they're asking someone to look into applying to Google's oss fuzzing project? And how does your whitelisted set compare to Xen's stripped-down qemu support?
        rwmj 10y ago
        I'm actually fuzzing qemu's block device layer as we speak. From the latest run: │ run time : 11 days, 23 hrs, 12 min, 49 sec │ cycles done : 0 │ │ last new path : 0 days, 12 hrs, 55 min, 7 sec │ total paths : 364 │ │ last uniq crash : none seen yet │ uniq crashes : 0 │ │ last uniq hang : 0 days, 4 hrs, 4 min, 36 sec │ uniq hangs : 2 │ Re the comparison with Xen's qemu, you can grab the sources for RHEL's qemu-kvm and qemu-kvm-rhev packages and examine the driver whitelists, patches and ./configure line yourself.
        bonzini 10y ago
- sydney6 10y ago
  Ways we harden our KVM hypervisor at Google Cloud: Security in plaintext: https://news.ycombinator.com/item?id=13483603 https://news.ycombinator.com/item?id=13483603
j_s 10y ago
Thanks for the tip! Too late to edit, bummer. [Edit:If you're right I] would have switched it to: "not for the faint of heart!" My understanding was from Qubes choosing Xen and also AWS (they both deal with Xen advisories instead). The Qubes Architecture Specification goes into detail starting on page 11: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf https://www.qubes-os.org/attachment/wiki/QubesArchitecture/a... KVM uses the open source qemu emulator for [I/O emulattion]. [...] The I/O emulator is a complex piece of software, and thus it is reasonable to assume that it contains bugs and that it can be exploited by the attacker. In fact both Xen and KVM assume that the I/O emulator can be compromised and they both try to protect the rest of the system from a potentially compromised I/O emulator. Also pointed out elsewhere on thread: Google skipping QEMU. Edit: I am digging into it more, but I don't see KVM+QEMU on any top-tier provider (GCE, AWS, Azure [Hyper-V-ish])? My understanding was the only time QEMU was required was to emulate processor architectures, eg. x86 on ARM or vice-versa. QEMU is also used by some reverse-engineering/anti-malware emulation stuff.
- bonzini 10y ago
  In China KVM is probably more common than Xen; Aliyun and Tencent use it, and Huawei is transitioning from Xen to KVM (with QEMU). KVM is probably more common among whoever uses OpenStack. Xen also requires a hardware emulator to run HVM guests (including, but not limited to, Windows VMs). I don't know about now, but it definitely used to be QEMU for AWS. QEMU can do emulation, but with KVM you use the hypervisor to run code at full speed until it has to interact with the emulated hardware.
  - j_s 10y ago
    Thanks for the follow-up! I'll settle for "no tier-1 US provider publicly admits to using QEMU". :) I didn't dig too far into the AWS vulnerability list to try to find QEMU; XEN shows up right away! Ok: QEMU is last mentioned July 2015, and in none of the mentions is AWS vulnerable. https://www.google.com/?q=site:https://aws.amazon.com/security/security-bulletins/+qemu https://www.google.com/?q=site:https://aws.amazon.com/securi...
secur101 10y ago
My logic is: Without a hardened kernel, LSM can be trivially bypassed and seccomp seems to whitelist everything under the sun. This only leaves us with QEMU code quality to rely on. Since Grsec is not longer available this becomes even more urgent. Xen relies on stubdoms to isolate QEMU from their TCB which leaves them with bugs in the hypervisor itself as the only avenue of attack. The number of Xen-only bugs vs Linux is way fewer. Please correct me if I'm wrong. @bonzini I use virtio-9p for shared folders all the time why did you dismiss that as a non-issue: https://news.ycombinator.com/item?id=13755021 https://news.ycombinator.com/item?id=13755021 If you are a KVM dev please look seriously into using an advanced, intelligent fuzzer like the DARPA Grand Challenge winner Shellphish. It can find security bugs and propose patches for them: https://github.com/shellphish https://github.com/shellphish http://angr.io/ http://angr.io/ * Security aside I find Libvirt more wanting in UX. The single biggest roadblock is the lack of a virtual appliance implementation that new comers can point and import to from Virt-Manager. I hope this gets resolved down the line.
secur101 10y ago
I'm still rooting for KVM because it has the best hardware compatibility, it in mainline and its performance is the best. However the security situation with QEMU is not as rosy as some of you portray it (and its not as bad as it looks either). The average customer/user is going to look at the number of QEMU CVEs on cvedetails and compare them to the number of Xen's and go with the latter. However most of the QEMU bugs only affect legacy emulated hardware or components not enabled in a KVM guest, most won't think that far. It would clear things up if you have a table on your site showing which QEMU vulnerabilties affect a specified default configuration of a RHEL/Debian guest out of the box in libvirt. See this for example: https://www.qubes-os.org/security/xsa/ https://www.qubes-os.org/security/xsa/ What I want to see: * Adoption of QEMU-lite as the default mode for Linux guests. There's no point to running Linux in almost any emulated hardware. * A builtin monitoring solution like Google has that detects excessive DRAM bitflips [1] and cache misses [2] and terminates the guests to foil rowhammer and covert channel attacks. * A re-design of KSM thats not prone to rowhammer abuse [3] * [1] https://cloudplatform.googleblog.com/2017/01/7-ways-we-harden-our-KVM-hypervisor-at-Google-Cloud-security-in-plaintext.html https://cloudplatform.googleblog.com/2017/01/7-ways-we-harde... [2] https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_pessl.pdf https://www.usenix.org/system/files/conference/usenixsecurit... [3] http://www.cs.vu.nl/~kaveh/pubs/pdf/ffs-usenixsec16.pdf http://www.cs.vu.nl/~kaveh/pubs/pdf/ffs-usenixsec16.pdf
- bonzini 10y ago
  * I am not aware of any attacks against legacy hardware except for VENOM. Intel's QEMU-lite patches are disabling these devices for speed rather than security reasons. In any case, no external patches are needed right now to disable most legacy devices: QEMU's Q35 machine type doesn't have a default floppy controller and you can already remove the HPET, PIT, SATA controller and SMBIOS controller. What is left is used, albeit sometimes rarely, by the firmware or the OS (e.g. IOAPIC, RTC, PCI host bridge or ACPI); any replacement would be more likely to have holes than the current well-tested code. * Rowhammer detection is interesting, but not really related to virtualization. Thanks to KVM's design any such monitoring solution would apply equally to Linux containers. This is not the case for Xen, for example. * Besides Rowhammer, memory dedup is highly subject to side channel attacks. I think this is a much more important issue, and it already pretty much forces you to disable KSM in multi-tenant applications.

by bonzini 10y ago

0xFFC 10y ago

What do they use at Google?

rwmj 10y ago
Some proprietary code replacing qemu (but still using KVM). Since they still have to emulate PC devices, this just means they have a different set of security holes and fewer people reviewing the code.
- sitkack 10y ago
  Given QEMUs track record, I'd wager that the goog code gets more reviewers and more testing. QEMU is literally swiss cheese, or it emulates real swiss cheese, poorly.
  - bonzini 10y ago
    Any proof of what you are saying, or is it just FUD? Most QEMU CVEs are related to devices that should never be used in cloud provider scenarios (you'll often find that they are disabled in RHEL for this exact reason). If anything, prompt handling of vulnerabilities in those devices is a sign of taking security seriously...
    - j_s 10y ago
      Not sure about OP; it is more of an appeal to authority here: https://news.ycombinator.com/item?id=13484146 https://news.ycombinator.com/item?id=13484146 No doubt in my mind Google has the top-tier working on this, at least now that GCE is public-facing. I was impressed to read they actively monitor/mitigate Rowhammer, something I've not seen mentioned anywhere else (could just be my ignorance).
    - peterwwillis 10y ago
      QEMU is a 14 year old codebase designed for research and cross-platform emulation. It is not developed with a focus toward ongoing security testing/auditing. Xen uses a stripped-down QEMU to boot unpatched guest OSes. However, even Xen doesn't test its qemu-xen components extensively. Writing a new purpose-built emulator (assuming you know what you're doing) is a better idea. edit: Or use PV guests, and skip all potential QEMU flaws.
      - rwmj 10y ago
        Sorry, this is completely wrong FUD. Paolo Bonzini and I work at Red Hat and there is constant review of the QEMU codebase, both for code quality and security issues. Furthermore devices in RHEL are whitelisted. TCG ("cross-platform emulation") is not involved at all when you're using KVM. A newly written emulator would just have a new set of vulnerabilities. When you don't know what you're talking about, please just don't.
        peterwwillis 10y ago
        I'm glad to hear that! When did you implement fuzzing coverage? After the 2016 Qemu mailing list article I read where they're asking someone to look into applying to Google's oss fuzzing project? And how does your whitelisted set compare to Xen's stripped-down qemu support?
        rwmj 10y ago
        I'm actually fuzzing qemu's block device layer as we speak. From the latest run: │ run time : 11 days, 23 hrs, 12 min, 49 sec │ cycles done : 0 │ │ last new path : 0 days, 12 hrs, 55 min, 7 sec │ total paths : 364 │ │ last uniq crash : none seen yet │ uniq crashes : 0 │ │ last uniq hang : 0 days, 4 hrs, 4 min, 36 sec │ uniq hangs : 2 │ Re the comparison with Xen's qemu, you can grab the sources for RHEL's qemu-kvm and qemu-kvm-rhev packages and examine the driver whitelists, patches and ./configure line yourself.
      - bonzini 10y ago
sydney6 10y ago
Ways we harden our KVM hypervisor at Google Cloud: Security in plaintext: https://news.ycombinator.com/item?id=13483603 https://news.ycombinator.com/item?id=13483603

j_s 10y ago

Thanks for the tip! Too late to edit, bummer. [Edit:If you're right I] would have switched it to: "not for the faint of heart!" My understanding was from Qubes choosing Xen and also AWS (they both deal with Xen advisories instead). The Qubes Architecture Specification goes into detail starting on page 11: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf https://www.qubes-os.org/attachment/wiki/QubesArchitecture/a... KVM uses the open source qemu emulator for [I/O emulattion]. [...] The I/O emulator is a complex piece of software, and thus it is reasonable to assume that it contains bugs and that it can be exploited by the attacker. In fact both Xen and KVM assume that the I/O emulator can be compromised and they both try to protect the rest of the system from a potentially compromised I/O emulator. Also pointed out elsewhere on thread: Google skipping QEMU. Edit: I am digging into it more, but I don't see KVM+QEMU on any top-tier provider (GCE, AWS, Azure [Hyper-V-ish])? My understanding was the only time QEMU was required was to emulate processor architectures, eg. x86 on ARM or vice-versa. QEMU is also used by some reverse-engineering/anti-malware emulation stuff.

bonzini 10y ago
In China KVM is probably more common than Xen; Aliyun and Tencent use it, and Huawei is transitioning from Xen to KVM (with QEMU). KVM is probably more common among whoever uses OpenStack. Xen also requires a hardware emulator to run HVM guests (including, but not limited to, Windows VMs). I don't know about now, but it definitely used to be QEMU for AWS. QEMU can do emulation, but with KVM you use the hypervisor to run code at full speed until it has to interact with the emulated hardware.
- j_s 10y ago
  Thanks for the follow-up! I'll settle for "no tier-1 US provider publicly admits to using QEMU". :) I didn't dig too far into the AWS vulnerability list to try to find QEMU; XEN shows up right away! Ok: QEMU is last mentioned July 2015, and in none of the mentions is AWS vulnerable. https://www.google.com/?q=site:https://aws.amazon.com/security/security-bulletins/+qemu https://www.google.com/?q=site:https://aws.amazon.com/securi...

secur101 10y ago

My logic is: Without a hardened kernel, LSM can be trivially bypassed and seccomp seems to whitelist everything under the sun. This only leaves us with QEMU code quality to rely on. Since Grsec is not longer available this becomes even more urgent. Xen relies on stubdoms to isolate QEMU from their TCB which leaves them with bugs in the hypervisor itself as the only avenue of attack. The number of Xen-only bugs vs Linux is way fewer. Please correct me if I'm wrong. @bonzini I use virtio-9p for shared folders all the time why did you dismiss that as a non-issue: https://news.ycombinator.com/item?id=13755021 https://news.ycombinator.com/item?id=13755021 If you are a KVM dev please look seriously into using an advanced, intelligent fuzzer like the DARPA Grand Challenge winner Shellphish. It can find security bugs and propose patches for them: https://github.com/shellphish https://github.com/shellphish http://angr.io/ http://angr.io/ * Security aside I find Libvirt more wanting in UX. The single biggest roadblock is the lack of a virtual appliance implementation that new comers can point and import to from Virt-Manager. I hope this gets resolved down the line.

secur101 10y ago

I'm still rooting for KVM because it has the best hardware compatibility, it in mainline and its performance is the best. However the security situation with QEMU is not as rosy as some of you portray it (and its not as bad as it looks either). The average customer/user is going to look at the number of QEMU CVEs on cvedetails and compare them to the number of Xen's and go with the latter. However most of the QEMU bugs only affect legacy emulated hardware or components not enabled in a KVM guest, most won't think that far. It would clear things up if you have a table on your site showing which QEMU vulnerabilties affect a specified default configuration of a RHEL/Debian guest out of the box in libvirt. See this for example: https://www.qubes-os.org/security/xsa/ https://www.qubes-os.org/security/xsa/ What I want to see: * Adoption of QEMU-lite as the default mode for Linux guests. There's no point to running Linux in almost any emulated hardware. * A builtin monitoring solution like Google has that detects excessive DRAM bitflips [1] and cache misses [2] and terminates the guests to foil rowhammer and covert channel attacks. * A re-design of KSM thats not prone to rowhammer abuse [3] * [1] https://cloudplatform.googleblog.com/2017/01/7-ways-we-harden-our-KVM-hypervisor-at-Google-Cloud-security-in-plaintext.html https://cloudplatform.googleblog.com/2017/01/7-ways-we-harde... [2] https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_pessl.pdf https://www.usenix.org/system/files/conference/usenixsecurit... [3] http://www.cs.vu.nl/~kaveh/pubs/pdf/ffs-usenixsec16.pdf http://www.cs.vu.nl/~kaveh/pubs/pdf/ffs-usenixsec16.pdf

bonzini 10y ago
* I am not aware of any attacks against legacy hardware except for VENOM. Intel's QEMU-lite patches are disabling these devices for speed rather than security reasons. In any case, no external patches are needed right now to disable most legacy devices: QEMU's Q35 machine type doesn't have a default floppy controller and you can already remove the HPET, PIT, SATA controller and SMBIOS controller. What is left is used, albeit sometimes rarely, by the firmware or the OS (e.g. IOAPIC, RTC, PCI host bridge or ACPI); any replacement would be more likely to have holes than the current well-tested code. * Rowhammer detection is interesting, but not really related to virtualization. Thanks to KVM's design any such monitoring solution would apply equally to Linux containers. This is not the case for Xen, for example. * Besides Rowhammer, memory dedup is highly subject to side channel attacks. I think this is a much more important issue, and it already pretty much forces you to disable KSM in multi-tenant applications.