11 ms·

Ways we harden our KVM hypervisor at Google Cloud: Security in plaintext

199 points by EvgeniyZh 10y ago

nealmueller 10y ago
The authors have noteworthy bona fides. Andy used to work for the NSA, and Nelly has 16 patents with names like "Informed implicit enrollment and identification". https://www.linkedin.com/in/andrew-honig-82239750 https://www.linkedin.com/in/andrew-honig-82239750 https://www.linkedin.com/in/nelly-porter-708a10b https://www.linkedin.com/in/nelly-porter-708a10b
- deleted 10y ago
  [deleted]
Skunkleton 10y ago
I see that google has open sourced a VMM written in go [1], but I doubt this is the VMM discussed in the article. Has google open sourced this software? [1] https://github.com/google/novm https://github.com/google/novm
- andyhonig 10y ago
  Google has not open sourced this for a variety of technical and business reasons. I'm not going to get into all of them, but for example a lot of our device emulation is built upon internal Google services. It wouldn't work outside a Google data center. It's not out of the realm of possibility that Google open sources some parts of it in the future, but I don't know if that will ever happen.
  - jsolson 10y ago
    Adding to Andy's point here -- if you consider the services that back GCE in terms of the features offered by our virtual networks, virtual storage devices, etc. they necessarily have tight dependencies on "How Google is Built" to hit the performance targets we aim for. I, personally, am optimistic that we'll be able to do it for the core at some point. Something not mentioned in the original article is that architecturally our VMM allows us to easily change the set of available functionality (virtual devices, backing implementations of those, etc.). So, for example, the fuzzing Andy described as depending on our custom VMM isn't even linked into the VMM run in GCE, nor are custom devices we keep around for testing and development. I'd very much like it if we could layer the VMM in such a way that we could release a useful core that forms the basis of the VMM we consume internally. Today, though, the layering just isn't quite there (among other technical and business reasons, as Andy says), and there are Googley specifics in places that would make even the VMM core (minus devices, etc.) unviable outside of Google.
madez 10y ago
> Alerts are generated if any projects that cause an unusual number of Rowhammer errors. Is that correct english?
- mcinjosh 10y ago
  Almost; I think it parses correctly by substituting 'if' with 'for'.
  - nealmueller 10y ago
    Thanks for catching. Fixed.
    - deleted 10y ago
      [deleted]
  - deleted 10y ago
    [deleted]
koolba 10y ago
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual machine monitor that has the following security advantages over QEMU: Simple host and guest architecture support matrix. QEMU supports a large matrix of host and guest architectures, along with different modes and devices that significantly increase complexity. I've noticed that a lot of projects that do support multiple architectures, particularly obscure ones, tend to find oddball edge cases more easily than those that don't. For example, not assuming the endianness of the CPU arch forces you to get network code to cooperate well. > Because we support a single architecture and a relatively small number of devices, our emulator is much simpler. No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!".
- rrdharan 10y ago
  > No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. A little later in the post I believe this is somewhat addressed: > QEMU code lacks unit tests and has many interdependencies that would make unit testing extremely difficult. Personally based on my previous experience at VMware and passing familiarity with QEMU, I think they made the right call. (I work at Google, but not on the hypervisor)
- luu 10y ago
  > I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. > Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!". Is it possible you're either underestimating the effort it takes to make QEMU solid or overestimating the effort it takes to write an emulator? I worked at a company where I hacked up QEMU as a stopgap before we switched to an in-house solution (this wasn't Google, although I've also worked at Google). I made literally hundreds of bug fixes to get QEMU into what was, for us, a barely usable state and then someone else wrote a solution from scratch in maybe a month and a half or two months. I doubt I could have gotten QEMU into the state we needed in a couple months. And to be clear, when I say bug fixes, I don't mean features or things that could possibly arguably be "working as intended", I mean bugs like "instruction X does the wrong thing instead of doing what an actual CPU does". BTW, I don't mean to knock QEMU. It's great for what it is, but it's often the case that a special purpose piece of software tailored for a specific usecase is less effort than making a very general framework suitable for the same usecase. Even for our usecase, where QEMU was a very bad fit, the existence of QEMU let us get an MVP up in a week; I applied critical fixes to our hacked up QEMU while someone worked on the real solution, which gave us a two month head start over just writing something from scratch. But the effort it would have taken to make QEMU production worthy for us didn't seem worth it.
dkarapetyan 10y ago
> We have built an extensive set of proprietary fuzzing tools for KVM. We also do a thorough code review looking specifically for security issues each time we adopt a new feature or version of KVM. As a result, we've found many vulnerabilities in KVM over the past three years. About half of our discoveries come from code review and about half come from our fuzzers. Why aren't these tools being open sourced?
- devoply 10y ago
  Maybe because they think it's a competitive advantage.
  - SEJeff 10y ago
    More likely it is just heavily integrated with internal google tooling like chubby and borg. I suspect they still use American Fuzzy Lop[1] for much of their fuzzing even if they wrote some of it from scratch. The Linux Foundation's Core Infrastructure Initiative's[2] "Fuzzing Project" is more or less continuous integration running AFL[3] [1] http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/ [2] https://www.coreinfrastructure.org/grants https://www.coreinfrastructure.org/grants [3] https://fuzzing-project.org/ https://fuzzing-project.org/
- cpeterso 10y ago
  Because the fuzzers can be used by blackhats to find vulnerabilities to be exploited. The blackhats surely write their own fuzzers, but why give them a head start?
  - SEJeff 10y ago
    Say it with me, "Security through obscurity is no security at all". Surely you'd know this working for Mozilla :) Probably due to business reasons in that it isn't meant to be a general purpose fuzzer and is likely tied to internal google infra.
    - felipemnoa 10y ago
fulafel 10y ago
This sounds much more secure than AWS and their use of Xen.
- QUFB 10y ago
  How so? What leads you to believe that Amazon hasn't taken similar measures?
  - j_s 10y ago
    https://aws.amazon.com/security/security-bulletins/ https://aws.amazon.com/security/security-bulletins/ search "Xen Security" and "XSA Security" Many do say AWS is not affected; it is a bit of effort to review them all. The newest I found is: https://aws.amazon.com/security/security-bulletins/XSAsecurityadvisories-December/ https://aws.amazon.com/security/security-bulletins/XSAsecuri... (Dec. 2015) Interestingly, AWS was not affected by the August 2016 XSA Security Advisory (XSA-182) that impacted Qubes OS: http://blog.quarkslab.com/xen-exploitation-part-3-xsa-182-qubes-escape.html http://blog.quarkslab.com/xen-exploitation-part-3-xsa-182-qu... https://aws.amazon.com/blogs/aws/ec2-maintenance-update/ https://aws.amazon.com/blogs/aws/ec2-maintenance-update/ (Sep. 2014)
  - numbsafari 10y ago
    Amazon isn't confident enough in their solution such that they will only support customers with high levels of regulatory and compliance requirements (e.g. HIPAA) if they are using dedicated physical hosts. https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/ https://aws.amazon.com/blogs/security/frequently-asked-quest...
    - nnx 10y ago
      Does Google Cloud support HIPAA workloads on regular shared instances? I thought HIPAA itself mandates physical dedicated host separation.
      - 10y ago
neom 10y ago
If there is a DigitalOcean engineer in the house, I'd be curious to know how this compares to how they run their KVM setup.
- QUFB 10y ago
  Given that it took 4 years of continuous customer requests just to get support for booting a custom kernel on Digital Ocean, I'd be surprised if they're doing anything this elaborate.
  - bitexploder 10y ago
    This is a great example of the advantages only a tiny number of technical companies have. It is great technology and great that Google can do this, but who has the resources to competently implement a hypervisor and customize it to their infrastructure?
    - moreorless 10y ago
      Vultr supported custom ISOs for a long long time now.
      - regularfry 10y ago
        Among several others. DO are going for the 80% case, not full spectrum capability.
  - floatboth 10y ago
    It's usually the elaborate, "cloudy" stuff that prevents booting custom OSes (mostly just networked storage, e.g. https://github.com/scaleway/image-proposals/issues/11 https://github.com/scaleway/image-proposals/issues/11 Scaleway would need a Network Block Device driver to run FreeBSD). Normal "VPS" providers often support this, e.g. prgmr just lets you boot into a recovery image (Linux, FreeBSD, NetBSD) and do whatever you want to the virtual hard drive, install anything that runs on Xen there. It's like running a Xen VM on your own computer, except it's someone else's computer. Interestingly, Digital Ocean uses local storage. I guess the issue was just with the boot process.
nellydpa 10y ago
Hey folks, thanks for great comments, keep them coming.I am one of the article authors, AMA.
andyhonig 10y ago
I'm one of the authors of this post, AMA.
- mjg59 10y ago
  Right now there doesn't seem to have been a lot of work in exposing security features to guests. Is there interest in supporting stuff like Secure Boot or any form of runtime attestation?
  - bonzini 10y ago
    QEMU + KVM + OVMF supports Secure Boot (including SMM emulation for the trusted base), or are you talking specifically of GCE?
    - mjg59 10y ago
      Yeah, specifically GCE
  - andyhonig 10y ago
    Thanks for the feedback, unfortunately I can't publicly comment on future product plans.
    - deegles 10y ago
      AM(almost)A :)
deleted 10y ago
[deleted]
Scaevolus 10y ago
Google Compute Engine has live migration, so host/hypervisor updates (including security updates!) can be applied without taking VMs offline. AWS doesn't support live migration. Azure has a partial solution with in-place migrations, which involves taking VMs down for ~30 seconds. (I work on Google Container Engine.)
- kim0 10y ago
  Can you explain the azure solution some more please, to the best of your knowledge
  - Scaevolus 10y ago
    This links to the announcement and has some speculation about how it's implemented. It looks like it might be basically a snapshot and a fast host reboot? https://www.petri.com/new-azure-iaas-features-announced-august-2016 https://www.petri.com/new-azure-iaas-features-announced-augu...
  - rdmsr 10y ago
    Probably means that upgrades are done in-place. They don't move the VM between machines, but they do something like move it from the old process to and upgraded version (or in some other way upgrade the software underneath).
- cperciva 10y ago
  Google Compute Engine has live migration, so host/hypervisor updates (including security updates!) can be applied without taking VMs offline. AWS doesn't support live migration. I don't work for AWS, so I can only speculate about their internal technology; but they've demonstrated an ability to apply security updates without requiring guests to reboot.
  - jvolkman 10y ago
    I believe in certain situations they've been able to use ksplice or similar mechanisms.
- 10y ago
ascotan 10y ago
Q. How do you secure your infrastructure Google? A. We hire thousands of programmers to write custom software. There are no known vulnerabilities. No. Known. Vulnerabilities.
- aiiane 10y ago
  That's not even an accurate representation of what is publicly claimed; Google runs a VRP specifically because it acknowledges that software sometimes has bugs and it's committed to fixing them. Google also employs quite a few of the security researchers that contribute to identifying and fixing security issues both within and beyond Google.
- stefanha 10y ago
  I don't work for Google but have seen KVM Forum presentations and mailing list discussions where they contributed back to KVM. They have world-class security researchers working on keeping KVM secure. Your comment implies they rely on security through obscurity and it misses the legitimate security work they are doing. Did you read anything at all before posting your comment? Start here: http://www.linux-kvm.org/images/f/f6/01x02-KVMHardening.pdf http://www.linux-kvm.org/images/f/f6/01x02-KVMHardening.pdf
shinyshorts 10y ago
Ok, sounds fancy enough... until the money dries up, the hydrocephalic Indian and Russian programmers have left and mere mortal engineers decide to move back to stuff that everyone else uses and understands. Also, am I the only one who freaks out by this seemingly harmless remark: "errors are detected and cause any potentially offending guest VMs to be terminated."? google messing about with AI and these, oh im sure really well intended, algorithms in place. Quite scary. They've also shown with their go bot that they will play covert games.. apply AI unexpectedly. Im not sure we can trust Google any longer.
throwaway7767 10y ago
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual machine monitor that has the following security advantages over QEMU: [...] No history of security problems. QEMU has a long track record of security bugs, such as VENOM, and it's unclear what vulnerabilities may still be lurking in the code. Has this alternative VMM/hardware emulator been released? As far as I can tell, the answer is no. In that light, it's more than a little weird to congratulate yourself on not having a "long track record of security bugs" in your internal-use-only unreleased tool compared to generally available software in wide use.
floatboth 10y ago
By the way, FreeBSD's vmm doesn't use QEMU, a fresh new userspace part (bhyve) was written. I think the same is happening with OpenBSD's vmm. It seems kinda weird that Linux KVM is commonly used with QEMU…
jron 10y ago
Does anyone have a better summary of the Xen vs KVM security debate? The best I know of is the brief Qubes architecture document from 2010: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf https://www.qubes-os.org/attachment/wiki/QubesArchitecture/a... (Section 3). KVM using Google's hardened device emulation may eliminate some of Qube's QEMU concerns. Qubes author critique on the state of Xen security: https://lists.xen.org/archives/html/xen-devel/2015-11/msg00601.html https://lists.xen.org/archives/html/xen-devel/2015-11/msg006...

199 points by EvgeniyZh 10y ago

The authors have noteworthy bona fides. Andy used to work for the NSA, and Nelly has 16 patents with names like "Informed implicit enrollment and identification". https://www.linkedin.com/in/andrew-honig-82239750 https://www.linkedin.com/in/andrew-honig-82239750 https://www.linkedin.com/in/nelly-porter-708a10b https://www.linkedin.com/in/nelly-porter-708a10b

deleted 10y ago
[deleted]

Skunkleton 10y ago

I see that google has open sourced a VMM written in go [1], but I doubt this is the VMM discussed in the article. Has google open sourced this software? [1] https://github.com/google/novm https://github.com/google/novm

andyhonig 10y ago
Google has not open sourced this for a variety of technical and business reasons. I'm not going to get into all of them, but for example a lot of our device emulation is built upon internal Google services. It wouldn't work outside a Google data center. It's not out of the realm of possibility that Google open sources some parts of it in the future, but I don't know if that will ever happen.
- jsolson 10y ago
  Adding to Andy's point here -- if you consider the services that back GCE in terms of the features offered by our virtual networks, virtual storage devices, etc. they necessarily have tight dependencies on "How Google is Built" to hit the performance targets we aim for. I, personally, am optimistic that we'll be able to do it for the core at some point. Something not mentioned in the original article is that architecturally our VMM allows us to easily change the set of available functionality (virtual devices, backing implementations of those, etc.). So, for example, the fuzzing Andy described as depending on our custom VMM isn't even linked into the VMM run in GCE, nor are custom devices we keep around for testing and development. I'd very much like it if we could layer the VMM in such a way that we could release a useful core that forms the basis of the VMM we consume internally. Today, though, the layering just isn't quite there (among other technical and business reasons, as Andy says), and there are Googley specifics in places that would make even the VMM core (minus devices, etc.) unviable outside of Google.

madez 10y ago

> Alerts are generated if any projects that cause an unusual number of Rowhammer errors. Is that correct english?

mcinjosh 10y ago
Almost; I think it parses correctly by substituting 'if' with 'for'.
- nealmueller 10y ago
  Thanks for catching. Fixed.
  - deleted 10y ago
    [deleted]
- deleted 10y ago
  [deleted]

koolba 10y ago

> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual machine monitor that has the following security advantages over QEMU: Simple host and guest architecture support matrix. QEMU supports a large matrix of host and guest architectures, along with different modes and devices that significantly increase complexity. I've noticed that a lot of projects that do support multiple architectures, particularly obscure ones, tend to find oddball edge cases more easily than those that don't. For example, not assuming the endianness of the CPU arch forces you to get network code to cooperate well. > Because we support a single architecture and a relatively small number of devices, our emulator is much simpler. No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!".

rrdharan 10y ago
> No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. A little later in the post I believe this is somewhat addressed: > QEMU code lacks unit tests and has many interdependencies that would make unit testing extremely difficult. Personally based on my previous experience at VMware and passing familiarity with QEMU, I think they made the right call. (I work at Google, but not on the hypervisor)
luu 10y ago
> I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. > Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!". Is it possible you're either underestimating the effort it takes to make QEMU solid or overestimating the effort it takes to write an emulator? I worked at a company where I hacked up QEMU as a stopgap before we switched to an in-house solution (this wasn't Google, although I've also worked at Google). I made literally hundreds of bug fixes to get QEMU into what was, for us, a barely usable state and then someone else wrote a solution from scratch in maybe a month and a half or two months. I doubt I could have gotten QEMU into the state we needed in a couple months. And to be clear, when I say bug fixes, I don't mean features or things that could possibly arguably be "working as intended", I mean bugs like "instruction X does the wrong thing instead of doing what an actual CPU does". BTW, I don't mean to knock QEMU. It's great for what it is, but it's often the case that a special purpose piece of software tailored for a specific usecase is less effort than making a very general framework suitable for the same usecase. Even for our usecase, where QEMU was a very bad fit, the existence of QEMU let us get an MVP up in a week; I applied critical fixes to our hacked up QEMU while someone worked on the real solution, which gave us a two month head start over just writing something from scratch. But the effort it would have taken to make QEMU production worthy for us didn't seem worth it.

dkarapetyan 10y ago

> We have built an extensive set of proprietary fuzzing tools for KVM. We also do a thorough code review looking specifically for security issues each time we adopt a new feature or version of KVM. As a result, we've found many vulnerabilities in KVM over the past three years. About half of our discoveries come from code review and about half come from our fuzzers. Why aren't these tools being open sourced?

devoply 10y ago
Maybe because they think it's a competitive advantage.
- SEJeff 10y ago
  More likely it is just heavily integrated with internal google tooling like chubby and borg. I suspect they still use American Fuzzy Lop[1] for much of their fuzzing even if they wrote some of it from scratch. The Linux Foundation's Core Infrastructure Initiative's[2] "Fuzzing Project" is more or less continuous integration running AFL[3] [1] http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/ [2] https://www.coreinfrastructure.org/grants https://www.coreinfrastructure.org/grants [3] https://fuzzing-project.org/ https://fuzzing-project.org/
cpeterso 10y ago
Because the fuzzers can be used by blackhats to find vulnerabilities to be exploited. The blackhats surely write their own fuzzers, but why give them a head start?
- SEJeff 10y ago
  Say it with me, "Security through obscurity is no security at all". Surely you'd know this working for Mozilla :) Probably due to business reasons in that it isn't meant to be a general purpose fuzzer and is likely tied to internal google infra.
  - felipemnoa 10y ago

fulafel 10y ago

This sounds much more secure than AWS and their use of Xen.

QUFB 10y ago
How so? What leads you to believe that Amazon hasn't taken similar measures?
- j_s 10y ago
  https://aws.amazon.com/security/security-bulletins/ https://aws.amazon.com/security/security-bulletins/ search "Xen Security" and "XSA Security" Many do say AWS is not affected; it is a bit of effort to review them all. The newest I found is: https://aws.amazon.com/security/security-bulletins/XSAsecurityadvisories-December/ https://aws.amazon.com/security/security-bulletins/XSAsecuri... (Dec. 2015) Interestingly, AWS was not affected by the August 2016 XSA Security Advisory (XSA-182) that impacted Qubes OS: http://blog.quarkslab.com/xen-exploitation-part-3-xsa-182-qubes-escape.html http://blog.quarkslab.com/xen-exploitation-part-3-xsa-182-qu... https://aws.amazon.com/blogs/aws/ec2-maintenance-update/ https://aws.amazon.com/blogs/aws/ec2-maintenance-update/ (Sep. 2014)
- numbsafari 10y ago
  Amazon isn't confident enough in their solution such that they will only support customers with high levels of regulatory and compliance requirements (e.g. HIPAA) if they are using dedicated physical hosts. https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/ https://aws.amazon.com/blogs/security/frequently-asked-quest...
  - nnx 10y ago
    Does Google Cloud support HIPAA workloads on regular shared instances? I thought HIPAA itself mandates physical dedicated host separation.
    - 10y ago

neom 10y ago

If there is a DigitalOcean engineer in the house, I'd be curious to know how this compares to how they run their KVM setup.

QUFB 10y ago
Given that it took 4 years of continuous customer requests just to get support for booting a custom kernel on Digital Ocean, I'd be surprised if they're doing anything this elaborate.
- bitexploder 10y ago
  This is a great example of the advantages only a tiny number of technical companies have. It is great technology and great that Google can do this, but who has the resources to competently implement a hypervisor and customize it to their infrastructure?
  - moreorless 10y ago
    Vultr supported custom ISOs for a long long time now.
    - regularfry 10y ago
      Among several others. DO are going for the 80% case, not full spectrum capability.
- floatboth 10y ago
  It's usually the elaborate, "cloudy" stuff that prevents booting custom OSes (mostly just networked storage, e.g. https://github.com/scaleway/image-proposals/issues/11 https://github.com/scaleway/image-proposals/issues/11 Scaleway would need a Network Block Device driver to run FreeBSD). Normal "VPS" providers often support this, e.g. prgmr just lets you boot into a recovery image (Linux, FreeBSD, NetBSD) and do whatever you want to the virtual hard drive, install anything that runs on Xen there. It's like running a Xen VM on your own computer, except it's someone else's computer. Interestingly, Digital Ocean uses local storage. I guess the issue was just with the boot process.

nellydpa 10y ago

Hey folks, thanks for great comments, keep them coming.I am one of the article authors, AMA.

andyhonig 10y ago

I'm one of the authors of this post, AMA.

mjg59 10y ago
Right now there doesn't seem to have been a lot of work in exposing security features to guests. Is there interest in supporting stuff like Secure Boot or any form of runtime attestation?
- bonzini 10y ago
  QEMU + KVM + OVMF supports Secure Boot (including SMM emulation for the trusted base), or are you talking specifically of GCE?
  - mjg59 10y ago
    Yeah, specifically GCE
- andyhonig 10y ago
  Thanks for the feedback, unfortunately I can't publicly comment on future product plans.
  - deegles 10y ago
    AM(almost)A :)

deleted 10y ago

[deleted]

Scaevolus 10y ago

Google Compute Engine has live migration, so host/hypervisor updates (including security updates!) can be applied without taking VMs offline. AWS doesn't support live migration. Azure has a partial solution with in-place migrations, which involves taking VMs down for ~30 seconds. (I work on Google Container Engine.)

kim0 10y ago
Can you explain the azure solution some more please, to the best of your knowledge
- Scaevolus 10y ago
  This links to the announcement and has some speculation about how it's implemented. It looks like it might be basically a snapshot and a fast host reboot? https://www.petri.com/new-azure-iaas-features-announced-august-2016 https://www.petri.com/new-azure-iaas-features-announced-augu...
- rdmsr 10y ago
  Probably means that upgrades are done in-place. They don't move the VM between machines, but they do something like move it from the old process to and upgraded version (or in some other way upgrade the software underneath).
cperciva 10y ago
Google Compute Engine has live migration, so host/hypervisor updates (including security updates!) can be applied without taking VMs offline. AWS doesn't support live migration. I don't work for AWS, so I can only speculate about their internal technology; but they've demonstrated an ability to apply security updates without requiring guests to reboot.
- jvolkman 10y ago
  I believe in certain situations they've been able to use ksplice or similar mechanisms.
10y ago

ascotan 10y ago

Q. How do you secure your infrastructure Google? A. We hire thousands of programmers to write custom software. There are no known vulnerabilities. No. Known. Vulnerabilities.

aiiane 10y ago
That's not even an accurate representation of what is publicly claimed; Google runs a VRP specifically because it acknowledges that software sometimes has bugs and it's committed to fixing them. Google also employs quite a few of the security researchers that contribute to identifying and fixing security issues both within and beyond Google.
stefanha 10y ago
I don't work for Google but have seen KVM Forum presentations and mailing list discussions where they contributed back to KVM. They have world-class security researchers working on keeping KVM secure. Your comment implies they rely on security through obscurity and it misses the legitimate security work they are doing. Did you read anything at all before posting your comment? Start here: http://www.linux-kvm.org/images/f/f6/01x02-KVMHardening.pdf http://www.linux-kvm.org/images/f/f6/01x02-KVMHardening.pdf

shinyshorts 10y ago

Ok, sounds fancy enough... until the money dries up, the hydrocephalic Indian and Russian programmers have left and mere mortal engineers decide to move back to stuff that everyone else uses and understands. Also, am I the only one who freaks out by this seemingly harmless remark: "errors are detected and cause any potentially offending guest VMs to be terminated."? google messing about with AI and these, oh im sure really well intended, algorithms in place. Quite scary. They've also shown with their go bot that they will play covert games.. apply AI unexpectedly. Im not sure we can trust Google any longer.

throwaway7767 10y ago

> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual machine monitor that has the following security advantages over QEMU: [...] No history of security problems. QEMU has a long track record of security bugs, such as VENOM, and it's unclear what vulnerabilities may still be lurking in the code. Has this alternative VMM/hardware emulator been released? As far as I can tell, the answer is no. In that light, it's more than a little weird to congratulate yourself on not having a "long track record of security bugs" in your internal-use-only unreleased tool compared to generally available software in wide use.

floatboth 10y ago

By the way, FreeBSD's vmm doesn't use QEMU, a fresh new userspace part (bhyve) was written. I think the same is happening with OpenBSD's vmm. It seems kinda weird that Linux KVM is commonly used with QEMU…

jron 10y ago

Does anyone have a better summary of the Xen vs KVM security debate? The best I know of is the brief Qubes architecture document from 2010: https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf https://www.qubes-os.org/attachment/wiki/QubesArchitecture/a... (Section 3). KVM using Google's hardened device emulation may eliminate some of Qube's QEMU concerns. Qubes author critique on the state of Xen security: https://lists.xen.org/archives/html/xen-devel/2015-11/msg00601.html https://lists.xen.org/archives/html/xen-devel/2015-11/msg006...