5 ms·

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest explo

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y

_rs 2y ago
I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?
- bonzini 2y ago
  AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.
  - tptacek 2y ago
    Lambda and Fargate.
    - dastbe 2y ago
      unless something has changed in the past year, fargate still runs each task in a single use ec2 vm with no further isolation around containers in a task.
      - deleted 2y ago
        [deleted]
    - my123 2y ago
      It was true for Fargate some time ago, but is not true anymore since quite a while. All Fargate tasks run on EC2 instances today.
      - easton 2y ago
        …which is probably the reason why task launches take 3-5 business weeks
      - tptacek 2y ago
        Ah, interesting. Thanks for the correction!
  - 9front 2y ago
    EC2 instances are using the Xen hypervisor. At least that's what reported by hostnamectl.
    - wmf 2y ago
      EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.
      - 9front 2y ago
        I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023 Kernel: Linux 6.1.106-116.188.amzn2023.x86_64 Architecture: x86-64 Hardware Vendor: Xen Hardware Model: HVM domU Firmware Version: 4.11.amazon
        wmf 2y ago
        What instance type is it?
        bonzini 2y ago
        KVM can emulate the Xen hypercall interface. Amazon is not using Xen anymore.
        simcop2387 2y ago
        I'm not quite sure the status of it at least, but reported back in 2017 that they are moving off Xen https://www.theregister.com/2017/11/07/aws_writes_new_kvm_based_hypervisor_to_make_its_cloud_go_faster https://www.theregister.com/2017/11/07/aws_writes_new_kvm_ba... It could be that it's not all over and tied to specific machine types still, or there's something they've done to make it report to the guest still that it's xen based for some compatibility reasons.
- daneel_w 2y ago
  QEMU can use a number of different hypervisors, KVM and Xen being the two most common ones. Additionally it can also emulate any architecture if one would want/need that.
TimTheTinker 2y ago
> When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. So there existed known guest-exploitable vulnerabilities as recently as 8 years ago. Maybe that, combined with the fact that QEMU is not written in Rust, is what is causing Oxide to decide against QEMU. I think it's fair to say that any sufficiently large codebase originally written in C or C++ has memory safety bugs. Yes, the Oxide RFD author may be phrasing this using weasel words; and memory safety bugs may not be exploitable at a given point in a codebase's history. But I don't think that makes Oxide's decision invalid.
- bonzini 2y ago
  That would be a damn good record though, isn't it? (I am fairly sure that more were found since, but the point is that these are pretty rare). Firecracker, which is written in Rust, had one in 2019: https://www.cve.org/CVERecord?id=CVE-2019-18960 https://www.cve.org/CVERecord?id=CVE-2019-18960 Also QEMU's fuzzing is very sophisticated. Most recent vulnerabilities were found that way rather than by security researchers, which I don't think it's the case for "competitors".
  - TimTheTinker 2y ago
    You're not wrong, and that is very impressive. There's nothing like well-applied fuzzing to improve security. But I still don't think that makes Oxide's decision or my comment necessarily invalid, if only because of an a priori decision to stick with Rust system-wide -- it raises the floor on software quality.
    - akira2501 2y ago
      > it raises the floor on software quality. Languages cannot possibly do this.
      - TimTheTinker 2y ago
        I believe TypeScript and Rust are both strong examples of languages that do this (for different reasons and in different ways). It's also possible for a language to raise the ceiling of software quality, and Zig is an excellent example. I'm thinking of "floors" and "ceilings" as the outer bounds of what happens in real, everyday life within particular software ecosystems in terms of software quality. By "quality" I mean all of capabilities, performance, and absence of problems. It takes a team of great engineers (and management willing to take a risk) to benefit from a raised ceiling. TigerBeetle[0] is an example of what happens when you pair a great team, great research, and a high-ceiling language. [0] https://tigerbeetle.com/ https://tigerbeetle.com/
dvdbloc 2y ago
What do the big three use?
- paxys 2y ago
  AWS – Nitro (based on KVM) Google – "KVM-based hypervisor" Azure – Hyper-V You can of course assume that all of them heavily customize the underlying implemenation for their own needs and for their own hardware. And then they have stuff like Firecracker, GVisor etc. layered on top depending on the product line.
- daneel_w 2y ago
  Some more data: Oracle Cloud - QEMU/KVM Scaleway - QEMU/KVM
  - bonzini 2y ago
    IBM cloud, DigitalOcean, Linode, OVH, Hetzner,...
anonfordays 2y ago
>Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. QEMU typically uses KVM for the hypervisor, so the vulnerabilities will be KVM anyway. The big three all use KVM now. Oxide decided to go with bhyve instead of KVM.
- bonzini 2y ago
  No, QEMU is a huge C program which can have its own vulnerabilities. Usually QEMU runs heavily confined, but remote code execution in QEMU (remote = "from the guest") can be a first step towards exploiting a more serious local escalation via a kernel vulnerability. This second vulnerability can be in KVM or in any other part of the kernel.
- 6c696e7578 2y ago
  Azure uses hyper-v, unless things have changed massively, the linux they run for infra and customers is in hyper-v.
- cmeacham98 2y ago
  > The big three all use KVM now. This isn't true - Azure uses Hyper-V (https://learn.microsoft.com/en-us/azure/security/fundamentals/hypervisor https://learn.microsoft.com/en-us/azure/security/fundamental...), and AWS uses an in-house hypervisor called Nitro (https://aws.amazon.com/ec2/nitro/ https://aws.amazon.com/ec2/nitro/).
  - anonfordays 2y ago
    >This isn't true - Azure uses Hyper-V I thought Azure was moving/moved to KVM for Linux, but I was wrong. >AWS uses an in-house hypervisor called Nitro Nitro uses KVM under the hood.
    - cthalupa 2y ago
      Nitro does not use QEMU or many other parts traditionally used with KVM, however. https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtual...
deleted 2y ago
[deleted]
hinkley 2y ago
If they are being precise, then “reliability and security” means something different than “security and reliability”. How many reliability bugs has QEMU experienced in this time? The man power to go on site and deal with in the field problems could be crippling. You often pick the boring problems for this reason. High touch is super expensive. Just look at Ferrari.

_rs 2y ago

I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?

bonzini 2y ago
AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.
- tptacek 2y ago
  Lambda and Fargate.
  - dastbe 2y ago
    unless something has changed in the past year, fargate still runs each task in a single use ec2 vm with no further isolation around containers in a task.
    - deleted 2y ago
      [deleted]
  - my123 2y ago
    It was true for Fargate some time ago, but is not true anymore since quite a while. All Fargate tasks run on EC2 instances today.
    - easton 2y ago
      …which is probably the reason why task launches take 3-5 business weeks
    - tptacek 2y ago
      Ah, interesting. Thanks for the correction!
- 9front 2y ago
  EC2 instances are using the Xen hypervisor. At least that's what reported by hostnamectl.
  - wmf 2y ago
    EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.
    - 9front 2y ago
      I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023 Kernel: Linux 6.1.106-116.188.amzn2023.x86_64 Architecture: x86-64 Hardware Vendor: Xen Hardware Model: HVM domU Firmware Version: 4.11.amazon
      - wmf 2y ago
        What instance type is it?
      - bonzini 2y ago
        KVM can emulate the Xen hypercall interface. Amazon is not using Xen anymore.
      - simcop2387 2y ago
        I'm not quite sure the status of it at least, but reported back in 2017 that they are moving off Xen https://www.theregister.com/2017/11/07/aws_writes_new_kvm_based_hypervisor_to_make_its_cloud_go_faster https://www.theregister.com/2017/11/07/aws_writes_new_kvm_ba... It could be that it's not all over and tied to specific machine types still, or there's something they've done to make it report to the guest still that it's xen based for some compatibility reasons.
daneel_w 2y ago
QEMU can use a number of different hypervisors, KVM and Xen being the two most common ones. Additionally it can also emulate any architecture if one would want/need that.

TimTheTinker 2y ago

> When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. So there existed known guest-exploitable vulnerabilities as recently as 8 years ago. Maybe that, combined with the fact that QEMU is not written in Rust, is what is causing Oxide to decide against QEMU. I think it's fair to say that any sufficiently large codebase originally written in C or C++ has memory safety bugs. Yes, the Oxide RFD author may be phrasing this using weasel words; and memory safety bugs may not be exploitable at a given point in a codebase's history. But I don't think that makes Oxide's decision invalid.

bonzini 2y ago
That would be a damn good record though, isn't it? (I am fairly sure that more were found since, but the point is that these are pretty rare). Firecracker, which is written in Rust, had one in 2019: https://www.cve.org/CVERecord?id=CVE-2019-18960 https://www.cve.org/CVERecord?id=CVE-2019-18960 Also QEMU's fuzzing is very sophisticated. Most recent vulnerabilities were found that way rather than by security researchers, which I don't think it's the case for "competitors".
- TimTheTinker 2y ago
  You're not wrong, and that is very impressive. There's nothing like well-applied fuzzing to improve security. But I still don't think that makes Oxide's decision or my comment necessarily invalid, if only because of an a priori decision to stick with Rust system-wide -- it raises the floor on software quality.
  - akira2501 2y ago
    > it raises the floor on software quality. Languages cannot possibly do this.
    - TimTheTinker 2y ago
      I believe TypeScript and Rust are both strong examples of languages that do this (for different reasons and in different ways). It's also possible for a language to raise the ceiling of software quality, and Zig is an excellent example. I'm thinking of "floors" and "ceilings" as the outer bounds of what happens in real, everyday life within particular software ecosystems in terms of software quality. By "quality" I mean all of capabilities, performance, and absence of problems. It takes a team of great engineers (and management willing to take a risk) to benefit from a raised ceiling. TigerBeetle[0] is an example of what happens when you pair a great team, great research, and a high-ceiling language. [0] https://tigerbeetle.com/ https://tigerbeetle.com/

dvdbloc 2y ago

What do the big three use?

paxys 2y ago
AWS – Nitro (based on KVM) Google – "KVM-based hypervisor" Azure – Hyper-V You can of course assume that all of them heavily customize the underlying implemenation for their own needs and for their own hardware. And then they have stuff like Firecracker, GVisor etc. layered on top depending on the product line.
daneel_w 2y ago
Some more data: Oracle Cloud - QEMU/KVM Scaleway - QEMU/KVM
- bonzini 2y ago
  IBM cloud, DigitalOcean, Linode, OVH, Hetzner,...

anonfordays 2y ago

>Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. QEMU typically uses KVM for the hypervisor, so the vulnerabilities will be KVM anyway. The big three all use KVM now. Oxide decided to go with bhyve instead of KVM.

bonzini 2y ago
No, QEMU is a huge C program which can have its own vulnerabilities. Usually QEMU runs heavily confined, but remote code execution in QEMU (remote = "from the guest") can be a first step towards exploiting a more serious local escalation via a kernel vulnerability. This second vulnerability can be in KVM or in any other part of the kernel.
6c696e7578 2y ago
Azure uses hyper-v, unless things have changed massively, the linux they run for infra and customers is in hyper-v.
cmeacham98 2y ago
> The big three all use KVM now. This isn't true - Azure uses Hyper-V (https://learn.microsoft.com/en-us/azure/security/fundamentals/hypervisor https://learn.microsoft.com/en-us/azure/security/fundamental...), and AWS uses an in-house hypervisor called Nitro (https://aws.amazon.com/ec2/nitro/ https://aws.amazon.com/ec2/nitro/).
- anonfordays 2y ago
  >This isn't true - Azure uses Hyper-V I thought Azure was moving/moved to KVM for Linux, but I was wrong. >AWS uses an in-house hypervisor called Nitro Nitro uses KVM under the hood.
  - cthalupa 2y ago
    Nitro does not use QEMU or many other parts traditionally used with KVM, however. https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtual...

deleted 2y ago

[deleted]

hinkley 2y ago

If they are being precise, then “reliability and security” means something different than “security and reliability”. How many reliability bugs has QEMU experienced in this time? The man power to go on site and deal with in the field problems could be crippling. You often pick the boring problems for this reason. High touch is super expensive. Just look at Ferrari.