5 ms·
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual
by koolba 10y ago
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual machine monitor and hardware emulation. Instead, we wrote our own user-space virtual machine monitor that has the following security advantages over QEMU: Simple host and guest architecture support matrix. QEMU supports a large matrix of host and guest architectures, along with different modes and devices that significantly increase complexity.
I've noticed that a lot of projects that do support multiple architectures, particularly obscure ones, tend to find oddball edge cases more easily than those that don't. For example, not assuming the endianness of the CPU arch forces you to get network code to cooperate well.
> Because we support a single architecture and a relatively small number of devices, our emulator is much simpler.
No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable.
Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!".
- rrdharan 10y ago> No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. A little later in the post I believe this is somewhat addressed: > QEMU code lacks unit tests and has many interdependencies that would make unit testing extremely difficult. Personally based on my previous experience at VMware and passing familiarity with QEMU, I think they made the right call. (I work at Google, but not on the hypervisor)
- luu 10y ago> I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. > Then again when you're Google and have the resources to build a VM runtime from the ground up it's easier to convince management that "This is the right decision!". Is it possible you're either underestimating the effort it takes to make QEMU solid or overestimating the effort it takes to write an emulator? I worked at a company where I hacked up QEMU as a stopgap before we switched to an in-house solution (this wasn't Google, although I've also worked at Google). I made literally hundreds of bug fixes to get QEMU into what was, for us, a barely usable state and then someone else wrote a solution from scratch in maybe a month and a half or two months. I doubt I could have gotten QEMU into the state we needed in a couple months. And to be clear, when I say bug fixes, I don't mean features or things that could possibly arguably be "working as intended", I mean bugs like "instruction X does the wrong thing instead of doing what an actual CPU does". BTW, I don't mean to knock QEMU. It's great for what it is, but it's often the case that a special purpose piece of software tailored for a specific usecase is less effort than making a very general framework suitable for the same usecase. Even for our usecase, where QEMU was a very bad fit, the existence of QEMU let us get an MVP up in a week; I applied critical fixes to our hacked up QEMU while someone worked on the real solution, which gave us a two month head start over just writing something from scratch. But the effort it would have taken to make QEMU production worthy for us didn't seem worth it.
- koolba 10y ago> Is it possible you're either underestimating the effort it takes to make QEMU solid or overestimating the effort it takes to write an emulator? Not just possible, it's highly likely.
- walterbell 10y agoAre there open-source efforts to create a better emulator?
- wmf 10y agoIntel is working on QEMU-lite and there are some QEMU alternatives that aren't focused on emulating the full PC architecture like novm and kvmtool.
- bonzini 10y agoDepends on how you define better. If better means running old games and demos more faithfully, there's DOSEMU and DOSBOX. If better means emulating newer processors on older ones, there's Bochs. None of them supports KVM (except maybe DOSEMU??). For KVM use, QEMU is pretty much the only choice with support for a wide range of guests, architectures, and features. lkvm (aka kvmtool) doesn't support Windows, UEFI, s390 hosts, live migration, etc. At the same time, QEMU's binary code translator is improving. As pm215 said elsewhere, 64-bit ARM guest support is much better than x86 support, and we're also working on SMP which is scalable and pretty much state-of-the-art for cross-architecture emulators (of course QEMU is already scalable to multiple guest CPUs when using KVM, but doing it in an emulator is a different story).
- bonzini 10y agoKVM does not use any of the CPU/instruction emulation in QEMU. It only uses the device emulation code and the interface to the host (asynchronous I/O, sockets, VNC, etc.). We are adding unit tests for a lot of new code, and some parts of the code (especially the block device backends) have a comprehensive set of regression tests. Also, distributions can disable obsolete devices if they wish. Red Hat does that in RHEL, for both security and supportability reasons. So if you want a free hardened QEMU, use CentOS. :-) Several other companies do so, including Nutanix and Virtuozzo.
- aseipp 10y ago> I've noticed that a lot of projects that do support multiple architectures, particularly obscure ones, tend to find oddball edge cases more easily than those that don't. For example, not assuming the endianness of the CPU arch forces you to get network code to cooperate well. It's a balancing act, like anything is. Do you add or reject this patch from a contributor? This new feature someone wants, or a bug fixed? Is it better off designed/done in a different way? Is this kind of work maintainable e.g. 3 years from now when I'm not working on it? Can we reliably continue to support these systems or know someone who will care for them? Can we reasonably understand the entire system, and evolve it safely? Do you need more moving parts than are absolutely required? Does that use case even matter, everything else aside? The last one is very important. Of course there's something to be said about software systems that maintain high quality code while supporting a diverse set of use cases and environments, like you mention. But -- I'd probably say that, that result? It's likely more a function of focused design than it is a function of "trying to target a lot of architectures". Maybe targeting lots of architectures was a design goal, but nonetheless, the designing aspect is what's important. No amount of portability can fix fundamental misunderstandings of what you're trying to implement, of course. And that's where a lot of problems can creep in. In this case, they have a very clear view of what they want, and a lot of what QEMU does is very irrelevant. It may be part of QEMU's design to be portable, but ultimately it is a lot of unnecessary, moving parts. You can cut down its scope dramatically with this knowledge -- from a security POV, that's very often going to be a win, to remove that surface area. (Also, I'm definitely not saying QEMU is bloated or something, either. It's great software and I use it every day, just to be clear.)
- mappu 10y agoThe KVM API is quite simple: https://lwn.net/Articles/658511/ https://lwn.net/Articles/658511/
- jsolson 10y ago> No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if they're only for the specific architectures they're running (most likely x86-64) would have been just as usable. There are some folks who have this view, but doing it from scratch also has the advantage that it integrates much more cleanly with Google's global shared codebase. There's a huge body of existing work that I can leverage more or less trivially. This includes things like Google's internal metrics and monitoring framework, our RPC framework, etc. Yes, you could bolt these onto the side of qemu, but qemu is a C codebase and most of Google (including the custom VMM described in the article) is not. Additionally, when software is built using the same style, tools, and best practices as the rest of Google's codebase, it makes it easy for other engineers in the company to contribute. We benefit from Google-wide code cleanups, *SAN analysis tools, codebase-wide refactorings that make code easier to reason about the correctness of, etc. Several years ago I think the question would've been a lot more difficult to answer, but today I the advantages of the route taken are unambiguous. (my team owns the virtual network devices visible to GCE VM and the first chunk of the on-host dataplane, one virtual hardware component of the custom VMM we run :)