7 ms·

QEMU Internals

323 points by Nusyne 5y ago

pthreads 5y ago
Thank you. On the same subject can someone recommend a book or any other resource to learn about virtual machine internals? My goal is to try to build a toy clone of VirtualBox/VMWare. So far I have found one -- Virtual Machines by James E. Smith and Ravi Nair.
- hag 5y ago
  I've always been intrigued by virtual machines and emulation as well. I've always wanted to try and make an emulator of some kind. I don't know much about the internals of VirtualBox, but my suggestion would be to start "easy" with one CPU/Computer System/Game Console and go from there. That's what I finally did with the 6502 and Commodore 64.
  - pizza234 5y ago
    Conventionally, one starts from the CHIP-8, which is indeed a virtual machine rather than a system in a strict sense. What I've found difficult is the step beyond that. NES and GameBoy are typical steps, however, I've been very frustrated by the confusing documentation of the GameBoy. There are 3/4 references, but one of them has significant mistakes, while another is incomplete. On the other hand, the Pan Docs should be complete and accurate. I'm not sure if there is an easy middle ground, that, at the same time, is also well documented. The Atary 2600 is architecturally simpler but less documented, and also requires very accurate timings. I've read somebody suggesting systems like Channel F, Astrocade and Odyssey2, but I'm not sure they're well documented. I've personally lost my interest once I've found that building an emulator was essentially fighting specifications rather than actually building something.
    - toast0 5y ago
      I built about a third of a NES emulator. The nesdev wiki is mostly decent, although there's a fair number of things where it seems like the first people to figure things out got stuff kind of backwards, and if you flip it, it's a lot easier, that's the sort of fighting the specifications I think you're talking about. All that said, emulating the CPU was pretty fun. There's a CPU test rom out there you can run with tracing and compare to the published results. I also got the background tiling from the PPU done, but the foreground processing has a lot of steps, so I indefinitely paused for now. Also, I had amazingly poor performance, so I wasn't super motivated to continue. The 2600 has a very similar cpu, but the very limited Stella output chip means most games are very timing dependent, which means you have to be super accurate, which adds difficulty. I think you should try to be cycle accurate anyway, but it's easy to mess that up, and having some freedom would be nice.
      - bambataa 5y ago
        I did a GameBoy and similarly found the CPU enjoyable and the PPU a huge pain. Perhaps if I understood graphics better, I would have enjoyed it more, but like you say it just felt like a lot of steps.
        toast0 5y ago
        I don't know if the GameBoy PPU has the background vs foreground split. The background processing was pretty reasonable, and once you got it kind of working, it was fun to debug and get it actually working. My favorite thing was when I was processing everything in the wrong order so the menu of the rom I was using to test had all the words backwards. But the foreground / object sprites have this huge pipeline. IIRC, the PPU determines which sprites to draw in line X + 1 during line X. After that, it has to load the data for each object, etc etc. It was just discouraging. Plus since my frame rate is so low, I have to sit at a blank screen for quite some time waiting for the game to show anything, and longer for the demo to start (I don't have controls)... Anyhow, glad I'm not alone ;)
    - andrewf 5y ago
      A subset of CP/M calls is a pretty simple "rest of the system" to implement on top of an 8080/Z80 CPU emulation. (It's a bit of a cheat - like qemu's "Linux user mode emulation" or early version of DOSBox, because you restrict software to interacting with a high-level software interface, there are no lower-level details to aim for fidelity with)
- DarmokJalad1701 5y ago
  For a really simple emulator project (not quite the level of VirtualBox), check the "IntCode" challenges from AdventOfCode 2019.
  - sammorrowdrums 5y ago
    Those were so fun! I loved my little VM as it progressed and played pong, and commanded robots and rendered the output etc. It's a really great fun way to learn the key concepts.
  - Scipio_Afri 5y ago
    I believe this is the direct link to the mentioned challenges: https://adventofcode.com/2019 https://adventofcode.com/2019
- vitno 5y ago
  I work on virtual machines at Google. I usually suggest "Hardware and Software Support for Virtualization" [1] to new team members without a virtualization background. [1] https://www.amazon.com/Hardware-Software-Virtualization-Synthesis-Architecture/dp/1627056939 https://www.amazon.com/Hardware-Software-Virtualization-Synt...
  - bogomipz 5y ago
    This looks like a good read, thanks. I'm curious what your background is. How does one go about getting into that specialty at an org like Google? I've understood that Borg and GKE containers at Google generally always run in a VM. Is this where your work is(platform) or are you more research oriented?
    - vitno 5y ago
      Generally "systems-y software" is my background. I joined Google for a semi-experimental operating system project and from there it was a small jump to virtualization when I decided I was interested in doing something else. I'm definitely on the platform side, but been doing some interesting stuff recently :)
      - divtiwari 5y ago
        > semi-experimental operating system project Fuchsia? :)
      - bogomipz 5y ago
        Thanks. Do you have any other recommendations for "systems-y software" books or resources you think would be helpful for a people pursuing similar roles?
- ahefner 5y ago
  "KVM host in a few lines of code" (https://zserge.com/posts/kvm/ https://zserge.com/posts/kvm/) is a fun article to get started with.
- teleforce 5y ago
  The sibling's comment book recommendation "Hardware and Software Support for Virtualization" book is on point and it's written by one of the co-founders of VMware. Another book on Libvirt will be handy since it is the de facto API for most virtualization including VMs and containers[1]. [1]https://www.amazon.com/Foundations-Libvirt-Development-Maintain-Environment/dp/1484248619 https://www.amazon.com/Foundations-Libvirt-Development-Maint...
- deleted 5y ago
  [deleted]
- alert0 5y ago
  Fuzz week shows how to make make a snapshot / resettable jitting hypervisor. https://m.youtube.com/playlist?list=PLSkhUfcCXvqHsOy2VUxuoAf5m_7c8RqvO https://m.youtube.com/playlist?list=PLSkhUfcCXvqHsOy2VUxuoAf...
  - harry8 5y ago
    9 lectures for a total of 59+ hours! Wow. Commitment required.
    - alert0 5y ago
      Yea, it's streaming so there are some debugging sessions in there, but the information is gold.
- tkhattra 5y ago
  Hardware and Software Support for Virtualization Synthesis Lectures on Computer Architecture (2017) https://www.morganclaypool.com/doi/abs/10.2200/S00754ED1V01Y201701CAC038 https://www.morganclaypool.com/doi/abs/10.2200/S00754ED1V01Y... Bringing Virtualization to the x86 Architecture with the Original VMware Workstation (2012) https://dl.acm.org/doi/abs/10.1145/2382553.2382554 https://dl.acm.org/doi/abs/10.1145/2382553.2382554
junon 5y ago
This is very well organized, wow.
whoisburbansky 5y ago
I don't mean this to disparage Airbus in any way but after Boeing's issues with the 737 MAX I'd assumed a fairly poor culture of software at airplane manufacturers in general. Super glad to see work like this coming out of Airbus, really makes me rethink my earlier assumptions about software competence in the field.
- hhh 5y ago
  Airbus also has the Airbus Defense and Space group as well, it’s not just all airplanes :)
  - deleted 5y ago
    [deleted]
- Veserv 5y ago
  That is such a bizarre viewpoint from my perspective. The absolute deathtrap that is the 737 MAX had two software-related critical failures in 400,000 flights. That constitutes a whole system per-flight software reliability of 2 in ~400,000 or a ~99.9995%, 5 9s. Obviously that is still unacceptable as that is far below the software standard amongst all commercial airplanes where software has not been implicated in a crash for at least the last 10 years except for the 737 MAX. Even if we include the two 737 MAX crashes into the statistics, the whole system per-flight software reliability of all commercial airplanes over the last decade is at least 2 in ~100,000,000 or ~99.999998% or 7 9s. The standard in airplane software is literally 5000x more reliable than AWS SLA guarantees and 500x the holy grail in server software of 5 9s. Even the 737 MAX is 20x better than the AWS guarantee and 2x more reliable than 5 9s. Airplane software is not bad, we just rightfully expect a lot from systems that lives depend on, so even systems that are better than best-in-class non-safety software are completely unacceptable which may give the impression that they are bad in absolute terms as they fail to live up to our expectations.
  - zaphirplane 5y ago
    That’s an interesting way to look at uptime no pun intended thou I wouldn’t buy a Toyota that exploded every 400,000 trips world wide Or bank with a bank that lost all my money every 400,000 transactions world wide

323 points by Nusyne 5y ago

Thank you. On the same subject can someone recommend a book or any other resource to learn about virtual machine internals? My goal is to try to build a toy clone of VirtualBox/VMWare. So far I have found one -- Virtual Machines by James E. Smith and Ravi Nair.

hag 5y ago
I've always been intrigued by virtual machines and emulation as well. I've always wanted to try and make an emulator of some kind. I don't know much about the internals of VirtualBox, but my suggestion would be to start "easy" with one CPU/Computer System/Game Console and go from there. That's what I finally did with the 6502 and Commodore 64.
- pizza234 5y ago
  Conventionally, one starts from the CHIP-8, which is indeed a virtual machine rather than a system in a strict sense. What I've found difficult is the step beyond that. NES and GameBoy are typical steps, however, I've been very frustrated by the confusing documentation of the GameBoy. There are 3/4 references, but one of them has significant mistakes, while another is incomplete. On the other hand, the Pan Docs should be complete and accurate. I'm not sure if there is an easy middle ground, that, at the same time, is also well documented. The Atary 2600 is architecturally simpler but less documented, and also requires very accurate timings. I've read somebody suggesting systems like Channel F, Astrocade and Odyssey2, but I'm not sure they're well documented. I've personally lost my interest once I've found that building an emulator was essentially fighting specifications rather than actually building something.
  - toast0 5y ago
    I built about a third of a NES emulator. The nesdev wiki is mostly decent, although there's a fair number of things where it seems like the first people to figure things out got stuff kind of backwards, and if you flip it, it's a lot easier, that's the sort of fighting the specifications I think you're talking about. All that said, emulating the CPU was pretty fun. There's a CPU test rom out there you can run with tracing and compare to the published results. I also got the background tiling from the PPU done, but the foreground processing has a lot of steps, so I indefinitely paused for now. Also, I had amazingly poor performance, so I wasn't super motivated to continue. The 2600 has a very similar cpu, but the very limited Stella output chip means most games are very timing dependent, which means you have to be super accurate, which adds difficulty. I think you should try to be cycle accurate anyway, but it's easy to mess that up, and having some freedom would be nice.
    - bambataa 5y ago
      I did a GameBoy and similarly found the CPU enjoyable and the PPU a huge pain. Perhaps if I understood graphics better, I would have enjoyed it more, but like you say it just felt like a lot of steps.
      - toast0 5y ago
        I don't know if the GameBoy PPU has the background vs foreground split. The background processing was pretty reasonable, and once you got it kind of working, it was fun to debug and get it actually working. My favorite thing was when I was processing everything in the wrong order so the menu of the rom I was using to test had all the words backwards. But the foreground / object sprites have this huge pipeline. IIRC, the PPU determines which sprites to draw in line X + 1 during line X. After that, it has to load the data for each object, etc etc. It was just discouraging. Plus since my frame rate is so low, I have to sit at a blank screen for quite some time waiting for the game to show anything, and longer for the demo to start (I don't have controls)... Anyhow, glad I'm not alone ;)
  - andrewf 5y ago
    A subset of CP/M calls is a pretty simple "rest of the system" to implement on top of an 8080/Z80 CPU emulation. (It's a bit of a cheat - like qemu's "Linux user mode emulation" or early version of DOSBox, because you restrict software to interacting with a high-level software interface, there are no lower-level details to aim for fidelity with)
DarmokJalad1701 5y ago
For a really simple emulator project (not quite the level of VirtualBox), check the "IntCode" challenges from AdventOfCode 2019.
- sammorrowdrums 5y ago
  Those were so fun! I loved my little VM as it progressed and played pong, and commanded robots and rendered the output etc. It's a really great fun way to learn the key concepts.
- Scipio_Afri 5y ago
  I believe this is the direct link to the mentioned challenges: https://adventofcode.com/2019 https://adventofcode.com/2019
vitno 5y ago
I work on virtual machines at Google. I usually suggest "Hardware and Software Support for Virtualization" [1] to new team members without a virtualization background. [1] https://www.amazon.com/Hardware-Software-Virtualization-Synthesis-Architecture/dp/1627056939 https://www.amazon.com/Hardware-Software-Virtualization-Synt...
- bogomipz 5y ago
  This looks like a good read, thanks. I'm curious what your background is. How does one go about getting into that specialty at an org like Google? I've understood that Borg and GKE containers at Google generally always run in a VM. Is this where your work is(platform) or are you more research oriented?
  - vitno 5y ago
    Generally "systems-y software" is my background. I joined Google for a semi-experimental operating system project and from there it was a small jump to virtualization when I decided I was interested in doing something else. I'm definitely on the platform side, but been doing some interesting stuff recently :)
    - divtiwari 5y ago
      > semi-experimental operating system project Fuchsia? :)
    - bogomipz 5y ago
      Thanks. Do you have any other recommendations for "systems-y software" books or resources you think would be helpful for a people pursuing similar roles?
ahefner 5y ago
"KVM host in a few lines of code" (https://zserge.com/posts/kvm/ https://zserge.com/posts/kvm/) is a fun article to get started with.
teleforce 5y ago
The sibling's comment book recommendation "Hardware and Software Support for Virtualization" book is on point and it's written by one of the co-founders of VMware. Another book on Libvirt will be handy since it is the de facto API for most virtualization including VMs and containers[1]. [1]https://www.amazon.com/Foundations-Libvirt-Development-Maintain-Environment/dp/1484248619 https://www.amazon.com/Foundations-Libvirt-Development-Maint...
deleted 5y ago
[deleted]
alert0 5y ago
Fuzz week shows how to make make a snapshot / resettable jitting hypervisor. https://m.youtube.com/playlist?list=PLSkhUfcCXvqHsOy2VUxuoAf5m_7c8RqvO https://m.youtube.com/playlist?list=PLSkhUfcCXvqHsOy2VUxuoAf...
- harry8 5y ago
  9 lectures for a total of 59+ hours! Wow. Commitment required.
  - alert0 5y ago
    Yea, it's streaming so there are some debugging sessions in there, but the information is gold.
tkhattra 5y ago
Hardware and Software Support for Virtualization Synthesis Lectures on Computer Architecture (2017) https://www.morganclaypool.com/doi/abs/10.2200/S00754ED1V01Y201701CAC038 https://www.morganclaypool.com/doi/abs/10.2200/S00754ED1V01Y... Bringing Virtualization to the x86 Architecture with the Original VMware Workstation (2012) https://dl.acm.org/doi/abs/10.1145/2382553.2382554 https://dl.acm.org/doi/abs/10.1145/2382553.2382554

junon 5y ago

This is very well organized, wow.

whoisburbansky 5y ago

I don't mean this to disparage Airbus in any way but after Boeing's issues with the 737 MAX I'd assumed a fairly poor culture of software at airplane manufacturers in general. Super glad to see work like this coming out of Airbus, really makes me rethink my earlier assumptions about software competence in the field.

hhh 5y ago
Airbus also has the Airbus Defense and Space group as well, it’s not just all airplanes :)
- deleted 5y ago
  [deleted]
Veserv 5y ago
That is such a bizarre viewpoint from my perspective. The absolute deathtrap that is the 737 MAX had two software-related critical failures in 400,000 flights. That constitutes a whole system per-flight software reliability of 2 in ~400,000 or a ~99.9995%, 5 9s. Obviously that is still unacceptable as that is far below the software standard amongst all commercial airplanes where software has not been implicated in a crash for at least the last 10 years except for the 737 MAX. Even if we include the two 737 MAX crashes into the statistics, the whole system per-flight software reliability of all commercial airplanes over the last decade is at least 2 in ~100,000,000 or ~99.999998% or 7 9s. The standard in airplane software is literally 5000x more reliable than AWS SLA guarantees and 500x the holy grail in server software of 5 9s. Even the 737 MAX is 20x better than the AWS guarantee and 2x more reliable than 5 9s. Airplane software is not bad, we just rightfully expect a lot from systems that lives depend on, so even systems that are better than best-in-class non-safety software are completely unacceptable which may give the impression that they are bad in absolute terms as they fail to live up to our expectations.
- zaphirplane 5y ago
  That’s an interesting way to look at uptime no pun intended thou I wouldn’t buy a Toyota that exploded every 400,000 trips world wide Or bank with a bank that lost all my money every 400,000 transactions world wide