5 ms·

Deterministic Replay of QEMU Emulation

175 points by Intralexical 2y ago

jester1337 2y ago
I remember we were working on this exact topic at my University chair ~8-10 years ago. I think it never fully took off. Several Master students worked on it for a while. I like that it's now in QEMU!
repelsteeltje 2y ago
I think this is awesome. While it might seem like a small feature, it opens a huge door. It's similar to what reproducible build infrastructure has done for finding bugs, attestation that binary matches source, immutability, etc. Can imagine this is useful for finding bugs in hardware designs too.
waschl 2y ago
Tried to apply it for debugging on my own OS, but couldn’t get it finally running after several days of trial and error… https://github.com/jbreu/jos?tab=readme-ov-file#reverse-debugging-via-qemu https://github.com/jbreu/jos?tab=readme-ov-file#reverse-debu...
- junon 2y ago
  Yeah QEMU's story for this sort of thing is pretty rough around the edges for OS dev. Wishing there was something like Unicorn-but-with-devices for making osdev tooling.
  - SoothingSorbet 2y ago
    There's also panda[1], but I never got it working myself. I share your frustration, as it would help greatly with debugging, especially with nondeterminstic bugs. I likewise never got QEMU's record/replay to work. [1] https://github.com/panda-re/panda https://github.com/panda-re/panda
- b112 2y ago
  Qemu is great, the DEVs and all who worked on her deserve applause, except for documentation. It's like someone creating a huge Japanese titanium indestructible fighting robot, but then using aluminum in the feet/heels. So much of my qemu work spent on randomly changing options, with no change documentation, discovering features, with no documentation, options with no reason or indication why, manpages out of date, READMEs not updated, changelog not there, etc.
  - cedws 2y ago
    Agreed, the CLI in particular is a complete mess.
    - bonzini 2y ago
      If there are parts that specifically you'd want to have better documentation for, please let me know here. Generally we've been moving command line towards a scheme where each option describes an aspect of either the guest (a device, the board type, the CPU model) or the interface to the host (a file holding the contents of the disk, the network bridge to attach to, how to show graphic contents), with some options providing both as a shortcut (for example -nic, -audio, -serial).
  - bonzini 2y ago
    Documentation is not great I admit. The problem is that we don't have anyone who is a capable tech writer in the team. It's not something that you can improvise. However, all incompatible changes are documented and also announced at least 8 months in advance. https://www.qemu.org/docs/master/about/removed-features.html https://www.qemu.org/docs/master/about/removed-features.html https://www.qemu.org/docs/master/about/deprecated.html https://www.qemu.org/docs/master/about/deprecated.html It may seem like there are many, but in practice they are in very old, mostly unused or very badly designed corners. For example configuration of audio was overhauled last year, and is now the same as basically all other backends (e.g. -audio pa,model=sb16; compare with -nic user,model=e1000 for a network card).
    - justinclift 2y ago
moondev 2y ago
Such a casual and low-key introduction of what sounds like an incredible new capability. 1 . Would something like this replace packer for creating machine images? 2. Curious how quickly the replay log grows and how it compares to a CoW snapshot. 3. Will be interesting what the log looks like and what doors could open up creating or generating it by other means.
- owyn 2y ago
  It is very cool, but I think some version of this feature has been around for years? This commit is from 7 years ago, and it looks like the code originates back to 2010. https://github.com/qemu/qemu/blob/v2.9.0/docs/replay.txt https://github.com/qemu/qemu/blob/v2.9.0/docs/replay.txt That said, I was not aware of it until I saw this post, and I definitely want to play around with it.
  - Intralexical 2y ago
    Well, I guess it was probably new when the doc page was first written to introduce it. > That said, I was not aware of it until I saw this post, and I definitely want to play around with it. You could almost say it was too casual and low-key. ;)
majke 2y ago
This is a big deal. With some tooling around it can be amazing. I can think of using this for testing, and as a vehicle to change a programming paradigm of existing/legacy software (run a thing, and roll it back aggressively from outside of a vm)
- m000 2y ago
  Indeed, the tooling is the problem. And I wouldn't hold my breath to see this tooling being implemented, as the feature has been around for quite a bit. IMHO, PANDA [1] remains a better/more practical choice for whole-system record/replay analysis. It already offers quite a bit of tooling (including a python interface), as well as hooks to build your own. It does have its own shortcomings (speed and not being in-sync with the latest QEMU), but at least you're not limited to gdb-based debugging. [1] https://panda.re/ https://panda.re/
- darby_nine 2y ago
  This is the central premise of Antithesis: https://antithesis.com https://antithesis.com (no affiliation)
justinclift 2y ago
Anyone have clear ideas/guidelines for how much ram/disk/etc this is likely to need for a "reasonable" capture? Say capturing a Qt application as it corrupts its internal state during startup, in order to work out what's corrupting its internal state?
- londons_explore 2y ago
  I'm gonna guess not that much - easily doable on a typical desktop. If it were a problem, you can skip recording your emulated machines bootup process, and simply take a snapshot when you're about to start your QT application. That snapshot probably only takes about 10% extra RAM because most of RAM contents wont change between the snapshot and the live system.
- jraph 2y ago
  I don't know, however a key element is: > Record/replay system is based on saving and replaying non-deterministic events > The following non-deterministic data from peripheral devices is saved into the log: mouse and keyboard input, network packets, audio controller input, serial port input, and hardware clocks (they are non-deterministic too, because their values are taken from the host machine). Inputs from simulated hardware, memory of VM, software interrupts, and execution of instructions are not saved into the log, because they are deterministic and can be replayed by simulating the behavior of virtual machine starting from initial state. So, it's probably not much, you can probably comfortably save minutes of qemu sessions. Also note the existence of the rr debugger [1], which allows you to reverse debug applications with a ~10% performance hit while recording. To achieve this, it records results of syscalls (only). It will serialize thread events, so have the effect of running applications like on a single core CPU. [1] https://rr-project.org/ https://rr-project.org/
vessenes 2y ago
Seems to me like one of the highest and best uses of this right now would be adding verifiable builds to … literally anything. You no longer need a verifiable-build-capable compiler or language — you can just run the compile and packaging step through a deterministic QEMU session. Does this sound right? I’m trying to figure out where uncontrollable randomness would come in during a compile phase, and coming up blank.
- 0points 2y ago
  Historically, major causes for non-determinism was embedded timestamps and unsorted file listings created by the build tools. I have not followed the progress recently, but https://reproducible-builds.org/ https://reproducible-builds.org/ is a starting point if you are interested. There is a sane path forward for reproducibility on bare metal, no custom emulation is needed.
  - vessenes 2y ago
    Thanks for the link, I’m aware of reproducible-builds.org. Both your causes seem trivially fixable here - the QEMU builds could have a standard system clock time they start with, and an ‘unsorted’ file listing made in a deterministic OS environment will keep the same file order, no? By comparison the rb.org site says you need to start with stripping all that stuff out of your build process, for the reasons you refer to.
    - repelsteeltje 2y ago
      > Both your causes seem trivially fixable here [] You'd be amazed about the amount of indeterminsim lurking in the guts of depencies all the way into libc and os ... Like locale, fs
    - commercialnix 2y ago
      I like your thinking. Deterministic replay with QEMU is supplemental to the larger goal of reproducible builds. The communities concerned with the topic of reproducible software not only expect cohesive human-readable code that runs deterministically to produce binary reproducible results, but their originally stated goals require it. Deterministic replay with QEMU is a "power tool" in the larger picture of these efforts.
rrdharan 2y ago
VMware had this many years ago; it was very cool: http://stackframe.blogspot.com/2007/10/configuring-application-debugging-with.html http://stackframe.blogspot.com/2007/10/configuring-applicati... http://www.replaydebugging.com/2008/08/vmware-workstation-65-reverse-and.html http://www.replaydebugging.com/2008/08/vmware-workstation-65...
- justinclift 2y ago
  Any idea if that works in modern VMware Workstation? It's currently on version 17, whereas that post was for version 6.5. VMware Workstation has such disjointed development spurts that it wouldn't surprise me if the feature had been ripped out at some point. Other useful features such as machine groups have been. :(
  - roca 2y ago
    It was removed in 2011.
  - rrdharan 2y ago
    http://www.replaydebugging.com/2011/09/goodbye-replay-debugging.html http://www.replaydebugging.com/2011/09/goodbye-replay-debugg...
    - justinclift 2y ago
      Thanks. Yeah, I kind of expected that. :(
    - m000 2y ago
      I guess we can say it was too cool.

175 points by Intralexical 2y ago

I remember we were working on this exact topic at my University chair ~8-10 years ago. I think it never fully took off. Several Master students worked on it for a while. I like that it's now in QEMU!

repelsteeltje 2y ago

I think this is awesome. While it might seem like a small feature, it opens a huge door. It's similar to what reproducible build infrastructure has done for finding bugs, attestation that binary matches source, immutability, etc. Can imagine this is useful for finding bugs in hardware designs too.

waschl 2y ago

Tried to apply it for debugging on my own OS, but couldn’t get it finally running after several days of trial and error… https://github.com/jbreu/jos?tab=readme-ov-file#reverse-debugging-via-qemu https://github.com/jbreu/jos?tab=readme-ov-file#reverse-debu...

junon 2y ago
Yeah QEMU's story for this sort of thing is pretty rough around the edges for OS dev. Wishing there was something like Unicorn-but-with-devices for making osdev tooling.
- SoothingSorbet 2y ago
  There's also panda[1], but I never got it working myself. I share your frustration, as it would help greatly with debugging, especially with nondeterminstic bugs. I likewise never got QEMU's record/replay to work. [1] https://github.com/panda-re/panda https://github.com/panda-re/panda
b112 2y ago
Qemu is great, the DEVs and all who worked on her deserve applause, except for documentation. It's like someone creating a huge Japanese titanium indestructible fighting robot, but then using aluminum in the feet/heels. So much of my qemu work spent on randomly changing options, with no change documentation, discovering features, with no documentation, options with no reason or indication why, manpages out of date, READMEs not updated, changelog not there, etc.
- cedws 2y ago
  Agreed, the CLI in particular is a complete mess.
  - bonzini 2y ago
    If there are parts that specifically you'd want to have better documentation for, please let me know here. Generally we've been moving command line towards a scheme where each option describes an aspect of either the guest (a device, the board type, the CPU model) or the interface to the host (a file holding the contents of the disk, the network bridge to attach to, how to show graphic contents), with some options providing both as a shortcut (for example -nic, -audio, -serial).
- bonzini 2y ago
  Documentation is not great I admit. The problem is that we don't have anyone who is a capable tech writer in the team. It's not something that you can improvise. However, all incompatible changes are documented and also announced at least 8 months in advance. https://www.qemu.org/docs/master/about/removed-features.html https://www.qemu.org/docs/master/about/removed-features.html https://www.qemu.org/docs/master/about/deprecated.html https://www.qemu.org/docs/master/about/deprecated.html It may seem like there are many, but in practice they are in very old, mostly unused or very badly designed corners. For example configuration of audio was overhauled last year, and is now the same as basically all other backends (e.g. -audio pa,model=sb16; compare with -nic user,model=e1000 for a network card).
  - justinclift 2y ago

moondev 2y ago

Such a casual and low-key introduction of what sounds like an incredible new capability. 1 . Would something like this replace packer for creating machine images? 2. Curious how quickly the replay log grows and how it compares to a CoW snapshot. 3. Will be interesting what the log looks like and what doors could open up creating or generating it by other means.

owyn 2y ago
It is very cool, but I think some version of this feature has been around for years? This commit is from 7 years ago, and it looks like the code originates back to 2010. https://github.com/qemu/qemu/blob/v2.9.0/docs/replay.txt https://github.com/qemu/qemu/blob/v2.9.0/docs/replay.txt That said, I was not aware of it until I saw this post, and I definitely want to play around with it.
- Intralexical 2y ago
  Well, I guess it was probably new when the doc page was first written to introduce it. > That said, I was not aware of it until I saw this post, and I definitely want to play around with it. You could almost say it was too casual and low-key. ;)

majke 2y ago

This is a big deal. With some tooling around it can be amazing. I can think of using this for testing, and as a vehicle to change a programming paradigm of existing/legacy software (run a thing, and roll it back aggressively from outside of a vm)

m000 2y ago
Indeed, the tooling is the problem. And I wouldn't hold my breath to see this tooling being implemented, as the feature has been around for quite a bit. IMHO, PANDA [1] remains a better/more practical choice for whole-system record/replay analysis. It already offers quite a bit of tooling (including a python interface), as well as hooks to build your own. It does have its own shortcomings (speed and not being in-sync with the latest QEMU), but at least you're not limited to gdb-based debugging. [1] https://panda.re/ https://panda.re/
darby_nine 2y ago
This is the central premise of Antithesis: https://antithesis.com https://antithesis.com (no affiliation)

justinclift 2y ago

Anyone have clear ideas/guidelines for how much ram/disk/etc this is likely to need for a "reasonable" capture? Say capturing a Qt application as it corrupts its internal state during startup, in order to work out what's corrupting its internal state?

londons_explore 2y ago
I'm gonna guess not that much - easily doable on a typical desktop. If it were a problem, you can skip recording your emulated machines bootup process, and simply take a snapshot when you're about to start your QT application. That snapshot probably only takes about 10% extra RAM because most of RAM contents wont change between the snapshot and the live system.
jraph 2y ago
I don't know, however a key element is: > Record/replay system is based on saving and replaying non-deterministic events > The following non-deterministic data from peripheral devices is saved into the log: mouse and keyboard input, network packets, audio controller input, serial port input, and hardware clocks (they are non-deterministic too, because their values are taken from the host machine). Inputs from simulated hardware, memory of VM, software interrupts, and execution of instructions are not saved into the log, because they are deterministic and can be replayed by simulating the behavior of virtual machine starting from initial state. So, it's probably not much, you can probably comfortably save minutes of qemu sessions. Also note the existence of the rr debugger [1], which allows you to reverse debug applications with a ~10% performance hit while recording. To achieve this, it records results of syscalls (only). It will serialize thread events, so have the effect of running applications like on a single core CPU. [1] https://rr-project.org/ https://rr-project.org/

vessenes 2y ago

Seems to me like one of the highest and best uses of this right now would be adding verifiable builds to … literally anything. You no longer need a verifiable-build-capable compiler or language — you can just run the compile and packaging step through a deterministic QEMU session. Does this sound right? I’m trying to figure out where uncontrollable randomness would come in during a compile phase, and coming up blank.

0points 2y ago
Historically, major causes for non-determinism was embedded timestamps and unsorted file listings created by the build tools. I have not followed the progress recently, but https://reproducible-builds.org/ https://reproducible-builds.org/ is a starting point if you are interested. There is a sane path forward for reproducibility on bare metal, no custom emulation is needed.
- vessenes 2y ago
  Thanks for the link, I’m aware of reproducible-builds.org. Both your causes seem trivially fixable here - the QEMU builds could have a standard system clock time they start with, and an ‘unsorted’ file listing made in a deterministic OS environment will keep the same file order, no? By comparison the rb.org site says you need to start with stripping all that stuff out of your build process, for the reasons you refer to.
  - repelsteeltje 2y ago
    > Both your causes seem trivially fixable here [] You'd be amazed about the amount of indeterminsim lurking in the guts of depencies all the way into libc and os ... Like locale, fs
  - commercialnix 2y ago
    I like your thinking. Deterministic replay with QEMU is supplemental to the larger goal of reproducible builds. The communities concerned with the topic of reproducible software not only expect cohesive human-readable code that runs deterministically to produce binary reproducible results, but their originally stated goals require it. Deterministic replay with QEMU is a "power tool" in the larger picture of these efforts.

rrdharan 2y ago

VMware had this many years ago; it was very cool: http://stackframe.blogspot.com/2007/10/configuring-application-debugging-with.html http://stackframe.blogspot.com/2007/10/configuring-applicati... http://www.replaydebugging.com/2008/08/vmware-workstation-65-reverse-and.html http://www.replaydebugging.com/2008/08/vmware-workstation-65...

justinclift 2y ago
Any idea if that works in modern VMware Workstation? It's currently on version 17, whereas that post was for version 6.5. VMware Workstation has such disjointed development spurts that it wouldn't surprise me if the feature had been ripped out at some point. Other useful features such as machine groups have been. :(
- roca 2y ago
  It was removed in 2011.
- rrdharan 2y ago
  http://www.replaydebugging.com/2011/09/goodbye-replay-debugging.html http://www.replaydebugging.com/2011/09/goodbye-replay-debugg...
  - justinclift 2y ago
    Thanks. Yeah, I kind of expected that. :(
  - m000 2y ago
    I guess we can say it was too cool.