6 ms·

Debugging operating systems with time-traveling virtual machines (2005) [pdf]

145 points by Intralexical 2y ago

Intralexical 2y ago
[flagged]
- ForOldHack 2y ago
  Related work: IBM first mainframes were so unreliable, that they put a feature in that you could restore the machine to the instruction that crashed, and continue processing, and you could back out of the last few instructions. State of the art for the late 1950s. It was not virtual, that would become standard through on less than a decade. Optimizing çopilers came with source tapes, and a huge expensive.
- deleted 2y ago
  [deleted]
drewg123 2y ago
This needs a [2005] qualifier
Veserv 2y ago
A history of other time traveling debugging papers and products (including this one): https://jakob.engbloms.se/archives/1554 https://jakob.engbloms.se/archives/1554 https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564
- deleted 2y ago
  [deleted]
fatcunt 2y ago
Microsoft has, or had, a similar technology they use internally, called TKO: https://www.microsoft.com/security/blog/2020/05/04/mitigating-vulnerabilities-endpoint-network-stacks https://www.microsoft.com/security/blog/2020/05/04/mitigatin... It's written in Rust and is based around a version of Bochs modified for deterministic execution. It's got time-travel debugging (with WinDbg), which works by replaying forward from the nearest snapshot to the point at which the user is asking to move backwards to. The primary author of this software wanted to open source it, but the higher-ups at MSFT refused. He's been working on similar projects in a personal capacity though, e.g. https://gamozolabs.github.io/fuzzing/2020/12/06/fuzzos.html https://gamozolabs.github.io/fuzzing/2020/12/06/fuzzos.html
- Cyph0n 2y ago
  Watched some of his streams a while back. One of the most talented/productive devs I have ever seen tbh.
- grepfru_it 2y ago
  The trick at Microsoft is to start working on your project in your spare time. Then incorporate it into your project at MSFT. You get the clout associated with having an open source project but then you also get to use it internally as a sanctioned tool
  - DaiPlusPlus 2y ago
    > The trick at Microsoft is to start working on your project in your spare time. Then incorporate it into your project at MSFT. (Ex-msftie here) Microsoft is great for being amongst the tiny number of software companies that not only allow their FTEs to have their own software projects without attempting to claim any kind of ownership (compare with Apple where I’m told having even a public GitHub account without manager-approval is grounds for termination…) - but MS even actively encourages it too (though I cynically note that was when the Windows Phone App Store’s app-shortage was thought-of as a serious problem) BUUUUT - they do require you to strictly avoid cross-over with their own products: you can’t just re-use your own code in a company project, or even reimplement it from-memory. Yes, there are ways of doing this properly (involving weeks of meetings with LCA arranging a permanent irrevocable - and possibly exclusive - rights transfer), wasting time better spent on other things) - so your project had better be something special and you’d need to give-up hope of getting rich from it (spare a very modest bonus the year you do the LCA paperwork if your skiplevel sees any value in it).
    - pjmlp 2y ago
      Yeah, and that is how stuff like C++/CX gets killed, replaced by a side project that was never at the same tooling level in Visual Studio, even though there was a CppCon talk promising otherwise, and currently is in maintenance state because "it achived its goals", who cares about those paying customers that had to rewrite their code to a lesser capable tool, now abandonend.
roca 2y ago
I wonder if this inspired the VMWare VM record-and-replay functionality that came out in 2008. They discontinued it in 2011, but it's important to me because we used it at Mozilla to great effect and that made it easier for me to get Mozilla to support the development of rr, which started in 2011.
- icholy 2y ago
  I don't get how rr isn't more popular.
- soamv 2y ago
  It might have, I remember attending a talk by Peter Chen when I was at VMware around that time, and I know there was some kind of collaboration. (I wasn't involved in record-replay at VMware, but I was interested in it due to some ancient work I did with userspace record-replay debugging, https://lizard.sf.net https://lizard.sf.net). rr is fantastic work, mad props! And the multiprocess stuff in pernosco looks super neat.
- purple-dragon 2y ago
  Yes. There’s a whole family of papers by Chen, his students, and other VMware folks in this vein.
- gwd 2y ago
  Yes, at some point when I was a student I went out to VMWare and gave a presentation to a team of their engineers. The thing about execution replay is that, in the general case, it's incredibly fragile. I happened to chat to some people from VMWare in 2008 and they said that there was a team of 10 people whose only job it was to fix execution replay when it broke. The main thing I learned from my PhD on execution replay was how to debug insane bugs. Presumably rr, focusing on processes, offers a more constrained environment that's easier to log & replay.
  - mark_undoio 2y ago
    > Presumably rr, focusing on processes, offers a more constrained environment that's easier to log & replay. As someone who's worked elsewhere on time travel debug, I'm really curious on @roca's take on this - because I'd have expected a full-VM solution to be easier to make reliable. Hardware-level behaviour sounds harder but it's well-constrained. The behaviour an OS can rely on from the hardware is large but well-documented and slow to change. In contrast, the process boundary is really ill-defined and permeable. Also, when you need things to be precisely the same, you notice bits of kernel behaviour leaking into the user space ABI in unexpected ways.
userbinator 2y ago
Debuggers have had history tracing functionality for a long time, but being extremely slow and consuming a lot of storage meant it was rarely used except for very specific cases. Now that CPUs are faster and the average machine has a lot more RAM, it becomes more feasible to do this.

145 points by Intralexical 2y ago

Intralexical 2y ago

[flagged]

ForOldHack 2y ago
Related work: IBM first mainframes were so unreliable, that they put a feature in that you could restore the machine to the instruction that crashed, and continue processing, and you could back out of the last few instructions. State of the art for the late 1950s. It was not virtual, that would become standard through on less than a decade. Optimizing çopilers came with source tapes, and a huge expensive.
deleted 2y ago
[deleted]

drewg123 2y ago

This needs a [2005] qualifier

Veserv 2y ago

A history of other time traveling debugging papers and products (including this one): https://jakob.engbloms.se/archives/1554 https://jakob.engbloms.se/archives/1554 https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564

deleted 2y ago
[deleted]

fatcunt 2y ago

Microsoft has, or had, a similar technology they use internally, called TKO: https://www.microsoft.com/security/blog/2020/05/04/mitigating-vulnerabilities-endpoint-network-stacks https://www.microsoft.com/security/blog/2020/05/04/mitigatin... It's written in Rust and is based around a version of Bochs modified for deterministic execution. It's got time-travel debugging (with WinDbg), which works by replaying forward from the nearest snapshot to the point at which the user is asking to move backwards to. The primary author of this software wanted to open source it, but the higher-ups at MSFT refused. He's been working on similar projects in a personal capacity though, e.g. https://gamozolabs.github.io/fuzzing/2020/12/06/fuzzos.html https://gamozolabs.github.io/fuzzing/2020/12/06/fuzzos.html

Cyph0n 2y ago
Watched some of his streams a while back. One of the most talented/productive devs I have ever seen tbh.
grepfru_it 2y ago
The trick at Microsoft is to start working on your project in your spare time. Then incorporate it into your project at MSFT. You get the clout associated with having an open source project but then you also get to use it internally as a sanctioned tool
- DaiPlusPlus 2y ago
  > The trick at Microsoft is to start working on your project in your spare time. Then incorporate it into your project at MSFT. (Ex-msftie here) Microsoft is great for being amongst the tiny number of software companies that not only allow their FTEs to have their own software projects without attempting to claim any kind of ownership (compare with Apple where I’m told having even a public GitHub account without manager-approval is grounds for termination…) - but MS even actively encourages it too (though I cynically note that was when the Windows Phone App Store’s app-shortage was thought-of as a serious problem) BUUUUT - they do require you to strictly avoid cross-over with their own products: you can’t just re-use your own code in a company project, or even reimplement it from-memory. Yes, there are ways of doing this properly (involving weeks of meetings with LCA arranging a permanent irrevocable - and possibly exclusive - rights transfer), wasting time better spent on other things) - so your project had better be something special and you’d need to give-up hope of getting rich from it (spare a very modest bonus the year you do the LCA paperwork if your skiplevel sees any value in it).
  - pjmlp 2y ago
    Yeah, and that is how stuff like C++/CX gets killed, replaced by a side project that was never at the same tooling level in Visual Studio, even though there was a CppCon talk promising otherwise, and currently is in maintenance state because "it achived its goals", who cares about those paying customers that had to rewrite their code to a lesser capable tool, now abandonend.

roca 2y ago

I wonder if this inspired the VMWare VM record-and-replay functionality that came out in 2008. They discontinued it in 2011, but it's important to me because we used it at Mozilla to great effect and that made it easier for me to get Mozilla to support the development of rr, which started in 2011.

icholy 2y ago
I don't get how rr isn't more popular.
soamv 2y ago
It might have, I remember attending a talk by Peter Chen when I was at VMware around that time, and I know there was some kind of collaboration. (I wasn't involved in record-replay at VMware, but I was interested in it due to some ancient work I did with userspace record-replay debugging, https://lizard.sf.net https://lizard.sf.net). rr is fantastic work, mad props! And the multiprocess stuff in pernosco looks super neat.
purple-dragon 2y ago
Yes. There’s a whole family of papers by Chen, his students, and other VMware folks in this vein.
gwd 2y ago
Yes, at some point when I was a student I went out to VMWare and gave a presentation to a team of their engineers. The thing about execution replay is that, in the general case, it's incredibly fragile. I happened to chat to some people from VMWare in 2008 and they said that there was a team of 10 people whose only job it was to fix execution replay when it broke. The main thing I learned from my PhD on execution replay was how to debug insane bugs. Presumably rr, focusing on processes, offers a more constrained environment that's easier to log & replay.
- mark_undoio 2y ago
  > Presumably rr, focusing on processes, offers a more constrained environment that's easier to log & replay. As someone who's worked elsewhere on time travel debug, I'm really curious on @roca's take on this - because I'd have expected a full-VM solution to be easier to make reliable. Hardware-level behaviour sounds harder but it's well-constrained. The behaviour an OS can rely on from the hardware is large but well-documented and slow to change. In contrast, the process boundary is really ill-defined and permeable. Also, when you need things to be precisely the same, you notice bits of kernel behaviour leaking into the user space ABI in unexpected ways.

userbinator 2y ago

Debuggers have had history tracing functionality for a long time, but being extremely slow and consuming a lot of storage meant it was rarely used except for very specific cases. Now that CPUs are faster and the average machine has a lot more RAM, it becomes more feasible to do this.