3 ms·

The assumption here is that you're modifying the hypervisor itself. No-one expects to be able to bolt a record-replay tool onto a third-party hypervisor.

by roca 2y ago

Veserv 2y ago
Well, that is what you did with rr (bolted it onto a third-party unmodified OS) so I thought I would cover all the bases. Once you are in modification land it really depends on the kind of hypervisor and its implementation. Like doing it for a OS, a well-designed, consistent, and tight interface makes it pretty easy. But, the unconstrained multithreading for multicore VMs makes it problematic for record-replay solutions if you do not want to serialize.
- mark_undoio 2y ago
  Yes, this is true. Genuine parallelism will absolutely be a problem there - similarly to how it is for user-level record/replay tech. And serialising is going to have more impact when you're recording a whole OS vs just one application. Microsoft's TTD (a user-level tech) does allow genuine parallelism, I believe. But my understanding is that it has slower (though still very impressive) single-threaded performance as a result, so there's a tradeoff. But perhaps you could do something more like that. We believe it's possible to handle parallelism in record/replay tech while still having good single-threaded performance - but it implies a much more complex implementation to do so safely.
  - Veserv 2y ago
    The literature on Microsoft TTD [1][2] indicates it is a instruction emulator full-trace approach. It is a little unclear if it is a basic emulator or a instrumented JIT approach, but the reported ~10x overhead is more consistent with a basic emulator. Though maybe their recording implementation is just inefficient causing such excess overhead. [1] https://msrc.microsoft.com/blog/2019/05/time-travel-debugging-its-a-blast-from-the-past/ https://msrc.microsoft.com/blog/2019/05/time-travel-debuggin... [2] https://www.usenix.org/legacy/events/vee06/full_papers/p154-bhansali.pdf https://www.usenix.org/legacy/events/vee06/full_papers/p154-...
- roca 2y ago
  The Linux kernel supports pretty powerful APIs for inspecting and managing processes, like ptrace and seccomp and /proc, from another userspace process. There is no comparable set of APIs for hypervisors, unless Xen has something maybe? And we don't handle unrestricted multithreading in rr. If we wanted to it would complicate things a lot and I don't know if existing kernel APIs would be adequate. We're skating along the edge of feasibility as it is.

The assumption here is that you're modifying the hypervisor itself. No-one expects to be able to bolt a record-replay tool onto a third-party hypervisor.

by roca 2y ago

The assumption here is that you're modifying the hypervisor itself. No-one expects to be able to bolt a record-replay tool onto a third-party hypervisor.

Veserv 2y ago

Well, that is what you did with rr (bolted it onto a third-party unmodified OS) so I thought I would cover all the bases. Once you are in modification land it really depends on the kind of hypervisor and its implementation. Like doing it for a OS, a well-designed, consistent, and tight interface makes it pretty easy. But, the unconstrained multithreading for multicore VMs makes it problematic for record-replay solutions if you do not want to serialize.

mark_undoio 2y ago
Yes, this is true. Genuine parallelism will absolutely be a problem there - similarly to how it is for user-level record/replay tech. And serialising is going to have more impact when you're recording a whole OS vs just one application. Microsoft's TTD (a user-level tech) does allow genuine parallelism, I believe. But my understanding is that it has slower (though still very impressive) single-threaded performance as a result, so there's a tradeoff. But perhaps you could do something more like that. We believe it's possible to handle parallelism in record/replay tech while still having good single-threaded performance - but it implies a much more complex implementation to do so safely.
- Veserv 2y ago
  The literature on Microsoft TTD [1][2] indicates it is a instruction emulator full-trace approach. It is a little unclear if it is a basic emulator or a instrumented JIT approach, but the reported ~10x overhead is more consistent with a basic emulator. Though maybe their recording implementation is just inefficient causing such excess overhead. [1] https://msrc.microsoft.com/blog/2019/05/time-travel-debugging-its-a-blast-from-the-past/ https://msrc.microsoft.com/blog/2019/05/time-travel-debuggin... [2] https://www.usenix.org/legacy/events/vee06/full_papers/p154-bhansali.pdf https://www.usenix.org/legacy/events/vee06/full_papers/p154-...
roca 2y ago
The Linux kernel supports pretty powerful APIs for inspecting and managing processes, like ptrace and seccomp and /proc, from another userspace process. There is no comparable set of APIs for hypervisors, unless Xen has something maybe? And we don't handle unrestricted multithreading in rr. If we wanted to it would complicate things a lot and I don't know if existing kernel APIs would be adequate. We're skating along the edge of feasibility as it is.