4 ms·
Some small-ish items: -- Virtualization of rr's required PMU features on Azure and GCP -- Virtualization of Intel's CPUID faulting on AWS and other cloud prov
by roca 6y ago
Some small-ish items:
-- Virtualization of rr's required PMU features on Azure and GCP
-- Virtualization of Intel's CPUID faulting on AWS and other cloud providers (Linux KVM has it, but it's not enabled in AWS)
-- Fix the reliability bug(s) in the retired-conditional-branches counter on AMD Ryzen so rr works there
-- Implement CPUID faulting in AMD Ryzen
-- Reliable instructions-retired counter on AMD/Intel (for more reliable/simpler operation)
-- Trap on LL/SC failures on ARM (AArch64 I guess) and other archs (so we can port rr there), with Linux kernel APIs to expose that to userspace
-- Similar traps for Intel HTM (XBEGIN/XEND) (less important now that's mostly been disabled for security reasons)
Bigger items:
-- Invest in kernel-integrated record-and-replay for Linux and other OSes
-- Implement QuickRec or some other hardware support for multicore record-replay on at least some CPU SKUs
(If anyone thinks they can help with any of that, talk to me!)
> what do you think is keeping the overhead so high in the average case described in the post (50%)?
I assume you've read https://arxiv.org/abs/1705.05937 https://arxiv.org/abs/1705.05937 ? For many workloads the major unavoidable cost is context switching. With rr's approach every regular tracee context switch (intra or inter process) turns into at least 2 inter-process context switches (tracee to rr, rr to tracee). This is bad for thread ping/pong-like behaviour. For many other workloads the cost is simply the loss of parallelism as we force everything onto a single core. For other workloads there is a high cost due to system calls that require context switches to rr because we don't currently accelerate them with "syscall buffering". The last one can be mostly engineered around, the former two are inherent to rr's approach.
Who do you work for? :-)
- glandium 6y agoI wonder if bpf could help a little with the performance.
- KenoFischer 6y agoIt's already being used :). That said, I've talked to the upstream kernel maintainers about the ability to attach a blog program to ptrace, which could possibly improve performance for the ping pong case if made sufficiently general. That's also the crux though, since sufficiently general probably means eBPF, which is still not considered safe for unprivileged use cases.
- alexeldeib 6y agoThis is my first time hearing about rr, and I'm very curious. Your comment below about increasing generality/performance with bpf is very interesting. Can you share a little bit about how some of these features (looking at the first one for example) impact rr's ability to run in cloud environments? Does it work but with performance degradation? Without performance degradation but missing some features? I don't work at quite a low enough level to look at this list and quickly grasp the high level takeaway.
- roca 6y agorr works today on cloud instances with hardware performance counters available, on Intel CPUs. For example it works on a variety of AWS instances that are large enough that your VM occupies a whole CPU socket. c5(d).9xlarge or bigger works well. The performance impact of virtualization on rr seems small on AWS, small enough that we haven't tried to measure it. On AWS rr is mostly feature complete. The only missing feature is CPUID faulting, i.e. the ability to intercept CPUID instructions and fake their results. This means that taking rr recordings from one machine and replaying them on an AWS instance with a different CPU does not work. (The other direction does work.) (Pernosco uses AWS, but we have a closed-source binary instrumentation extension to rr replay that lifts this restriction.) As I mentioned above, there's no technical reason AFAIK why AWS could not virtualize CPUID faulting; regular Linux KVM supports this.