3 ms·
We use rr extensively for Julia development [1] (for a sense of scale, I probably look at hundreds of rr traces a month). There's two big challenges for us ther
by KenoFischer 6y ago
We use rr extensively for Julia development [1] (for a sense of scale, I probably look at hundreds of rr traces a month). There's two big challenges for us there
1) Many cloud environments are broken in various ways for rr recording - I spend way too much time guiding our users to the few that do work
2) The GDB frontend really isn't the best way to view all the underlying information
Pernosco is well on its way to address the second part of this. Its replayer is quite a bit nicer than the open source rr one and I have high hopes for the platform in general.
For the first one, all I can do is plead with the cloud vendors not to mess up the environment. Since this is HN, if you're working for a cloud provider, please ask your engineering team not to break rr. Otherwise I'll have to spend time migrating our users off your platform, which will be fun for neither of us.
Anyway, congrats to Robert and Kyle on the launch. Wish you all the best.
[1] https://julialang.org/blog/2020/05/rr/ https://julialang.org/blog/2020/05/rr/
- solarengineer 6y agoCould you share examples of how cloud providers break rr?
- roca 6y agoUsually they break rr by either blocking access to the hardware PMU entirely (e.g. on AWS, any instance that shares a CPU socket with other instances; on Azure/GCP, everywhere AFAIK), or by introducing subtle errors into the virtualized performance counter values that rr depends on (e.g. VMWare: https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.htm..., Xen: https://lists.xen.org/archives/html/xen-devel/2017-07/msg02242.html https://lists.xen.org/archives/html/xen-devel/2017-07/msg022...). Another feature rr benefits from is CPUID faulting, which is not virtualized by any of the major cloud providers (though Linux KVM does virtualize it): https://patchwork.kernel.org/project/kvm/patch/20161101013749.4003-8-khuey@kylehuey.com/ https://patchwork.kernel.org/project/kvm/patch/2016110101374.... That'd be "nice to have" but we can live without it.
- mwcampbell 6y agoCan you be more specific about which cloud environments do work with rr? In your blog post you said it works on Amazon EC2 instances above a certain size. Which instance types are known to work?
- roca 6y agoThe main thing is that rr recording needs access to the PMU. So generally AWS Nitro instances which occupy a whole CPU socket (or more than one) work. In particular AWS c5(d).9xlarge and larger are known to work. Other instance types (including metal instances) should work too.