5 ms·

Time Traveling Linux Bug Reporting: Coming in Julia 1.5

164 points by KenoFischer 6y ago

wallnuss 6y ago
I recently spend two weeks on and off hunting down a bug on a platform that didn't support `rr`. I am fairly confident to say that if I had `rr` available it would have taken me a couple of hours at most. Being able to run backwards from the point of failure and understand where a value is coming from is very powerful. Having this available in Julia directly is great, and will make it much easier to get bug-reports from users.
roca 6y ago
This is very cool stuff. Keno is a great rr contributor, and I'm grateful for his Pernosco shout-out :-). This sort of capability has always been part of my vision for rr/Pernosco. I want to see more projects like this doing record-and-replay capture of bugs in the field. With more projects depending on this sort of infrastructure, we can get more attention from OS/hardware people to make the record-and-replay systems better, and then I think we can reach a tipping point where it becomes best practice. Eventually we should be looking back and wondering why, as an industry, we didn't go in this direction earlier.
- KenoFischer 6y ago
  No, thank you for all your work on this :). Building something like rr is only possible with some really solid engineering and I think you managed to accomplish that. I think we're aligned on the vision here. I bring up rr every time I speak to hardware people, but of course the first question they ask is "how many people need this right now?". With this I'm hoping to be able to increase the answer to that by an order of magnitude or so ;).
- Veserv 6y ago
  If you don't mind me asking, you mention that you want more attention from OS/hardware people. What sort of things are you looking for from OS/hardware people that would make replay systems better? As for a more technical question on implementation. Assuming the post accurately represents the performance characteristics of rr, what do you think is keeping the overhead so high in the average case described in the post (50%)? Disclaimer: I work for a company that sells similar technology.
  - KenoFischer 6y ago
    I have done some experiments and it's reasonably possible to cut the average overhead to about 20% or so with known techniques. That said, I'm surprised you'd consider this high. I know of no system, including commercial ones that can even get close to these kind of low overhead numbers. Heck, I don't even know of one (other than rr) that actually works on applications of this complexity. If you have one, you should advertise it significantly more heavily ;).
    - Veserv 6y ago
      Well, if you use hardware-assist you can get 0% overhead with shared memory and without serializing cores. Unfortunately, only some chips expose the correct hardware. In terms of other publicly available systems, UndoDB claims properties similar to rr and appears to use the same general approach based on talks I have seen from the creator, but I have no first hand information on its actual properties. If only advertising would help more. Unfortunately for all companies in this business, few companies are willing to invest even a tiny amount in developer productivity let alone properly value it. For instance, do you know of any company that spends more than $20k/year per developer on tools to enhance software developer productivity? To go even further, how many companies would even consider spending the astronomical sum of $20k/year per developer on tools? Yet, a fully burdened developer is ~$200k/year, so, using the most basic business analysis of just cost savings (which undervalues to a mind-boggling degree), $20k/year is a mere 10% cost increase, but for some reason it is viewed as an astronomical and ridiculous price. In other sectors, businesses easily spend significant percentages of salary cost in tooling. EDA tools for EEs can run $50k/year, trucks for truck drivers amortize out to $30-50k/year, etc. Obviously, you need to justify a 10% cost increase with an appropriate return, but the fact that few companies even bother to do so is in my opinion the primary limiting factor on increased adoption of the techniques and technologies we are discussing since nobody values them.
      - roca 6y ago
        What hardware supports shared memory record-and-replay in the presence of data races? I'm not aware of any widely available hardware that does this. UndoDB's design is similar to rr but they use binary instrumentation at record time, so it's higher overhead than rr and a more complex implementation in some ways, but they don't depend on performance counters so they work on more architectures (AMD, ARM) and in any virtual guest. It's certainly true that it's hard to get people to pay for debugging tools. On the other hand, some big-name companies spend tons of money on internal tooling, so at least some companies are willing to spend on developer productivity in general. I think certain classes of tools have "traditionally" been free and that mindset is hard to change, and the majority of companies are reluctant to spend money on developer productivity in general. I don't have great ideas to tackle this other than "keep making tools better and better until the wall cracks". If you want to talk more off the record, feel free to contact me.
KenoFischer 6y ago
Hi all, I'm pretty excited to get this out there for you. I had published this this morning, so I could be around through the day to answer any questions, but things were a bit delayed by the HN algorithm gods (- thanks dang for rescuing it ;) ). That said, I'll check in periodically for the next hour or two if there's any questions I can answer.
tgflynn 6y ago
I'm surprised I haven't heard of rr before, it sounds like it could be a game changer for debugging many types of problems. How long has this project existed/been usable ? Am I correct in understanding that rr can be used with any application (ie. the application doesn't have to be built specifically to support it) ? That's the impression the usage introduction on the website gives: https://rr-project.org/ https://rr-project.org/.
- KenoFischer 6y ago
  > How long has this project existed/been usable ? It's been our (people who work on Julia) daily debugger since 2015 or so > Am I correct in understanding that rr can be used with any application Yes
boromi 6y ago
is this coming to Windows or Mac?

164 points by KenoFischer 6y ago

I recently spend two weeks on and off hunting down a bug on a platform that didn't support `rr`. I am fairly confident to say that if I had `rr` available it would have taken me a couple of hours at most. Being able to run backwards from the point of failure and understand where a value is coming from is very powerful. Having this available in Julia directly is great, and will make it much easier to get bug-reports from users.

roca 6y ago

This is very cool stuff. Keno is a great rr contributor, and I'm grateful for his Pernosco shout-out :-). This sort of capability has always been part of my vision for rr/Pernosco. I want to see more projects like this doing record-and-replay capture of bugs in the field. With more projects depending on this sort of infrastructure, we can get more attention from OS/hardware people to make the record-and-replay systems better, and then I think we can reach a tipping point where it becomes best practice. Eventually we should be looking back and wondering why, as an industry, we didn't go in this direction earlier.

KenoFischer 6y ago
No, thank you for all your work on this :). Building something like rr is only possible with some really solid engineering and I think you managed to accomplish that. I think we're aligned on the vision here. I bring up rr every time I speak to hardware people, but of course the first question they ask is "how many people need this right now?". With this I'm hoping to be able to increase the answer to that by an order of magnitude or so ;).
Veserv 6y ago
If you don't mind me asking, you mention that you want more attention from OS/hardware people. What sort of things are you looking for from OS/hardware people that would make replay systems better? As for a more technical question on implementation. Assuming the post accurately represents the performance characteristics of rr, what do you think is keeping the overhead so high in the average case described in the post (50%)? Disclaimer: I work for a company that sells similar technology.
- KenoFischer 6y ago
  I have done some experiments and it's reasonably possible to cut the average overhead to about 20% or so with known techniques. That said, I'm surprised you'd consider this high. I know of no system, including commercial ones that can even get close to these kind of low overhead numbers. Heck, I don't even know of one (other than rr) that actually works on applications of this complexity. If you have one, you should advertise it significantly more heavily ;).
  - Veserv 6y ago
    Well, if you use hardware-assist you can get 0% overhead with shared memory and without serializing cores. Unfortunately, only some chips expose the correct hardware. In terms of other publicly available systems, UndoDB claims properties similar to rr and appears to use the same general approach based on talks I have seen from the creator, but I have no first hand information on its actual properties. If only advertising would help more. Unfortunately for all companies in this business, few companies are willing to invest even a tiny amount in developer productivity let alone properly value it. For instance, do you know of any company that spends more than $20k/year per developer on tools to enhance software developer productivity? To go even further, how many companies would even consider spending the astronomical sum of $20k/year per developer on tools? Yet, a fully burdened developer is ~$200k/year, so, using the most basic business analysis of just cost savings (which undervalues to a mind-boggling degree), $20k/year is a mere 10% cost increase, but for some reason it is viewed as an astronomical and ridiculous price. In other sectors, businesses easily spend significant percentages of salary cost in tooling. EDA tools for EEs can run $50k/year, trucks for truck drivers amortize out to $30-50k/year, etc. Obviously, you need to justify a 10% cost increase with an appropriate return, but the fact that few companies even bother to do so is in my opinion the primary limiting factor on increased adoption of the techniques and technologies we are discussing since nobody values them.
    - roca 6y ago
      What hardware supports shared memory record-and-replay in the presence of data races? I'm not aware of any widely available hardware that does this. UndoDB's design is similar to rr but they use binary instrumentation at record time, so it's higher overhead than rr and a more complex implementation in some ways, but they don't depend on performance counters so they work on more architectures (AMD, ARM) and in any virtual guest. It's certainly true that it's hard to get people to pay for debugging tools. On the other hand, some big-name companies spend tons of money on internal tooling, so at least some companies are willing to spend on developer productivity in general. I think certain classes of tools have "traditionally" been free and that mindset is hard to change, and the majority of companies are reluctant to spend money on developer productivity in general. I don't have great ideas to tackle this other than "keep making tools better and better until the wall cracks". If you want to talk more off the record, feel free to contact me.

KenoFischer 6y ago

Hi all, I'm pretty excited to get this out there for you. I had published this this morning, so I could be around through the day to answer any questions, but things were a bit delayed by the HN algorithm gods (- thanks dang for rescuing it ;) ). That said, I'll check in periodically for the next hour or two if there's any questions I can answer.

tgflynn 6y ago

I'm surprised I haven't heard of rr before, it sounds like it could be a game changer for debugging many types of problems. How long has this project existed/been usable ? Am I correct in understanding that rr can be used with any application (ie. the application doesn't have to be built specifically to support it) ? That's the impression the usage introduction on the website gives: https://rr-project.org/ https://rr-project.org/.

KenoFischer 6y ago
> How long has this project existed/been usable ? It's been our (people who work on Julia) daily debugger since 2015 or so > Am I correct in understanding that rr can be used with any application Yes

boromi 6y ago

is this coming to Windows or Mac?