4 ms·

I have done some experiments and it's reasonably possible to cut the average overhead to about 20% or so with known techniques. That said, I'm surprised you'd c

I have done some experiments and it's reasonably possible to cut the average overhead to about 20% or so with known techniques. That said, I'm surprised you'd consider this high. I know of no system, including commercial ones that can even get close to these kind of low overhead numbers. Heck, I don't even know of one (other than rr) that actually works on applications of this complexity. If you have one, you should advertise it significantly more heavily ;).

Veserv 6y ago
Well, if you use hardware-assist you can get 0% overhead with shared memory and without serializing cores. Unfortunately, only some chips expose the correct hardware. In terms of other publicly available systems, UndoDB claims properties similar to rr and appears to use the same general approach based on talks I have seen from the creator, but I have no first hand information on its actual properties. If only advertising would help more. Unfortunately for all companies in this business, few companies are willing to invest even a tiny amount in developer productivity let alone properly value it. For instance, do you know of any company that spends more than $20k/year per developer on tools to enhance software developer productivity? To go even further, how many companies would even consider spending the astronomical sum of $20k/year per developer on tools? Yet, a fully burdened developer is ~$200k/year, so, using the most basic business analysis of just cost savings (which undervalues to a mind-boggling degree), $20k/year is a mere 10% cost increase, but for some reason it is viewed as an astronomical and ridiculous price. In other sectors, businesses easily spend significant percentages of salary cost in tooling. EDA tools for EEs can run $50k/year, trucks for truck drivers amortize out to $30-50k/year, etc. Obviously, you need to justify a 10% cost increase with an appropriate return, but the fact that few companies even bother to do so is in my opinion the primary limiting factor on increased adoption of the techniques and technologies we are discussing since nobody values them.
- roca 6y ago
  What hardware supports shared memory record-and-replay in the presence of data races? I'm not aware of any widely available hardware that does this. UndoDB's design is similar to rr but they use binary instrumentation at record time, so it's higher overhead than rr and a more complex implementation in some ways, but they don't depend on performance counters so they work on more architectures (AMD, ARM) and in any virtual guest. It's certainly true that it's hard to get people to pay for debugging tools. On the other hand, some big-name companies spend tons of money on internal tooling, so at least some companies are willing to spend on developer productivity in general. I think certain classes of tools have "traditionally" been free and that mindset is hard to change, and the majority of companies are reluctant to spend money on developer productivity in general. I don't have great ideas to tackle this other than "keep making tools better and better until the wall cracks". If you want to talk more off the record, feel free to contact me.
  - Veserv 6y ago
    An example is ARM systems including an ETM component that supports data tracing. You connect an external trace probe, usually via JTAG, to receive the trace events. This technique usually uses read/write and control-flow state for reconstruction. This has the advantage of not requiring a recording from the beginning to reconstruct since you can correctly work backward from the final state, but requires higher data bandwidth and storage.
    - roca 6y ago
      OK right. That isn't very scalable however since you're limited by the size and bandwidth of the memory buffer holding the recorded memory values.
touisteur 6y ago
Intel PT has really low overhead, from my experiments. The difficulty then is the size of the trace file and your ability to stream it to disk or network. There is support for it in core dumps now and gdb can do some time travel if it is present.
- KenoFischer 6y ago
  The 50% is average overhead and basically entirely due to context switch overhead and execution serialization. If the execution is entirely in userspace, rr's overhead is basically 0. I don't think PT will help here. That said, rr's performance on Intel chips is entirely acceptable for single threaded code. The big asks would be other architectures, or as roc mentioned, something like QuickRec for efficient multi-threaded recording.
- roca 6y ago
  PT doesn't get you close to what rr can do, which is reconstruct any program state at any time in the recorded execution. As Keno says in his blog post, the promise of rr is that if you have an rr recording, you are almost completely assured of having enough information to figure out the bug. That is what we see in practice, and it has big implications for developer workflow.

Veserv 6y ago

Well, if you use hardware-assist you can get 0% overhead with shared memory and without serializing cores. Unfortunately, only some chips expose the correct hardware. In terms of other publicly available systems, UndoDB claims properties similar to rr and appears to use the same general approach based on talks I have seen from the creator, but I have no first hand information on its actual properties. If only advertising would help more. Unfortunately for all companies in this business, few companies are willing to invest even a tiny amount in developer productivity let alone properly value it. For instance, do you know of any company that spends more than $20k/year per developer on tools to enhance software developer productivity? To go even further, how many companies would even consider spending the astronomical sum of $20k/year per developer on tools? Yet, a fully burdened developer is ~$200k/year, so, using the most basic business analysis of just cost savings (which undervalues to a mind-boggling degree), $20k/year is a mere 10% cost increase, but for some reason it is viewed as an astronomical and ridiculous price. In other sectors, businesses easily spend significant percentages of salary cost in tooling. EDA tools for EEs can run $50k/year, trucks for truck drivers amortize out to $30-50k/year, etc. Obviously, you need to justify a 10% cost increase with an appropriate return, but the fact that few companies even bother to do so is in my opinion the primary limiting factor on increased adoption of the techniques and technologies we are discussing since nobody values them.

roca 6y ago
What hardware supports shared memory record-and-replay in the presence of data races? I'm not aware of any widely available hardware that does this. UndoDB's design is similar to rr but they use binary instrumentation at record time, so it's higher overhead than rr and a more complex implementation in some ways, but they don't depend on performance counters so they work on more architectures (AMD, ARM) and in any virtual guest. It's certainly true that it's hard to get people to pay for debugging tools. On the other hand, some big-name companies spend tons of money on internal tooling, so at least some companies are willing to spend on developer productivity in general. I think certain classes of tools have "traditionally" been free and that mindset is hard to change, and the majority of companies are reluctant to spend money on developer productivity in general. I don't have great ideas to tackle this other than "keep making tools better and better until the wall cracks". If you want to talk more off the record, feel free to contact me.
- Veserv 6y ago
  An example is ARM systems including an ETM component that supports data tracing. You connect an external trace probe, usually via JTAG, to receive the trace events. This technique usually uses read/write and control-flow state for reconstruction. This has the advantage of not requiring a recording from the beginning to reconstruct since you can correctly work backward from the final state, but requires higher data bandwidth and storage.
  - roca 6y ago
    OK right. That isn't very scalable however since you're limited by the size and bandwidth of the memory buffer holding the recorded memory values.

touisteur 6y ago

Intel PT has really low overhead, from my experiments. The difficulty then is the size of the trace file and your ability to stream it to disk or network. There is support for it in core dumps now and gdb can do some time travel if it is present.

KenoFischer 6y ago
The 50% is average overhead and basically entirely due to context switch overhead and execution serialization. If the execution is entirely in userspace, rr's overhead is basically 0. I don't think PT will help here. That said, rr's performance on Intel chips is entirely acceptable for single threaded code. The big asks would be other architectures, or as roc mentioned, something like QuickRec for efficient multi-threaded recording.
roca 6y ago
PT doesn't get you close to what rr can do, which is reconstruct any program state at any time in the recorded execution. As Keno says in his blog post, the promise of rr is that if you have an rr recording, you are almost completely assured of having enough information to figure out the bug. That is what we see in practice, and it has big implications for developer workflow.