4 ms·

Reverse Debugging with GDB

76 points by mrmaddog 13y ago

martin_ 13y ago
I'm assuming this wasn't a big hit as little as been mentioned about it since its release in 2009?
- jlebar 13y ago
  IME it's not only slow, but also very memory-hungry. As a result it's hard to use it to debug heisenbugs, because you have to have a decent idea of where the bug is, so you can start recording at that point. But of course if you know even roughly where the bug was, debugging isn't nearly as hard... VMWare had record-and-replay debugging functionality which was functionally similar to gdb's backwards debugging, although its implementation was a lot different, which allowed it to scale to recording much longer threads of execution. We used this at Mozilla to find a number of super-tricky bugs; it was pretty awesome. Unfortunately this feature was nixed in vmware workstation 8 [1]. [1] http://www.replaydebugging.com/ http://www.replaydebugging.com/
  - martin_ 13y ago
    That's very interesting! Why did they remove that feature if it was so awesome?
  - khuey 13y ago
    We still have it working on a box in the office here. I used it last week ;-)
neur0mancer 13y ago
I tried it. Unfortunately, it is very slow.
- adamnemecek 13y ago
  It is. But that one time when you will be dealing with a Heisenbug, you will be thankful for it.
  - ginko 13y ago
    The thing with Heisenbugs is that they usually don't happen when you attach a debugger.
    - adamnemecek 13y ago
      That kind of depends. They can also happen not because you attached a debugger but because you set a break point in a particularly unfortunate location.
      - RobotCaleb 13y ago
        Oh man. I spent a couple of days tracking one of those down. When the light went on in my head I felt like such an idiot.
  - davidgerard 13y ago
    This.
- tintor 13y ago
  GDB reverse debugger is very slow as it is saving the execution state after every instruction executed. There is a commercial plugin for GDB reverse debugger: UndoDB. It is much faster than just GDB (they claim only 1.7x slowdown), but unfortunately not very stable for the product I'm working on.
  - gregthelaw 13y ago
    I'm a co-founder of the company. Lots of stability improvements over the past year or so: I'd urge you to try it again. Particularly v4 beta. Send email to beta@undo-software.com to get a beta license. I'd never claim it's perfect: it's software after all, and the fact most software has bugs is why we have a business in the first place ;) But we're used by some pretty hardcore folk these days, including Cadence, NASA, and some of the biggest banks in the world. And the PyPy guys seem pretty happy :) Oh, and of course, if you do find issues, please do report them to support@undo-software.com -- then we can fix them.
coldcode 13y ago
Reverse debugging sounds like putting bugs in. We do that every day already.
Groxx 13y ago
For Java, if it still runs: http://www.lambdacs.com/debugger/ http://www.lambdacs.com/debugger/ I really like the idea. Never knew GDB had implemented it (thanks for the link!). I'll definitely try this out :)
bravura 13y ago
I have a more modest request: Could debuggers please store the code paths and variable values, over the last 50 executions or perhaps the test suite (deterministic)? This is particularly useful for understanding someone else's code. [edit: a context-free version of reverse debugging] I've recently been enjoying using IntelliJ for Scala, because it can show me types, and jump to type declarations, and usages. However, I still have to do a bunch of print debugging when trying to understand someone else's code, because I want to understand what values certain variables take. This is a pain using a compiler as slow as Scala. Something that captured relevant information, so that it would be easier + faster to do print debugging against previous code runs or the test suite, would be awesome.
- jbert 13y ago
  It looks like the underlying feature which enables reverse debugging, Process Recording, could do what you describe: https://sourceware.org/gdb/wiki/ProcessRecord https://sourceware.org/gdb/wiki/ProcessRecord the data may only be in-process, if so it would need to be save-able and load-able.
- j_s 13y ago
  Microsoft sells this with the 'Ultimate' SKU of Visual Studio: http://msdn.microsoft.com/en-us/library/dd264915.aspx http://msdn.microsoft.com/en-us/library/dd264915.aspx
fijal 13y ago
We (PyPy developers) tried this and it's unusably slow. I don't know how they got that slow, but it takes a bit forever to execute 10mln instructions. We ended up using UndoDB gdb wrapper, which runs slower than they advertise (+- the same speed as valgrind so say 30x give or take), but is a lot more usable. They had stability problems in the past, but with the recent versions, we're able to run stuff like complex python programs on top of pypy on top of UndoDB, which is very impressive. Very useful for hard GC bugs.
peterhajas 13y ago
Cool! Does `lldb` have similar functionality?

76 points by mrmaddog 13y ago

martin_ 13y ago

I'm assuming this wasn't a big hit as little as been mentioned about it since its release in 2009?

jlebar 13y ago
IME it's not only slow, but also very memory-hungry. As a result it's hard to use it to debug heisenbugs, because you have to have a decent idea of where the bug is, so you can start recording at that point. But of course if you know even roughly where the bug was, debugging isn't nearly as hard... VMWare had record-and-replay debugging functionality which was functionally similar to gdb's backwards debugging, although its implementation was a lot different, which allowed it to scale to recording much longer threads of execution. We used this at Mozilla to find a number of super-tricky bugs; it was pretty awesome. Unfortunately this feature was nixed in vmware workstation 8 [1]. [1] http://www.replaydebugging.com/ http://www.replaydebugging.com/
- martin_ 13y ago
  That's very interesting! Why did they remove that feature if it was so awesome?
- khuey 13y ago
  We still have it working on a box in the office here. I used it last week ;-)

neur0mancer 13y ago

I tried it. Unfortunately, it is very slow.

adamnemecek 13y ago
It is. But that one time when you will be dealing with a Heisenbug, you will be thankful for it.
- ginko 13y ago
  The thing with Heisenbugs is that they usually don't happen when you attach a debugger.
  - adamnemecek 13y ago
    That kind of depends. They can also happen not because you attached a debugger but because you set a break point in a particularly unfortunate location.
    - RobotCaleb 13y ago
      Oh man. I spent a couple of days tracking one of those down. When the light went on in my head I felt like such an idiot.
- davidgerard 13y ago
  This.
tintor 13y ago
GDB reverse debugger is very slow as it is saving the execution state after every instruction executed. There is a commercial plugin for GDB reverse debugger: UndoDB. It is much faster than just GDB (they claim only 1.7x slowdown), but unfortunately not very stable for the product I'm working on.
- gregthelaw 13y ago
  I'm a co-founder of the company. Lots of stability improvements over the past year or so: I'd urge you to try it again. Particularly v4 beta. Send email to beta@undo-software.com to get a beta license. I'd never claim it's perfect: it's software after all, and the fact most software has bugs is why we have a business in the first place ;) But we're used by some pretty hardcore folk these days, including Cadence, NASA, and some of the biggest banks in the world. And the PyPy guys seem pretty happy :) Oh, and of course, if you do find issues, please do report them to support@undo-software.com -- then we can fix them.

coldcode 13y ago

Reverse debugging sounds like putting bugs in. We do that every day already.

Groxx 13y ago

For Java, if it still runs: http://www.lambdacs.com/debugger/ http://www.lambdacs.com/debugger/ I really like the idea. Never knew GDB had implemented it (thanks for the link!). I'll definitely try this out :)

bravura 13y ago

I have a more modest request: Could debuggers please store the code paths and variable values, over the last 50 executions or perhaps the test suite (deterministic)? This is particularly useful for understanding someone else's code. [edit: a context-free version of reverse debugging] I've recently been enjoying using IntelliJ for Scala, because it can show me types, and jump to type declarations, and usages. However, I still have to do a bunch of print debugging when trying to understand someone else's code, because I want to understand what values certain variables take. This is a pain using a compiler as slow as Scala. Something that captured relevant information, so that it would be easier + faster to do print debugging against previous code runs or the test suite, would be awesome.

jbert 13y ago
It looks like the underlying feature which enables reverse debugging, Process Recording, could do what you describe: https://sourceware.org/gdb/wiki/ProcessRecord https://sourceware.org/gdb/wiki/ProcessRecord the data may only be in-process, if so it would need to be save-able and load-able.
j_s 13y ago
Microsoft sells this with the 'Ultimate' SKU of Visual Studio: http://msdn.microsoft.com/en-us/library/dd264915.aspx http://msdn.microsoft.com/en-us/library/dd264915.aspx

fijal 13y ago

We (PyPy developers) tried this and it's unusably slow. I don't know how they got that slow, but it takes a bit forever to execute 10mln instructions. We ended up using UndoDB gdb wrapper, which runs slower than they advertise (+- the same speed as valgrind so say 30x give or take), but is a lot more usable. They had stability problems in the past, but with the recent versions, we're able to run stuff like complex python programs on top of pypy on top of UndoDB, which is very impressive. Very useful for hard GC bugs.

peterhajas 13y ago

Cool! Does `lldb` have similar functionality?