9 ms·

Reverse Debugging for Python

169 points by scribu 10y ago

d33 10y ago
Cool, but why do you have to disable ALSR for recording to work?
- wyldfire 10y ago
  If the implementation records the state of your program at each time interval as "the values stored at particular memory locations", it's much simpler to put those values back at those locations than it would be to recognize the memory map and decompose the state in order to play it back at the arbitrarily established offsets used with ASLR. As a practical matter, ASLR should have little if any impact to nearly all programs design, so disabling it at development/debug time should not come at a big cost. BTW this post states "There is no fundamental reason for [ASLR] restriction, but it is some work to fix." EDIT: I see, your focus is specifically on recording. Disregard this comment, good point.
- moyix 10y ago
  If I had to guess it just simplifies the implementation. If you have deterministic load addresses then you can just reload the program on replay. If you have ASLR on, then you need to record where each library/stack/heap is and arrange for them to be placed at the same place when you replay. It's doable (though I don't know offhand how to specify the stack/heap location from userspace) but extra work. I've done similar things before – in PANDA we don't strictly need to snapshot the full device state when creating a recording; it would be enough to just keep track of which memory regions are I/O and reconstitute that mapping. But QEMU's savevm/loadvm saves and restores that mapping as a side effect so it's easier to just let that happen. Edit: also, instead of disabling ASLR system-wide, it might be better to just use "setarch `uname -m` -R <pypy>", which disables it for just a single process.
- masklinn 10y ago
  Cf "Current issues" section: > Only works on Linux, and only with Address Space Layout Randomization (ASLR) disabled. There is no fundamental reason for either restriction, but it is some work to fix.
- DanWaterworth 10y ago
  Here's a python program that should be deterministic, but isn't without disabling ASLR: print id(object())
  - masklinn 10y ago
    Why should this program be deterministic?
    - moyix 10y ago
      Reversible debugging (also known as record and replay) requires making the program behave exactly the same way every time. That way you can "execute backward": at statement n of the program, restart it from the beginning, and stop at statement n-1 (in practice you would take snapshots of the program state along the way so you don't have to re-execute all the way from the beginning of the program). In the example given here, you could get away with it since control flow doesn't depend on the value of id(object()). But for example, if you had: if (id(object()) >> 12) & 1: print "foo" else: print "bar" You would need to guarantee that id(object()) always returned the same value so that the execution follows the same path.
    - DanWaterworth 10y ago
      Ah, you've caught me out on my sloppy use of English. What I should have said is that the address space layout is a source of non-determinism that would otherwise need to be accounted for in the recording of a program. Further, I hold to the view that given a program, knowledge of the thread execution schedule/scheduler and the results of external calls, a program's execution should be deterministic. Thinking about this now, I realize disabling ASLR shouldn't be enough to fix this problem since recording and replaying are sufficiently different.
      - masklinn 10y ago
        > What I should have said is that the address space layout is a source of non-determinism that would otherwise need to be accounted for in the recording of a program. Ah yes, it needs to be deterministic in the context of a time-traveling debugger, I was thinking about the general case, sorry. > Further, I hold to the view that given a program, knowledge of the thread execution schedule/scheduler and the results of external calls, a program's execution should be deterministic. Surely it is, if you precisely know all timings and IO and can replay them there's nowhere for non-determinism to creep in. That's not really an acceptable scale of required knowledge for a time-traveling debugger though.
- mzs 10y ago
  "no longer necessary from revision ff376ccacb36."
wyldfire 10y ago
First we get awesome CPy C-extension emulation and now this? Thanks, team! Completely serious: when can PyPy obsolete CPython? Why don't more CPython core devs work on PyPy? Besides perhaps some remaining gap for C-extensions support, what else is missing?
- d33 10y ago
  Platforms I guess. Plain C code is much more portable than JIT and PyPy is reinventing JIT implementation as opposed to using something like LLVM, so they're much more limited in terms of supported target platforms.
  - wyldfire 10y ago
    Oh, my mistake -- I had assumed that it did use LLVM. Good point about portability, CPython has tons of supported targets and PyPy just a handful. Also CPython is really easy to build IMO.
    - loeg 10y ago
      LLVM isn't as widely portable as generic C, either. ARM and x86 are well supported in LLVM, but CPython runs fine on MIPS, Sparc, PPC, etc (via GCC or other C compilers).
  - DanWaterworth 10y ago
    You can compile PyPy without its JIT.
    - loeg 10y ago
      Yes, although without JIT it's slower than CPython. So, what's the point?
      - DanWaterworth 10y ago
        If you are happy using CPython, you can't be terribly concerned with performance. The point is, pypy will run on more platforms than its JIT targets.
baq 10y ago
amazing as usual :) no python 3 as usual :( 99% of my production code is python 3.
- masklinn 10y ago
  > no python 3 as usual :( 99% of my production code is python 3. Contribute work or funding[0] to pypy3? [0] http://pypy.org/py3donate.html http://pypy.org/py3donate.html [1] http://doc.pypy.org/en/latest/release-pypy3.3-v5.2-alpha1.html http://doc.pypy.org/en/latest/release-pypy3.3-v5.2-alpha1.ht...
  - corysama 10y ago
    Is the tech world really holding back pypy3 for a lack of $40k? I would expect there are multiple individuals in this thread who could top that off without feeling much sting. Let alone the tech companies they work for who would benefit greatly from it.
    - flubert 10y ago
      Just out of curiosity, at what level of wealth or yearly income would you think someone would have to have in order to drop $40k on a project like that?
      - corysama 10y ago
        I meet a lot of different engineers in different situations around SF all the time. Recently met one who is at a point that he owns several (>2) Teslas... Someone with $400K in liquid assets could do it without being embarrassed. I would be surprised if there wasn't as least one person reading these comments in that situation. Would that person care deeply enough to say "Here! Just finish it!" ?? Not likely. However, I'm very surprised that there isn't at least one multi-million dollar company who would value this higher than few person-months of internal engineering time.
denfromufa 10y ago
How is this more powerful than what I can already do in PTVS or PyCharm? I can move around the current executing line freely back-and-forth!
- deleted 10y ago
  [deleted]
- corysama 10y ago
  I believe in PTVS you can set the current executing line higher up in a function and execute those lines again, thus further mutating the state of the program. But, with this, you can roll back to earlier points of execution. Thus, un-mutating the state of the program. You can continue to roll backwards in execution to before the function was called, further and further until you observe some earlier code making a change that eventually led to the problem you are debugging.
  - denfromufa 10y ago
    Then I guess http://www.pythontutor.com http://www.pythontutor.com could do this long before!
    - corysama 10y ago
      Can it do this for arbitrary programs running on your command line?
      - denfromufa 10y ago
        Not sure, seems only in browser
    - Veedrac 10y ago
      Python Tutor is different; it records a very small subset of the information available. You can't, say, reverse and then run some introspection code.
lucio 10y ago
Shouldn't his be a "reversible debugger" ? or a time-travel debugger as the Elm debugger is called?
- lgas 10y ago
  No, the way things work is that people name things whatever they want.
gregthelaw 10y ago
Great to see. I'm biassed of course, but reversible debugging is the future (or should that be the past? Sorry.) If you'll excuse the shameless plug, http://undo.io/ http://undo.io/ for the same with C/C++. I'd like to think we were at least a little influential in this! :) https://vimeo.com/160863576 https://vimeo.com/160863576 Greg
- tperry98 10y ago
  Also worth noting that Undo is hiring: http://undo.io/about-us/careers/ http://undo.io/about-us/careers/
- justinclift 10y ago
  No pricing on the website, so it's likely to turn off many people here.
Dowwie 10y ago
Does the author provide no link to the project source or have I missed it?
holri 10y ago
Isn't logging enough to track down a lot of such problems?
- heartsucker 10y ago
  If you logged the stack at every step along with every variable in memory, yeah. The point of debugging like this is to not have to insert 40 log.trace() statements with 30 variables just to track it all down. It's interactive and does it all for you.
  - holri 10y ago
    Usually I do not need every variable in memory. Often the relevant spot can be very well circumscribed.

169 points by scribu 10y ago

d33 10y ago

Cool, but why do you have to disable ALSR for recording to work?

wyldfire 10y ago
If the implementation records the state of your program at each time interval as "the values stored at particular memory locations", it's much simpler to put those values back at those locations than it would be to recognize the memory map and decompose the state in order to play it back at the arbitrarily established offsets used with ASLR. As a practical matter, ASLR should have little if any impact to nearly all programs design, so disabling it at development/debug time should not come at a big cost. BTW this post states "There is no fundamental reason for [ASLR] restriction, but it is some work to fix." EDIT: I see, your focus is specifically on recording. Disregard this comment, good point.
moyix 10y ago
If I had to guess it just simplifies the implementation. If you have deterministic load addresses then you can just reload the program on replay. If you have ASLR on, then you need to record where each library/stack/heap is and arrange for them to be placed at the same place when you replay. It's doable (though I don't know offhand how to specify the stack/heap location from userspace) but extra work. I've done similar things before – in PANDA we don't strictly need to snapshot the full device state when creating a recording; it would be enough to just keep track of which memory regions are I/O and reconstitute that mapping. But QEMU's savevm/loadvm saves and restores that mapping as a side effect so it's easier to just let that happen. Edit: also, instead of disabling ASLR system-wide, it might be better to just use "setarch `uname -m` -R <pypy>", which disables it for just a single process.
masklinn 10y ago
Cf "Current issues" section: > Only works on Linux, and only with Address Space Layout Randomization (ASLR) disabled. There is no fundamental reason for either restriction, but it is some work to fix.
DanWaterworth 10y ago
Here's a python program that should be deterministic, but isn't without disabling ASLR: print id(object())
- masklinn 10y ago
  Why should this program be deterministic?
  - moyix 10y ago
    Reversible debugging (also known as record and replay) requires making the program behave exactly the same way every time. That way you can "execute backward": at statement n of the program, restart it from the beginning, and stop at statement n-1 (in practice you would take snapshots of the program state along the way so you don't have to re-execute all the way from the beginning of the program). In the example given here, you could get away with it since control flow doesn't depend on the value of id(object()). But for example, if you had: if (id(object()) >> 12) & 1: print "foo" else: print "bar" You would need to guarantee that id(object()) always returned the same value so that the execution follows the same path.
  - DanWaterworth 10y ago
    Ah, you've caught me out on my sloppy use of English. What I should have said is that the address space layout is a source of non-determinism that would otherwise need to be accounted for in the recording of a program. Further, I hold to the view that given a program, knowledge of the thread execution schedule/scheduler and the results of external calls, a program's execution should be deterministic. Thinking about this now, I realize disabling ASLR shouldn't be enough to fix this problem since recording and replaying are sufficiently different.
    - masklinn 10y ago
      > What I should have said is that the address space layout is a source of non-determinism that would otherwise need to be accounted for in the recording of a program. Ah yes, it needs to be deterministic in the context of a time-traveling debugger, I was thinking about the general case, sorry. > Further, I hold to the view that given a program, knowledge of the thread execution schedule/scheduler and the results of external calls, a program's execution should be deterministic. Surely it is, if you precisely know all timings and IO and can replay them there's nowhere for non-determinism to creep in. That's not really an acceptable scale of required knowledge for a time-traveling debugger though.
mzs 10y ago
"no longer necessary from revision ff376ccacb36."

wyldfire 10y ago

First we get awesome CPy C-extension emulation and now this? Thanks, team! Completely serious: when can PyPy obsolete CPython? Why don't more CPython core devs work on PyPy? Besides perhaps some remaining gap for C-extensions support, what else is missing?

d33 10y ago
Platforms I guess. Plain C code is much more portable than JIT and PyPy is reinventing JIT implementation as opposed to using something like LLVM, so they're much more limited in terms of supported target platforms.
- wyldfire 10y ago
  Oh, my mistake -- I had assumed that it did use LLVM. Good point about portability, CPython has tons of supported targets and PyPy just a handful. Also CPython is really easy to build IMO.
  - loeg 10y ago
    LLVM isn't as widely portable as generic C, either. ARM and x86 are well supported in LLVM, but CPython runs fine on MIPS, Sparc, PPC, etc (via GCC or other C compilers).
- DanWaterworth 10y ago
  You can compile PyPy without its JIT.
  - loeg 10y ago
    Yes, although without JIT it's slower than CPython. So, what's the point?
    - DanWaterworth 10y ago
      If you are happy using CPython, you can't be terribly concerned with performance. The point is, pypy will run on more platforms than its JIT targets.

baq 10y ago

amazing as usual :) no python 3 as usual :( 99% of my production code is python 3.

masklinn 10y ago
> no python 3 as usual :( 99% of my production code is python 3. Contribute work or funding[0] to pypy3? [0] http://pypy.org/py3donate.html http://pypy.org/py3donate.html [1] http://doc.pypy.org/en/latest/release-pypy3.3-v5.2-alpha1.html http://doc.pypy.org/en/latest/release-pypy3.3-v5.2-alpha1.ht...
- corysama 10y ago
  Is the tech world really holding back pypy3 for a lack of $40k? I would expect there are multiple individuals in this thread who could top that off without feeling much sting. Let alone the tech companies they work for who would benefit greatly from it.
  - flubert 10y ago
    Just out of curiosity, at what level of wealth or yearly income would you think someone would have to have in order to drop $40k on a project like that?
    - corysama 10y ago
      I meet a lot of different engineers in different situations around SF all the time. Recently met one who is at a point that he owns several (>2) Teslas... Someone with $400K in liquid assets could do it without being embarrassed. I would be surprised if there wasn't as least one person reading these comments in that situation. Would that person care deeply enough to say "Here! Just finish it!" ?? Not likely. However, I'm very surprised that there isn't at least one multi-million dollar company who would value this higher than few person-months of internal engineering time.

denfromufa 10y ago

How is this more powerful than what I can already do in PTVS or PyCharm? I can move around the current executing line freely back-and-forth!

deleted 10y ago
[deleted]
corysama 10y ago
I believe in PTVS you can set the current executing line higher up in a function and execute those lines again, thus further mutating the state of the program. But, with this, you can roll back to earlier points of execution. Thus, un-mutating the state of the program. You can continue to roll backwards in execution to before the function was called, further and further until you observe some earlier code making a change that eventually led to the problem you are debugging.
- denfromufa 10y ago
  Then I guess http://www.pythontutor.com http://www.pythontutor.com could do this long before!
  - corysama 10y ago
    Can it do this for arbitrary programs running on your command line?
    - denfromufa 10y ago
      Not sure, seems only in browser
  - Veedrac 10y ago
    Python Tutor is different; it records a very small subset of the information available. You can't, say, reverse and then run some introspection code.

lucio 10y ago

Shouldn't his be a "reversible debugger" ? or a time-travel debugger as the Elm debugger is called?

lgas 10y ago
No, the way things work is that people name things whatever they want.

gregthelaw 10y ago

Great to see. I'm biassed of course, but reversible debugging is the future (or should that be the past? Sorry.) If you'll excuse the shameless plug, http://undo.io/ http://undo.io/ for the same with C/C++. I'd like to think we were at least a little influential in this! :) https://vimeo.com/160863576 https://vimeo.com/160863576 Greg

tperry98 10y ago
Also worth noting that Undo is hiring: http://undo.io/about-us/careers/ http://undo.io/about-us/careers/
justinclift 10y ago
No pricing on the website, so it's likely to turn off many people here.

Dowwie 10y ago

Does the author provide no link to the project source or have I missed it?

holri 10y ago

Isn't logging enough to track down a lot of such problems?

heartsucker 10y ago
If you logged the stack at every step along with every variable in memory, yeah. The point of debugging like this is to not have to insert 40 log.trace() statements with 30 variables just to track it all down. It's interactive and does it all for you.
- holri 10y ago
  Usually I do not need every variable in memory. Often the relevant spot can be very well circumscribed.