4 ms·

Where should the debugger set a breakpoint?

101 points by hasheddan 5y ago

John7499 5y ago
Thanks to this article I can learn more. Expand my knowledge and abilities. https://www.advancedmd.review/ https://www.advancedmd.review/
- deleted 5y ago
  [deleted]
nyanpasu64 5y ago
I recall that in either gdb or Qt Creator's gdb wrapper, parameters were sometimes being uninitialized when I breakpoint the function declaration line, then be initialized when I single-step to on/before the first line of code. I can't replicate it now though. ---- I'm interested in the promise of omniscient debugging to partially replace both debuggers, tracing tools, and logging (including printf debugging), for bugs in complex systems which you can reproduce in controlled environments (not in production). However I don't believe it will be perfect even there, because I've noticed that when debugging many binaries (especially non-symboled or worse-yet stripped ones) in gdb, it prints many variable values as "optimized out". This requires special debug builds which partly alleviates the problem (I still get better results with compiled-in debug prints). I've recently been tracing the execution of the pipewire daemon. Since it's a large library filled with virtual function calls constructed from C macros, it's painful to statically analyze by reading the code. I found a Callgrind profiling trace to be useful as a control flow analysis tool. Unlike perf, it logs every function called, counts the calls to each function, records all callers and how many times each caller called a function, and even records how many times each line of code calls another function (can distinguish between two calls to the same child in the same function). For example, I recently used KCacheGrind to identify that spa_alsa_open() was called 6 times, 4 times by one caller and once each by 2 other callers. And it returned early twice, and continued onto snd_pcm_open() 4 times. I also wanted to trace all ALSA functions being called across the trace (ltrace didn't work well since libasound.so.2 was loaded via dlopen, and I couldn't figure out how to make ltrace not burn CPU intercepting calls from one .so function to another). This was difficult to extract from the the Callgrind trace, and I had to rig together a script using `gprof2dot --format=callgrind -n 0 -e 0`, rg to convert the output XML file into a CSV of callers and callees, and csvsql to isolate all calls from non-snd-containing functions into snd-containing functions, so I could read the source code to see where these ALSA functions were being called. The result didn't tell me the order or parameters of the function calls, so I had to infer the order in some cases, and insert logging statements in other cases. I think Pernosco is promising in this area, but I haven't tried it yet because it's commercial and has limited free uploads. One idea (I don't know if it's implemented or not) is having the ability to run SQL-like queries to list all calls from one binary to another .so library. Another idea (on top of the caller-callee lists you already have) is creating control flow graphs like KCacheGrind.
- modeless 5y ago
  Have you seen the demos about how Pernosco can help with printf debugging? Things like being able to click any line of output to timewarp the debugger to the call that printed it, or being able to add print statements and see the results without even having to re-execute the program.
  - nyanpasu64 5y ago
    I looked at the list of demos a while back, but didn't watch in detail at the time.
    - Agingcoder 5y ago
      I suggest you give it try, it's quite remarkable (disclaimer : I'm a pernosco customer) . You need to adjust your mindset to the way it works (I haven't seen any similar tool elsewhere, so to me it looks very different from a traditional forward debugger), but it's well worth spending a little bit of time with it. In particular, because it shows the whole 'dataflow history' (how a specific value flows through your program), it makes learning an unknown codebase (or a codebase you think you know for that matter...) much easier. Other features such as 'show me all calls to function f along with their parameters' help provide similar insights. I don't like debuggers, and used to delay as much as possible reaching for gdb and its cousins, but with pernosco, it's pretty much the opposite : I know that I will find my (potentially nasty) bug within a few minutes, by focusing on the 'big picture' provided by pernosco instead of wasting time working around the limitations of a regular debugger.
      - nyanpasu64 5y ago
        I tried uploading a trace... which promptly failed due to Qt running RDRAND and Pernosco detecting a desync on playback. I do think it's a cool idea (assuming the binaries are symboled and everything). However in my recent debugging, I've noticed that sometimes I recompile and run programs to change what logging I add, or rerun programs because I screwed up my breakpoints (pernosco could fix), but sometimes I recompile to change the code behavior and observe how the visible output of an app changes (which probably can't be simulated in a trace in general, aside from identifying what downstream code would be affected by changing a variable). So this won't be a complete solution for debugging, especially given that uploading more teaces costs extra money, but I assume it will be a powerful tool to have.
sour-taste 5y ago
Interesting article. I'm curious how another debugging feature, reverse stepping works. Does anyone have a source on that?
- khuey 5y ago
  There are a lot of different techniques. In Pernosco we more or less do "SELECT program_state WHERE time=T_current-1". gdb's built in reverse debugging feature creates an undo stack when executing forwards, and executing backwards pops modifications off of that stack. rr uses forward execution to emulate reverse execution, backing up to a checkpoint, executing forwards once to figure out where to stop, and then executing a second time from that checkpoint to the stopping point.
  - jraph 5y ago
    To add to that for people who don't know how rr works: rr records the entire state of the program by saving all the return values of syscalls. When replaying the execution, you can run the executable natively, only replacing syscalls by their recorded values. In GDB, in addition to reverse stepping, you can also use checkpoints (save the entire state of the program) and restore them later. GDB's checkpointing feature is based on fork. "Going back" is done by "exec()'ing" the checkpoint, replacing the current execution by the saved one. One big limitation is that forking does not work for debugging programs with multiple processes, and fork is not well defined when multi-threading too. As part of my PhD, I extended [1] gdb using its Python API and allowed checkpointing with CRIU (Checkpoint/Restore In Userspace) which does not have this limitation. The "small" difficulty is that both GDB and CRIU use ptrace to attach to the process, and it is not possible to attach to the same process twice. The trick was to send SIGSUSPEND to the program, then detach gdb, then run CRIU on the suspended program, then reattach to the process with GDB and let it continue. This is hacky but it does work, amazingly, including breakpoints and watchpoints which are restored. CRIU even restores the right PID number by using setpid. An advantage of checkpointing is that you can restore the execution and try another path (rr cannot change the execution, it can only replay what it recorded). You are not limited by the maximum size of an undo stack. But you need to save in advance. With rr, you can explore the whole execution, any state of the program is reachable. It's easier and more robust than checkpointing. I'd argue that's what is most useful most of the time. Checkpointing can be useful to explore multiple executions without having to run the program from the start, or to explore normally difficult to reproduce / unreachable states by modifying values with the debugger. [1] https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde [2] https://criu.org/ https://criu.org/

101 points by hasheddan 5y ago

Thanks to this article I can learn more. Expand my knowledge and abilities. https://www.advancedmd.review/ https://www.advancedmd.review/

deleted 5y ago
[deleted]

nyanpasu64 5y ago

I recall that in either gdb or Qt Creator's gdb wrapper, parameters were sometimes being uninitialized when I breakpoint the function declaration line, then be initialized when I single-step to on/before the first line of code. I can't replicate it now though. ---- I'm interested in the promise of omniscient debugging to partially replace both debuggers, tracing tools, and logging (including printf debugging), for bugs in complex systems which you can reproduce in controlled environments (not in production). However I don't believe it will be perfect even there, because I've noticed that when debugging many binaries (especially non-symboled or worse-yet stripped ones) in gdb, it prints many variable values as "optimized out". This requires special debug builds which partly alleviates the problem (I still get better results with compiled-in debug prints). I've recently been tracing the execution of the pipewire daemon. Since it's a large library filled with virtual function calls constructed from C macros, it's painful to statically analyze by reading the code. I found a Callgrind profiling trace to be useful as a control flow analysis tool. Unlike perf, it logs every function called, counts the calls to each function, records all callers and how many times each caller called a function, and even records how many times each line of code calls another function (can distinguish between two calls to the same child in the same function). For example, I recently used KCacheGrind to identify that spa_alsa_open() was called 6 times, 4 times by one caller and once each by 2 other callers. And it returned early twice, and continued onto snd_pcm_open() 4 times. I also wanted to trace all ALSA functions being called across the trace (ltrace didn't work well since libasound.so.2 was loaded via dlopen, and I couldn't figure out how to make ltrace not burn CPU intercepting calls from one .so function to another). This was difficult to extract from the the Callgrind trace, and I had to rig together a script using `gprof2dot --format=callgrind -n 0 -e 0`, rg to convert the output XML file into a CSV of callers and callees, and csvsql to isolate all calls from non-snd-containing functions into snd-containing functions, so I could read the source code to see where these ALSA functions were being called. The result didn't tell me the order or parameters of the function calls, so I had to infer the order in some cases, and insert logging statements in other cases. I think Pernosco is promising in this area, but I haven't tried it yet because it's commercial and has limited free uploads. One idea (I don't know if it's implemented or not) is having the ability to run SQL-like queries to list all calls from one binary to another .so library. Another idea (on top of the caller-callee lists you already have) is creating control flow graphs like KCacheGrind.

modeless 5y ago
Have you seen the demos about how Pernosco can help with printf debugging? Things like being able to click any line of output to timewarp the debugger to the call that printed it, or being able to add print statements and see the results without even having to re-execute the program.
- nyanpasu64 5y ago
  I looked at the list of demos a while back, but didn't watch in detail at the time.
  - Agingcoder 5y ago
    I suggest you give it try, it's quite remarkable (disclaimer : I'm a pernosco customer) . You need to adjust your mindset to the way it works (I haven't seen any similar tool elsewhere, so to me it looks very different from a traditional forward debugger), but it's well worth spending a little bit of time with it. In particular, because it shows the whole 'dataflow history' (how a specific value flows through your program), it makes learning an unknown codebase (or a codebase you think you know for that matter...) much easier. Other features such as 'show me all calls to function f along with their parameters' help provide similar insights. I don't like debuggers, and used to delay as much as possible reaching for gdb and its cousins, but with pernosco, it's pretty much the opposite : I know that I will find my (potentially nasty) bug within a few minutes, by focusing on the 'big picture' provided by pernosco instead of wasting time working around the limitations of a regular debugger.
    - nyanpasu64 5y ago
      I tried uploading a trace... which promptly failed due to Qt running RDRAND and Pernosco detecting a desync on playback. I do think it's a cool idea (assuming the binaries are symboled and everything). However in my recent debugging, I've noticed that sometimes I recompile and run programs to change what logging I add, or rerun programs because I screwed up my breakpoints (pernosco could fix), but sometimes I recompile to change the code behavior and observe how the visible output of an app changes (which probably can't be simulated in a trace in general, aside from identifying what downstream code would be affected by changing a variable). So this won't be a complete solution for debugging, especially given that uploading more teaces costs extra money, but I assume it will be a powerful tool to have.

sour-taste 5y ago

Interesting article. I'm curious how another debugging feature, reverse stepping works. Does anyone have a source on that?

khuey 5y ago
There are a lot of different techniques. In Pernosco we more or less do "SELECT program_state WHERE time=T_current-1". gdb's built in reverse debugging feature creates an undo stack when executing forwards, and executing backwards pops modifications off of that stack. rr uses forward execution to emulate reverse execution, backing up to a checkpoint, executing forwards once to figure out where to stop, and then executing a second time from that checkpoint to the stopping point.
- jraph 5y ago
  To add to that for people who don't know how rr works: rr records the entire state of the program by saving all the return values of syscalls. When replaying the execution, you can run the executable natively, only replacing syscalls by their recorded values. In GDB, in addition to reverse stepping, you can also use checkpoints (save the entire state of the program) and restore them later. GDB's checkpointing feature is based on fork. "Going back" is done by "exec()'ing" the checkpoint, replacing the current execution by the saved one. One big limitation is that forking does not work for debugging programs with multiple processes, and fork is not well defined when multi-threading too. As part of my PhD, I extended [1] gdb using its Python API and allowed checkpointing with CRIU (Checkpoint/Restore In Userspace) which does not have this limitation. The "small" difficulty is that both GDB and CRIU use ptrace to attach to the process, and it is not possible to attach to the same process twice. The trick was to send SIGSUSPEND to the program, then detach gdb, then run CRIU on the suspended program, then reattach to the process with GDB and let it continue. This is hacky but it does work, amazingly, including breakpoints and watchpoints which are restored. CRIU even restores the right PID number by using setpid. An advantage of checkpointing is that you can restore the execution and try another path (rr cannot change the execution, it can only replay what it recorded). You are not limited by the maximum size of an undo stack. But you need to save in advance. With rr, you can explore the whole execution, any state of the program is reachable. It's easier and more robust than checkpointing. I'd argue that's what is most useful most of the time. Checkpointing can be useful to explore multiple executions without having to run the program from the start, or to explore normally difficult to reproduce / unreachable states by modifying values with the debugger. [1] https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde [2] https://criu.org/ https://criu.org/