7 ms·

Author missed one of the best features: easy access to hardware breakpoints. Breaking on a memory read or write, either a raw address or via a symbol, is one of

by swaits 1y ago

Author missed one of the best features: easy access to hardware breakpoints. Breaking on a memory read or write, either a raw address or via a symbol, is one of the most time saving debugging tools I know.

yakshaving_jgt 1y ago
Is there somewhere where this approach is described in more detail?
- lock1 1y ago
  Search for "watchpoint debugging". Usually in most garbage-collected environments, it just observes & breaks on symbols though, not raw addresses. Vscode (or an editor with ADP support) supports unconditional breakpoints, watchpoints, and logpoints (observe & log values to the debug console).
  - SAI_Peregrinus 1y ago
    What watchpoints observe depends on the device & environment. E.g. for ARM Cortex-M MCUs, the (optional but very common) Data Watchpoint and Trace Unit (DWT) provides 1 or 4 watchpoints by using hardware comparators. They operate on raw addresses, since they work at the machine code level. They issue a debug interrupt when the comparator matches the address & operation (read/write), which the debug probe (J-Link, ST-Link, or other such probe) detects & forwards to the debugger (GDB, LLDB, etc.). Hardware watchpoints are extremely useful for debugging memory corruption in embedded systems, since they bypass the code running on the device under test entirely.
- jesse__ 1y ago
  Very roughly, hardware watchpoints are memory addresses you ask the processor to issue an "event" for when they're read from, written to, or executed. This event is processed by the kernel, and passed through to the debugger, which breaks execution of the program on the instruction that issued the read/write/exec. A concrete use case for this is catching memory corruption. If your program corrupts a known piece of memory, just set a hardware watchpoint on that memory address and BOOM, the debugger breaks execution on exactly the line that's responsible for the corruption. It's a fucking godsend sometimes.
coderatlarge 1y ago
windbg used to offer scripting capabilities that teams could use to trigger validation of any number of internal data structures essentially at every breakpoint or watchpoint trigger. it was a tremendous way to detect subtle state corruption. and sharing scripts across teams was also a way to share knowledge of a complex binary that was often not encoded in asserts or other aspects of the codebase.
- bpye 1y ago
  This still exists? You can also use JavaScript to script/extend and there is a native code API too. Note: I do work at MSFT, I have used these capabilities but I’m not on the debugger team. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/javascript-debugger-scripting https://learn.microsoft.com/en-us/windows-hardware/drivers/d... https://www.timdbg.com/posts/whats-the-target-model/ https://www.timdbg.com/posts/whats-the-target-model/ https://github.com/microsoft/WinDbg-Samples/tree/master https://github.com/microsoft/WinDbg-Samples/tree/master
  - coderatlarge 1y ago
    thanks for the pointers glad to hear it’s all still there i haven’t seen this type of capability used in too many companies tbh and it seems like a lot of opportunity to improve stability and debugging speed and even code exploration/learning (did i break something ?)
    - delta_p_delta_x 1y ago
      WinDbg also has time-travel/record-replay debugging.
      - coderatlarge 1y ago
        that sounds like a super-powerful feature especially for multithreaded debugging though i’ve never used anything like it.
jesse__ 1y ago
Oh my god, same. This literally catches bugs with a smoking gun in their hand in a way that's completely impossible with printf. I'd upvote this 100 times if I could.
- sherincall 1y ago
  > completely impossible with printf Not printf exactly, but I've found bugs with a combination of mprotect, userfaultfd and backtrace_symbols when I couldn't use HW breakpoints. Basically, mark a set of pages as non-writable so that any writes trigger a pagefault, then register yourself as a pagefault handler for those and see who is doing the write, apply the write and move on. You can do this with LD_PRELOAD without even recompiling the debugee.
  - jesse__ 1y ago
    Oh yeah, I've done that trick before, it's quite handy. Sometimes catches bugs that would be tricky to find with watchpoints. I wrote an allocator that allocates an extra page either before or after the users allocation, and aligns the user allocation such that it is as close as possible (while respecting user-requested alignment) to butting up against that extra page (marked PROT_NONE). Can catch underflows or overflows with this method, but not both simultaneously. Very handy from time to time.
  - antonchekhov 1y ago
    Back in the early '90s, I was a big fan of the C/C++ debugging library "electric fence" (written by Bruce Perens) - it was a malloc() implementation that used mmap() to set the pages of the returned buffer such that any writes in that region (and even read accesses!) caused a segfault to happen, and the program halts, so you can examine the stack. It was a godsend.
praptak 1y ago
From the same toolbox: expression watch. Set a watch on the invariant being violated (say "bufpos < buflen") and get a breakpoint the moment it changes.
roca 1y ago
Especially with combined with reverse-execution in rr or UndoDB!
AndyKelley 1y ago
Combined with time travel it's mind-blowing. Stumble upon some corrupted memory? Just put a watch point on it, and run the program backwards. Two weeks just turned into two minutes.
- mwcz 1y ago
  What time traveling debugger(s) do you recommend? I'm a regular GDB user and keep meaning to try rr.
  - AndyKelley 1y ago
    Since you're already a GDB user, rr will probably feel comfortably familiar. https://rr-project.org/ https://rr-project.org/ already has enough example usage to get you going on the home page right there.
  - adgjlsfhk1 1y ago
    rr is excellent. the only problem is it doesn't work for Windows
    - delta_p_delta_x 1y ago
      It's a little odd to use GDB on Windows. Have you considered either the Visual Studio debugger, or WinDbg? WinDbg has time travel debugging, and is arguably 'more excellent': https://learn.microsoft.com/en-gb/windows-hardware/drivers/debuggercmds/time-travel-debugging-overview https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...

by swaits 1y ago

yakshaving_jgt 1y ago

Is there somewhere where this approach is described in more detail?

lock1 1y ago
Search for "watchpoint debugging". Usually in most garbage-collected environments, it just observes & breaks on symbols though, not raw addresses. Vscode (or an editor with ADP support) supports unconditional breakpoints, watchpoints, and logpoints (observe & log values to the debug console).
- SAI_Peregrinus 1y ago
  What watchpoints observe depends on the device & environment. E.g. for ARM Cortex-M MCUs, the (optional but very common) Data Watchpoint and Trace Unit (DWT) provides 1 or 4 watchpoints by using hardware comparators. They operate on raw addresses, since they work at the machine code level. They issue a debug interrupt when the comparator matches the address & operation (read/write), which the debug probe (J-Link, ST-Link, or other such probe) detects & forwards to the debugger (GDB, LLDB, etc.). Hardware watchpoints are extremely useful for debugging memory corruption in embedded systems, since they bypass the code running on the device under test entirely.
jesse__ 1y ago
Very roughly, hardware watchpoints are memory addresses you ask the processor to issue an "event" for when they're read from, written to, or executed. This event is processed by the kernel, and passed through to the debugger, which breaks execution of the program on the instruction that issued the read/write/exec. A concrete use case for this is catching memory corruption. If your program corrupts a known piece of memory, just set a hardware watchpoint on that memory address and BOOM, the debugger breaks execution on exactly the line that's responsible for the corruption. It's a fucking godsend sometimes.

coderatlarge 1y ago

windbg used to offer scripting capabilities that teams could use to trigger validation of any number of internal data structures essentially at every breakpoint or watchpoint trigger. it was a tremendous way to detect subtle state corruption. and sharing scripts across teams was also a way to share knowledge of a complex binary that was often not encoded in asserts or other aspects of the codebase.

bpye 1y ago
This still exists? You can also use JavaScript to script/extend and there is a native code API too. Note: I do work at MSFT, I have used these capabilities but I’m not on the debugger team. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/javascript-debugger-scripting https://learn.microsoft.com/en-us/windows-hardware/drivers/d... https://www.timdbg.com/posts/whats-the-target-model/ https://www.timdbg.com/posts/whats-the-target-model/ https://github.com/microsoft/WinDbg-Samples/tree/master https://github.com/microsoft/WinDbg-Samples/tree/master
- coderatlarge 1y ago
  thanks for the pointers glad to hear it’s all still there i haven’t seen this type of capability used in too many companies tbh and it seems like a lot of opportunity to improve stability and debugging speed and even code exploration/learning (did i break something ?)
  - delta_p_delta_x 1y ago
    WinDbg also has time-travel/record-replay debugging.
    - coderatlarge 1y ago
      that sounds like a super-powerful feature especially for multithreaded debugging though i’ve never used anything like it.

jesse__ 1y ago

Oh my god, same. This literally catches bugs with a smoking gun in their hand in a way that's completely impossible with printf. I'd upvote this 100 times if I could.

sherincall 1y ago
> completely impossible with printf Not printf exactly, but I've found bugs with a combination of mprotect, userfaultfd and backtrace_symbols when I couldn't use HW breakpoints. Basically, mark a set of pages as non-writable so that any writes trigger a pagefault, then register yourself as a pagefault handler for those and see who is doing the write, apply the write and move on. You can do this with LD_PRELOAD without even recompiling the debugee.
- jesse__ 1y ago
  Oh yeah, I've done that trick before, it's quite handy. Sometimes catches bugs that would be tricky to find with watchpoints. I wrote an allocator that allocates an extra page either before or after the users allocation, and aligns the user allocation such that it is as close as possible (while respecting user-requested alignment) to butting up against that extra page (marked PROT_NONE). Can catch underflows or overflows with this method, but not both simultaneously. Very handy from time to time.
- antonchekhov 1y ago
  Back in the early '90s, I was a big fan of the C/C++ debugging library "electric fence" (written by Bruce Perens) - it was a malloc() implementation that used mmap() to set the pages of the returned buffer such that any writes in that region (and even read accesses!) caused a segfault to happen, and the program halts, so you can examine the stack. It was a godsend.

praptak 1y ago

From the same toolbox: expression watch. Set a watch on the invariant being violated (say "bufpos < buflen") and get a breakpoint the moment it changes.

roca 1y ago

Especially with combined with reverse-execution in rr or UndoDB!

AndyKelley 1y ago

Combined with time travel it's mind-blowing. Stumble upon some corrupted memory? Just put a watch point on it, and run the program backwards. Two weeks just turned into two minutes.

mwcz 1y ago
What time traveling debugger(s) do you recommend? I'm a regular GDB user and keep meaning to try rr.
- AndyKelley 1y ago
  Since you're already a GDB user, rr will probably feel comfortably familiar. https://rr-project.org/ https://rr-project.org/ already has enough example usage to get you going on the home page right there.
- adgjlsfhk1 1y ago
  rr is excellent. the only problem is it doesn't work for Windows
  - delta_p_delta_x 1y ago
    It's a little odd to use GDB on Windows. Have you considered either the Visual Studio debugger, or WinDbg? WinDbg has time travel debugging, and is arguably 'more excellent': https://learn.microsoft.com/en-gb/windows-hardware/drivers/debuggercmds/time-travel-debugging-overview https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...