7 ms·

Writing a debugger from scratch: Breakpoints

320 points by ingve 3y ago

zubairq 3y ago
Really nice read. Does anyone know any other good articles or videos about how to write a debugger?
- emmanueloga_ 3y ago
  I was asking this myself this while reading the book "Crafting Interpreters". I posted a few resources I found on an issue about implementing debuggers [1] -- although honestly I still haven't gotten down to read all of them (or to implement a debugger! :-/). -- 1: https://github.com/munificent/craftinginterpreters/issues/922 https://github.com/munificent/craftinginterpreters/issues/92...
  - a1o 3y ago
    Besides breakpoints, any ideas on inspecting the value of a variable in each step, figuring out what variables are in scope, for the case of an interpreter?
    - emmanueloga_ 3y ago
      I’m guessing you’ll have to work with the scopes in the resolver: https://github.com/munificent/craftinginterpreters/blob/master/java/com/craftinginterpreters/lox/Resolver.java#L12 https://github.com/munificent/craftinginterpreters/blob/mast...
      - a1o 3y ago
        Ooh, thanks for this! :)
- mrazomor 3y ago
  I really liked https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ I learned a lot.
- wila 3y ago
  How about a book? I like "Advanced Windows Debugging" by Mario Hewardt and Daniel Pravat.
- Modified3019 3y ago
  Not what you asked for, but if that's your interest you'll probably appreciate https://justine.lol/blinkenlights/ https://justine.lol/blinkenlights/
- jansommer 3y ago
  I wrote this: https://ja.nsommer.dk/articles/x86-debugger-for-windows-and-wine.html https://ja.nsommer.dk/articles/x86-debugger-for-windows-and-... It's a debugger for Windows (and Wine), like the one in the article, written in C. It uses software breakpoints (infinite breakpoints)
- HansLambda 3y ago
  Take a look heere: https://eli.thegreenplace.net/tag/debuggers https://eli.thegreenplace.net/tag/debuggers It was eye-opening for me.
danparsonson 3y ago
Great article, thanks - one question I couldn't see answered there is, what do you do when you want to set more than four breakpoints at once?
- saagarjha 3y ago
  Use software breakpoints (which are mentioned but not described, the short story for those is you overwrite the address you care about with an illegal instruction and execution traps when it encounters that code, and then you undo it to continue).
- ithkuil 3y ago
  You use software breakpoints. Basically you overwrite the instruction you want to break at with a breakpoint instruction (e.g. int 3 on x86). This will cause the process to trap and the OS will then let the debugger process know about out somehow, e.g. via the SIGTRAP signal on Unix. The debugger then replaces the int 3 opcode (which is a single byte conveniently) with the first byte of the original instruction so that the execution can continue.
  - hinoki 3y ago
    If you revert the int 3 to the original instruction’s byte, when do you put it back? The breakpoint could still be active. In a trivial example, the breaking instruction could be a jump to itself, which you’d expect to immediately break into the debugger again. I thought the debugger had to emulate the instruction instead, but it’s not like I’ve ever implemented one…
    - ithkuil 3y ago
      Emulation is an option, rotating hardware debug registers is another option, detecting self-jumps is another option. I really only implemented a debugger for the esp8266 and it was just good enough for me and my team to get our job done so it didn't handle many edge cases like that
    - i_don_t_know 3y ago
tibbydudeza 3y ago
For the want of a decent JS React debugger.
- uxp8u61q 3y ago
  Debugging JS with vscode is probably the nicest debugging experience I've had, bar maybe C# with the latest VS.
  - kaycey2022 3y ago
    Is there a guide that covers this?
xvilka 3y ago
Writing cross platform (different OS, POSIX and not, different architectures, different endianess, etc) debugger, this is where all pain lies.
ekidd 3y ago
This is a great series! I noticed that the author was using https://github.com/hydro-project/rust-sitter https://github.com/hydro-project/rust-sitter as a parser. Which is based on https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/. I've been hearing about Tree-sitter a lot recently, so I dug into it. Tree-sitter is a tool for generating fast, incremental parsers. In particular, the algorithm is suited towards writing "language servers" for IDEs, which re-parse code incrementally as the user works. These kinds of incremental parsers have historically been a huge problem. It looks like Tree-sitter is an enormous practical advance in this area. And discovering that there's a way to use Tree-sitter from Rust is fantastic. From the post: #[rust_sitter::language] pub enum EvalExpr { Number( #[rust_sitter::leaf( pattern = r"(\d+|0x[0-9a-fA-F]+)", transform = parse_int )] u64 ), Symbol( #[rust_sitter::leaf( pattern = r"(([a-zA-Z0-9_@#.]+!)?[a-zA-Z0-9_@#.]+)", transform = parse_sym )] String ), // ... Getting easy access to fast, incremental parsing is a huge win. And Tree-Sitter has support for being used from a huge list of languages, not just Rust.
- junon 3y ago
  Tree sitter also has a bunch of deficiencies that don't make it ideal for a number of usecases, or sort of act bizarrely in some edge cases. Just evaluate tools like this cautiously, of course. But I like what it's done for the ecosystem as a whole!
  - a1o 3y ago
    I still don't know how to deal with forward declarations when using tree-sitter. :/
    - jamra 3y ago
      Because tree-sitter lexes as it parses, you may have to use an external scanner in order to deal with this kind of stuff. Where are you stuck trying to deal with forward declarations?
heelwood 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- _hl_ 3y ago
  @dang
Andrew018 3y ago
[dead]
matt3210 3y ago
Does anyone know a similar article using c/c++? Interesting concept.
- SebastienWae 3y ago
  Yes, the "Writing a Linux Debugger" series in C++. https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ And more generally there is "The Debugging Book" in python. https://www.debuggingbook.org/ https://www.debuggingbook.org/
  - emmanueloga_ 3y ago
    The last book is an example of a book that is really about debugging, not about debugger implementation. This is the case for most books you will find with a title matching /.debug./. Debugger knowledge seems to be scattered across the internet and language implementations. Also I never found a language implementation book that talks about how to make the implementation friendlier/compatible with writing a debugger.
- deleted 3y ago
  [deleted]
whartung 3y ago
I’ll share this anecdote told by a friend of mine. He was on a team building a Modula-2 compiler for OS/2, and his group was working on the debugger. At some point a debugger becomes feature complete enough that you can use the debugger to ... debug the debugger. But this was OS/2 which has true multiple processes (unlike it’s contemporary Windows 3.1). So you could, naturally, run the debugger in one process and attached it to another process which, just so happens to be another instance of the debugger. As with all things, while doing this they encountered bugs in the debugger that, well, needed to be debugged. He said there was a certain epiphany when they realized, because of the multi process nature of OS/2, that they could debug the debugger debugging the debugger. I would imagine this took a bit of focus. Turn away for a moment and probably really messes with your head.
- meepmorp 3y ago
  It's bugs all the way down.
  - mywittyname 3y ago
    I want to hear stories about debugger-on-debugger heisenbugs.
- cbsks 3y ago
  I worked at a company that made JTAG probes. When you wanted to debug the firmware on the probe you’d attach another probe to it. And if you encountered a bug while debugging that probe, then you’d attach another probe…
- timmisiak 3y ago
  I think my record when I was on the WinDbg team was 5 debuggers deep. I honestly think one of the best parts of writing a debugger is being your own recursive customer. I think that's something you only get to do for a few things. Debuggers, languages/compilers, and operating systems. And probably a few others.
  - robertlagrant 3y ago
dgb23 3y ago
This is amazing! I've been thinking about writing a debugger (for learning how they work etc.). This series is going to be a massive help!
- timmisiak 3y ago
  Honestly it's a great exercise for learning how low level stuff works in general. Happy to answer any questions you have!
elischleifer 3y ago
Working at Microsoft back in the early 00s I spent a lot of unfriendly hours with windbg. On one particular project we hunted for a terrible crash for months until it was uncovered that we were compiling against the single thread CRT when using threads extensively...whoops
parttimenerd 3y ago
I've done something similar in Python: A Python debugger from scratch in Python - https://github.com/parttimenerd/python-dbg/ https://github.com/parttimenerd/python-dbg/ - Part 1: https://mostlynerdless.de/blog/2023/09/20/lets-create-a-python-debugger-together-part-1/ https://mostlynerdless.de/blog/2023/09/20/lets-create-a-pyth... - Part 2: https://mostlynerdless.de/?p=1102&preview=1&_ppp=a17cda3e36 https://mostlynerdless.de/?p=1102&preview=1&_ppp=a17cda3e36

320 points by ingve 3y ago

zubairq 3y ago

Really nice read. Does anyone know any other good articles or videos about how to write a debugger?

emmanueloga_ 3y ago
I was asking this myself this while reading the book "Crafting Interpreters". I posted a few resources I found on an issue about implementing debuggers [1] -- although honestly I still haven't gotten down to read all of them (or to implement a debugger! :-/). -- 1: https://github.com/munificent/craftinginterpreters/issues/922 https://github.com/munificent/craftinginterpreters/issues/92...
- a1o 3y ago
  Besides breakpoints, any ideas on inspecting the value of a variable in each step, figuring out what variables are in scope, for the case of an interpreter?
  - emmanueloga_ 3y ago
    I’m guessing you’ll have to work with the scopes in the resolver: https://github.com/munificent/craftinginterpreters/blob/master/java/com/craftinginterpreters/lox/Resolver.java#L12 https://github.com/munificent/craftinginterpreters/blob/mast...
    - a1o 3y ago
      Ooh, thanks for this! :)
mrazomor 3y ago
I really liked https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ I learned a lot.
wila 3y ago
How about a book? I like "Advanced Windows Debugging" by Mario Hewardt and Daniel Pravat.
Modified3019 3y ago
Not what you asked for, but if that's your interest you'll probably appreciate https://justine.lol/blinkenlights/ https://justine.lol/blinkenlights/
jansommer 3y ago
I wrote this: https://ja.nsommer.dk/articles/x86-debugger-for-windows-and-wine.html https://ja.nsommer.dk/articles/x86-debugger-for-windows-and-... It's a debugger for Windows (and Wine), like the one in the article, written in C. It uses software breakpoints (infinite breakpoints)
HansLambda 3y ago
Take a look heere: https://eli.thegreenplace.net/tag/debuggers https://eli.thegreenplace.net/tag/debuggers It was eye-opening for me.

danparsonson 3y ago

Great article, thanks - one question I couldn't see answered there is, what do you do when you want to set more than four breakpoints at once?

saagarjha 3y ago
Use software breakpoints (which are mentioned but not described, the short story for those is you overwrite the address you care about with an illegal instruction and execution traps when it encounters that code, and then you undo it to continue).
ithkuil 3y ago
You use software breakpoints. Basically you overwrite the instruction you want to break at with a breakpoint instruction (e.g. int 3 on x86). This will cause the process to trap and the OS will then let the debugger process know about out somehow, e.g. via the SIGTRAP signal on Unix. The debugger then replaces the int 3 opcode (which is a single byte conveniently) with the first byte of the original instruction so that the execution can continue.
- hinoki 3y ago
  If you revert the int 3 to the original instruction’s byte, when do you put it back? The breakpoint could still be active. In a trivial example, the breaking instruction could be a jump to itself, which you’d expect to immediately break into the debugger again. I thought the debugger had to emulate the instruction instead, but it’s not like I’ve ever implemented one…
  - ithkuil 3y ago
    Emulation is an option, rotating hardware debug registers is another option, detecting self-jumps is another option. I really only implemented a debugger for the esp8266 and it was just good enough for me and my team to get our job done so it didn't handle many edge cases like that
  - i_don_t_know 3y ago

tibbydudeza 3y ago

For the want of a decent JS React debugger.

uxp8u61q 3y ago
Debugging JS with vscode is probably the nicest debugging experience I've had, bar maybe C# with the latest VS.
- kaycey2022 3y ago
  Is there a guide that covers this?

xvilka 3y ago

Writing cross platform (different OS, POSIX and not, different architectures, different endianess, etc) debugger, this is where all pain lies.

ekidd 3y ago

This is a great series! I noticed that the author was using https://github.com/hydro-project/rust-sitter https://github.com/hydro-project/rust-sitter as a parser. Which is based on https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/. I've been hearing about Tree-sitter a lot recently, so I dug into it. Tree-sitter is a tool for generating fast, incremental parsers. In particular, the algorithm is suited towards writing "language servers" for IDEs, which re-parse code incrementally as the user works. These kinds of incremental parsers have historically been a huge problem. It looks like Tree-sitter is an enormous practical advance in this area. And discovering that there's a way to use Tree-sitter from Rust is fantastic. From the post: #[rust_sitter::language] pub enum EvalExpr { Number( #[rust_sitter::leaf( pattern = r"(\d+|0x[0-9a-fA-F]+)", transform = parse_int )] u64 ), Symbol( #[rust_sitter::leaf( pattern = r"(([a-zA-Z0-9_@#.]+!)?[a-zA-Z0-9_@#.]+)", transform = parse_sym )] String ), // ... Getting easy access to fast, incremental parsing is a huge win. And Tree-Sitter has support for being used from a huge list of languages, not just Rust.

junon 3y ago
Tree sitter also has a bunch of deficiencies that don't make it ideal for a number of usecases, or sort of act bizarrely in some edge cases. Just evaluate tools like this cautiously, of course. But I like what it's done for the ecosystem as a whole!
- a1o 3y ago
  I still don't know how to deal with forward declarations when using tree-sitter. :/
  - jamra 3y ago
    Because tree-sitter lexes as it parses, you may have to use an external scanner in order to deal with this kind of stuff. Where are you stuck trying to deal with forward declarations?

heelwood 3y ago

[flagged]

deleted 3y ago
[deleted]
_hl_ 3y ago
@dang

Andrew018 3y ago

[dead]

matt3210 3y ago

Does anyone know a similar article using c/c++? Interesting concept.

SebastienWae 3y ago
Yes, the "Writing a Linux Debugger" series in C++. https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/ And more generally there is "The Debugging Book" in python. https://www.debuggingbook.org/ https://www.debuggingbook.org/
- emmanueloga_ 3y ago
  The last book is an example of a book that is really about debugging, not about debugger implementation. This is the case for most books you will find with a title matching /.debug./. Debugger knowledge seems to be scattered across the internet and language implementations. Also I never found a language implementation book that talks about how to make the implementation friendlier/compatible with writing a debugger.
deleted 3y ago
[deleted]

whartung 3y ago

I’ll share this anecdote told by a friend of mine. He was on a team building a Modula-2 compiler for OS/2, and his group was working on the debugger. At some point a debugger becomes feature complete enough that you can use the debugger to ... debug the debugger. But this was OS/2 which has true multiple processes (unlike it’s contemporary Windows 3.1). So you could, naturally, run the debugger in one process and attached it to another process which, just so happens to be another instance of the debugger. As with all things, while doing this they encountered bugs in the debugger that, well, needed to be debugged. He said there was a certain epiphany when they realized, because of the multi process nature of OS/2, that they could debug the debugger debugging the debugger. I would imagine this took a bit of focus. Turn away for a moment and probably really messes with your head.

meepmorp 3y ago
It's bugs all the way down.
- mywittyname 3y ago
  I want to hear stories about debugger-on-debugger heisenbugs.
cbsks 3y ago
I worked at a company that made JTAG probes. When you wanted to debug the firmware on the probe you’d attach another probe to it. And if you encountered a bug while debugging that probe, then you’d attach another probe…
timmisiak 3y ago
I think my record when I was on the WinDbg team was 5 debuggers deep. I honestly think one of the best parts of writing a debugger is being your own recursive customer. I think that's something you only get to do for a few things. Debuggers, languages/compilers, and operating systems. And probably a few others.
- robertlagrant 3y ago

dgb23 3y ago

This is amazing! I've been thinking about writing a debugger (for learning how they work etc.). This series is going to be a massive help!

timmisiak 3y ago
Honestly it's a great exercise for learning how low level stuff works in general. Happy to answer any questions you have!

elischleifer 3y ago

Working at Microsoft back in the early 00s I spent a lot of unfriendly hours with windbg. On one particular project we hunted for a terrible crash for months until it was uncovered that we were compiling against the single thread CRT when using threads extensively...whoops

parttimenerd 3y ago

I've done something similar in Python: A Python debugger from scratch in Python - https://github.com/parttimenerd/python-dbg/ https://github.com/parttimenerd/python-dbg/ - Part 1: https://mostlynerdless.de/blog/2023/09/20/lets-create-a-python-debugger-together-part-1/ https://mostlynerdless.de/blog/2023/09/20/lets-create-a-pyth... - Part 2: https://mostlynerdless.de/?p=1102&preview=1&_ppp=a17cda3e36 https://mostlynerdless.de/?p=1102&preview=1&_ppp=a17cda3e36