5 ms·

Making a low level Linux debugger, part 3: our first program

171 points by asrp 8y ago

pmarreck 8y ago
I now feel like debuggers are a design smell. I've used them less and less (to the point of not having to use them at all) as I've coded more test-first and designed things to be smaller (lower logic depth, etc.) I mean, all a debugger is, really, is a window into dynamic state, so if you control the, let's just call it what it is, the "rampant dispersion of state" at all times, you end up not needing a debugger (and producing fewer bugs) IMHO. Now I'm sure that kernel-level stuff wasn't designed like this, and probably won't be for some time (I'm hoping Rust changes that story, but not a low-level coder), so debuggers may still be necessary- just making a point that it seems possible to design in such a way that the need for one is drastically reduced, if not eliminated. Or at least, that's how it seems to me in high-level-language-land (things may differ at the assembly and C levels). EDIT: I apologize for OT, probably not the ideal place to start a "are debuggers really necessary if you design code correctly?" discussion, after all this is part 3 of a series, my bad
- yitchelle 8y ago
  I have been an embedded developer working quite close to the metal. Over the years, I have noticed that the further you get away from the metal, the less reliant you are on the debugger. In projects past, I have worked with MathLab developers who are deploying their code on a multicore CPU and they have no need to have a debugger (in the traditional sense) in their tool set.
  - thedirt0115 8y ago
    This is exactly what I was thinking, but couldn't phrase it quite as nicely. I never feel the need to reach for pdb when writing Python, but C on microcontroller? I need my gdb.
  - pmarreck 8y ago
    Why do you think debuggers are so necessary at the metal when they aren't further above the metal?
    - deleted 8y ago
      [deleted]
    - yitchelle 8y ago
      When debugging code that are close to the metal such as device drivers or interrupt service routines, the code's behavior is very tightly coupled to the metal. For example, the interrupt service routine that your debugging could not reacting fast enough and you need to find out why? Priority of the said interrupt is correctly configured and it is blocked by a higher priotiy interrupt. This type of behaviour difficult to track down if a debugger is not used.
- _wmd 8y ago
  It seems you've mastered some dark art unknown to an entire industry, you should totally write a book Meanwhile I'll rely on the smelly old debugger to dig me out of threading weirdness, memory corruption, resource leakages, other people's bugs, and to provide a window into a combinatorially massive input-dependent state my feeble mind simply can't fathom while writing tests for a tiny corner of a typical system that in aggregate amounts to millions of LOC.
  - jjoonathan 8y ago
    Nah, gradparent is not a wizard, just someone who has spent a lot of time in a single relatively simple environment that is mostly under their control. It's a beautiful thing when it happens. Most aren't so lucky.
    - tropo 8y ago
      He is taking after Linus Torvalds, who resisted adding a kernel debugger for many years. Linus used console spew, PC speaker beeps, and storing a hash of file/line into the PC's time/date.
      - hackcasual 8y ago
        That's less not needing a debugger by design, more how challenging it is to debug an operating system.
        tropo 8y ago
        Those last two (speaker and clock) yes, but otherwise no. Linus Torvalds really doesn't like using a debugger and he'd rather you didn't use one either. His words: https://yarchive.net/comp/linux/debuggers.html https://yarchive.net/comp/linux/debuggers.html
        pmarreck 8y ago
        He's not wrong. I still maintain that debuggers are a design smell.
- strong_silent_t 8y ago
  I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly. For example, if I'm working on a Python script under 1000 lines I would never use a debugger. I might use wireshark or linux system tools if something was really wrong, but usually the trackback or program output is sufficient. I've also tried to debug an Arduino project where I had a lot less knowledge of how the system works, and if I wasn't able to attach a debugger I don't want to know how long it would have taken me to find the problem (calling a null function pointer causes an interrupt that didn't have any handler). I've also tried to debug other people's projects in high level languages, without a debugger where after taking about an hour to figure it out based on pure reason, realizing it would have been very quick and easy to find with a debugger. I think if you look at what a program is doing, and at that point you can't form a testable hypothesis as to what is happening, at that point a debugger will certainly be faster if you know how to use it.
  - brokenmachine 8y ago
    How did you use a debugger on Arduino? I've been using Serial.print("WTF") more than I'd admit.
    - strong_silent_t 8y ago
      This is similar to what I did: https://learn.adafruit.com/proper-step-debugging-atsamd21-arduino-zero-m0/overview https://learn.adafruit.com/proper-step-debugging-atsamd21-ar... . That is for that specific platform, not sure if there is something similar for the other versions. I used the J-Link but I wanted to do it on Linux, so I used the JLink GDB Server along with the GDB build that comes with the Arduino IDE. It probably takes me a good 5 minutes to get everything set up, but the basic stuff works: breakpoints, stack trace, reading and writing memory locations. The setup is a little convoluted, I'm not sure if there is a better way, but here are my notes: Procedure for running w/ debugger attached. 1. disconnect debugger's usb 2. connect device via usb 3. arduino IDE program device 4. identify elf, set up gdb with elf. 5. plug in debugger 6. run jlink gdb server 7. attach gdb to jlink server "target remote:2331" 8. "monitor reset", "continue" to re-run target from beginning. 9. observe device is recognized by Arduino IDE 10. open Arduino IDE serial monitor to interact with device. Now, Ctrl+C in GDB halts device, and stack etc. can be inspected, and then it can continue running with "cont". To rerun the device with the same program without redoing everything, go to step 8 and reset the device. and continue from there. To find .elf on linux: ================================================== find /tmp -name sketch_name.ino.elf 2>/dev/null gdb location: ================================================== /home/user/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gdb JLink setup command: ================================================== JLinkGDBServer -device ATSAMD21G18A -if SWD -speed 4000 gdb attachment: ================================================== (gdb) target remote:2331
- remoteorbust 8y ago
  I find that I don't need a debugger until I do. People are always asking me what debugger I use for go and I tell them I've gone months without feeling the need for them. However every once in a while I need to use gdb + remote debugging features to figure out a particularly weird runtime issue and I'm glad it's there. Then I can often take the odd values I see in the debugger and start putting them into a test or a minimal repro.
  - pmarreck 8y ago
    This is the ideal use case IMHO
- asrp 8y ago
  I don't have that much to add to the other replies here. I used to never use pdb in Python for many years (not knowing or forgotten it exists) and doing perfectly well. Then I started using post-mortem debugging and finding/fixing bugs got so much faster. Basically, I'm using the computer to do the simulations I previously ran in my head (by evaluating things here and there). But that's just one point of data. Another debugger I now recommend people (in general) to look at is Squeak Smalltalk's debugger which allows a form of "top down" programming where undefined functions are called and filled in at runtime. The debugger/editor in this post borrows a little bit from that idea in spirit. Part 3 in particular shows how it can be used even when they are no bugs. About being off-topic, well, no-one has made another top level comment and by the time of your first post it was already 5 hours in. Some discussion is definitely better than none. So thank you for inciting that. If it burried some more on-topic discussion, that would be unfortunate though.
  - pmarreck 8y ago
    Ha, thanks for that!
- busterarm 8y ago
  Having access to a debugger when you need it is a necessity. Rampant use of it can, debatably, be seen as a crutch...but if you need it and you don't have it, you're going to have an extremely bad week.

171 points by asrp 8y ago

I now feel like debuggers are a design smell. I've used them less and less (to the point of not having to use them at all) as I've coded more test-first and designed things to be smaller (lower logic depth, etc.) I mean, all a debugger is, really, is a window into dynamic state, so if you control the, let's just call it what it is, the "rampant dispersion of state" at all times, you end up not needing a debugger (and producing fewer bugs) IMHO. Now I'm sure that kernel-level stuff wasn't designed like this, and probably won't be for some time (I'm hoping Rust changes that story, but not a low-level coder), so debuggers may still be necessary- just making a point that it seems possible to design in such a way that the need for one is drastically reduced, if not eliminated. Or at least, that's how it seems to me in high-level-language-land (things may differ at the assembly and C levels). EDIT: I apologize for OT, probably not the ideal place to start a "are debuggers really necessary if you design code correctly?" discussion, after all this is part 3 of a series, my bad

yitchelle 8y ago
I have been an embedded developer working quite close to the metal. Over the years, I have noticed that the further you get away from the metal, the less reliant you are on the debugger. In projects past, I have worked with MathLab developers who are deploying their code on a multicore CPU and they have no need to have a debugger (in the traditional sense) in their tool set.
- thedirt0115 8y ago
  This is exactly what I was thinking, but couldn't phrase it quite as nicely. I never feel the need to reach for pdb when writing Python, but C on microcontroller? I need my gdb.
- pmarreck 8y ago
  Why do you think debuggers are so necessary at the metal when they aren't further above the metal?
  - deleted 8y ago
    [deleted]
  - yitchelle 8y ago
    When debugging code that are close to the metal such as device drivers or interrupt service routines, the code's behavior is very tightly coupled to the metal. For example, the interrupt service routine that your debugging could not reacting fast enough and you need to find out why? Priority of the said interrupt is correctly configured and it is blocked by a higher priotiy interrupt. This type of behaviour difficult to track down if a debugger is not used.
_wmd 8y ago
It seems you've mastered some dark art unknown to an entire industry, you should totally write a book Meanwhile I'll rely on the smelly old debugger to dig me out of threading weirdness, memory corruption, resource leakages, other people's bugs, and to provide a window into a combinatorially massive input-dependent state my feeble mind simply can't fathom while writing tests for a tiny corner of a typical system that in aggregate amounts to millions of LOC.
- jjoonathan 8y ago
  Nah, gradparent is not a wizard, just someone who has spent a lot of time in a single relatively simple environment that is mostly under their control. It's a beautiful thing when it happens. Most aren't so lucky.
  - tropo 8y ago
    He is taking after Linus Torvalds, who resisted adding a kernel debugger for many years. Linus used console spew, PC speaker beeps, and storing a hash of file/line into the PC's time/date.
    - hackcasual 8y ago
      That's less not needing a debugger by design, more how challenging it is to debug an operating system.
      - tropo 8y ago
        Those last two (speaker and clock) yes, but otherwise no. Linus Torvalds really doesn't like using a debugger and he'd rather you didn't use one either. His words: https://yarchive.net/comp/linux/debuggers.html https://yarchive.net/comp/linux/debuggers.html
        pmarreck 8y ago
        He's not wrong. I still maintain that debuggers are a design smell.
strong_silent_t 8y ago
I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly. For example, if I'm working on a Python script under 1000 lines I would never use a debugger. I might use wireshark or linux system tools if something was really wrong, but usually the trackback or program output is sufficient. I've also tried to debug an Arduino project where I had a lot less knowledge of how the system works, and if I wasn't able to attach a debugger I don't want to know how long it would have taken me to find the problem (calling a null function pointer causes an interrupt that didn't have any handler). I've also tried to debug other people's projects in high level languages, without a debugger where after taking about an hour to figure it out based on pure reason, realizing it would have been very quick and easy to find with a debugger. I think if you look at what a program is doing, and at that point you can't form a testable hypothesis as to what is happening, at that point a debugger will certainly be faster if you know how to use it.
- brokenmachine 8y ago
  How did you use a debugger on Arduino? I've been using Serial.print("WTF") more than I'd admit.
  - strong_silent_t 8y ago
    This is similar to what I did: https://learn.adafruit.com/proper-step-debugging-atsamd21-arduino-zero-m0/overview https://learn.adafruit.com/proper-step-debugging-atsamd21-ar... . That is for that specific platform, not sure if there is something similar for the other versions. I used the J-Link but I wanted to do it on Linux, so I used the JLink GDB Server along with the GDB build that comes with the Arduino IDE. It probably takes me a good 5 minutes to get everything set up, but the basic stuff works: breakpoints, stack trace, reading and writing memory locations. The setup is a little convoluted, I'm not sure if there is a better way, but here are my notes: Procedure for running w/ debugger attached. 1. disconnect debugger's usb 2. connect device via usb 3. arduino IDE program device 4. identify elf, set up gdb with elf. 5. plug in debugger 6. run jlink gdb server 7. attach gdb to jlink server "target remote:2331" 8. "monitor reset", "continue" to re-run target from beginning. 9. observe device is recognized by Arduino IDE 10. open Arduino IDE serial monitor to interact with device. Now, Ctrl+C in GDB halts device, and stack etc. can be inspected, and then it can continue running with "cont". To rerun the device with the same program without redoing everything, go to step 8 and reset the device. and continue from there. To find .elf on linux: ================================================== find /tmp -name sketch_name.ino.elf 2>/dev/null gdb location: ================================================== /home/user/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gdb JLink setup command: ================================================== JLinkGDBServer -device ATSAMD21G18A -if SWD -speed 4000 gdb attachment: ================================================== (gdb) target remote:2331
remoteorbust 8y ago
I find that I don't need a debugger until I do. People are always asking me what debugger I use for go and I tell them I've gone months without feeling the need for them. However every once in a while I need to use gdb + remote debugging features to figure out a particularly weird runtime issue and I'm glad it's there. Then I can often take the odd values I see in the debugger and start putting them into a test or a minimal repro.
- pmarreck 8y ago
  This is the ideal use case IMHO
asrp 8y ago
I don't have that much to add to the other replies here. I used to never use pdb in Python for many years (not knowing or forgotten it exists) and doing perfectly well. Then I started using post-mortem debugging and finding/fixing bugs got so much faster. Basically, I'm using the computer to do the simulations I previously ran in my head (by evaluating things here and there). But that's just one point of data. Another debugger I now recommend people (in general) to look at is Squeak Smalltalk's debugger which allows a form of "top down" programming where undefined functions are called and filled in at runtime. The debugger/editor in this post borrows a little bit from that idea in spirit. Part 3 in particular shows how it can be used even when they are no bugs. About being off-topic, well, no-one has made another top level comment and by the time of your first post it was already 5 hours in. Some discussion is definitely better than none. So thank you for inciting that. If it burried some more on-topic discussion, that would be unfortunate though.
- pmarreck 8y ago
  Ha, thanks for that!
busterarm 8y ago
Having access to a debugger when you need it is a necessity. Rampant use of it can, debatably, be seen as a crutch...but if you need it and you don't have it, you're going to have an extremely bad week.