5 ms·

Some of this debugging is, per the title, "extreme" (trying to "rewind" the realities of source behavior through time until things start to click, dealing with

Some of this debugging is, per the title, "extreme" (trying to "rewind" the realities of source behavior through time until things start to click, dealing with ancient binaries, using disassembly/binary analysis tools to hack up branching logic). However, a lot of this is using tools that I'd expect everyone (even browser JS programmers, Java programmers, folks that write code in higher level languages) to have at least a passing familiarity with: strace/ltrace, nm/ldd, gdb...no matter who you are or what you do, there's immense benefit in trying to debug your programs like they're closed-source[1]. I'm perennially disappointed to find that extremely skilled and intelligent colleagues (far better at writing and designing code than me) are routinely stymied when asked to figure out why their code is acting a certain way when it runs--why a behavior of their code is occurring in an environment where their typical tools (language-specific debuggers, IDEs, etc.) aren't available. I think this is a skill-gap that our iteration-focused, "just try shit until it works" industry has instilled in a generation of programmers to their detriment. This is super important stuff--not just for debugging ability, but because understanding at least the "shallow end" of runtime-analysis tools (and with eBPF and core analysis tools, to say nothing of actual embedded/kernel programming, the pool gets a lot deeper, if you want) makes you become a way better programmer! This isn't only for folks who write code in runtime-less languages (and don't come at me about crt0); even people who write Python for a living (like me!), get bloody superpowers from understanding things like kernel memory management, syscall semantics below the standard library, and garbage collection behavior. The folks I find that don't have those skills aren't stupid or talentless, but rather victims of a series of roles and managers that have instilled a kind of Stockholm Syndrome of "when these things fail, there's nothing you can do about it, just consider it spooky and move on with your life", when getting at what's really going on is not just easy to do, but super beneficial for your career development! Put another way, we're all systems programmers in our day jobs--some of us are just in denial. /rant 1. https://jvns.ca/blog/2014/04/20/debug-your-programs-like-theyre-closed-source/ https://jvns.ca/blog/2014/04/20/debug-your-programs-like-the...

yamrzou 6y ago
How does one go about learning these kinds of skills?
- ruslan 6y ago
  Pratice. strace and gdb are your friends in catching evasive bugs. Personally I would go exactly same route, though I won't modify the binary but coined an environment for it to work instead.
- tonyarkles 6y ago
  I, personally, learned gradually how to use them by refusing to accept a bug fix when I didn’t understand the bug I was fixing. Gdb was first, because when I learned C it was the only freely available option. Strace likely was next when I started encountering problems that happened too long into a program’s execution to successfully use breakpoints to debug. Valgrind was likely next, when I ran into bugs where the memory corruption happened a long time before the actual crash (think use-after-free or uninitialized values).
  - gfxgirl 6y ago
    Every time I touch gdb I curse, "why isn't this easier to inspect code, watch variables, have multiple views, etc, like Visual Studio's debugger since 1995"
    - teddyh 6y ago
      There are numerous front-ends which does that.
- b0afc375b5 6y ago
  I was curious as well, so I'm currently reading this: https://dev.to/captainsafia/say-this-five-times-fast-strace-ptrace-dtrace-dtruss-3e1b https://dev.to/captainsafia/say-this-five-times-fast-strace-... where I stumbled upon and read the ezine: https://jvns.ca/blog/2015/04/14/strace-zine/ https://jvns.ca/blog/2015/04/14/strace-zine/
- zbentley 6y ago
  Speaking very personally and not axiomatically: by refusing to accept "fixes" that aren't fully understood. If that means rejecting the PR "stop using globals in $language because that triggers a 'slab allocation' (whatever that is) which stackoverflow says is bad", and instead insisting to your boss that you spend an extra week reading kernel source/stdlib source instead of moving on from the original 2-hour-scoped bug, so be it. The outcome is far more beneficial (for you and for the product/company) in the long term.
barrkel 6y ago
You won't have a good time trying to debug a JVM or Node JS runtime with gdb. There are too many threads, too many moving parts, and trying to stop the world to make sense of it is too invasive; you'll muck with timings and waits and timeouts and GC thresholds and all sorts, and your effort to figure out what's going on will have changed what's going on. And that's if gdb doesn't crash. IMO not enough people download and build the source. It's primitive, but adding logging or printfs to existing source, combined with strace if necessary, can be effective in a brutal way, where gdb - or any debugger - wouldn't get you far.
- sgc 6y ago
  I've always wondered if there shouldn't be a VSCode or other plugin that allows you to automatically throw in a bunch of printfs up a dependency chain to dump vars etc, and remove them just as easily.
  - usefulcat 6y ago
    I'm not sure that would be as useful as it sounds. A big part of what you get from manually adding printfs is familiarity with the code you're instrumenting.
  - user5994461 6y ago
    that's called a debugger, put a breakpoint somewhere, the application will pause right there and show you all the variables.
    - barrkel 6y ago
      Funny thing about debuggers is that they seem to encourage stepping - step in, step out, step over, single step, step to return. Stepping is an enormous waste of time with a debugger. Breakpoints are where it's at. Level up your breakpoints: logging breakpoints, conditional breakpoints, memory breakpoints (super handy when combined with deterministic memory allocation), syscall / library breakpoints - and soon you're going somewhere useful. Debugging is about formulating theories and proving them wrong, or right. Stepping doesn't scale; breakpoints can scale, if you know where to put them. I'm not sure that the gap between a debugger with breakpoints, and printfs in source, is that large, though. It's easier to write complex conditionals or log the state of a complex predicate in the source language than in whatever the debugger gives you - even if it gives you the native language, you typically have to write it all on a single line. And you need to dive deep into the source to understand where to put those breakpoints - or printfs. An underused debugging tool is an instrumenting profiler. The captured call graph is a useful clue for dynamic & indirect control flow in particular, without needing to decipher it from stepping or infer it from chasing source references and indirections. Things like generic message dispatchers and event callbacks are horrible to try and step through and aren't much better when eyeballing the source.
wdb 6y ago
I am not aware of strace/gdb/ldd working alternative for Windows. Still should look into strace looks really powerful
- user5994461 6y ago
  windows has visual studio, whose debugger is light years ahead of gdb. to capture syscall, there are procmon/regmon/sysmon from sysinternals, that are also light years ahead of anything available on linux and come with an amazing UI. for ldd I am not sure what it would be for, windows is not quite like linux when it comes to dynamic libraries. there is depends.exe from visual studio and there was a nice import analyzer tool I forgot the name.
TeMPOraL 6y ago
Seconding the post. The post focuses on strace & friends in the Linux/Mac world; for Windows, a powerful tool to look into is Event Tracing for Windows, which lets you trace events, syscalls and performance counters for everything that's running. I.e. a global trace of all (or at least most) processes. Two main tools to look into are xperf, the cmdline tool for configuring and running trace sessions, and Windows Performance Analyzer, a GUI app for exploring recorded traces (additionally, there's Windows Performance Recorder, which seems to be an official GUI for xperf, but I haven't used it myself yet). This is essentially a superpower that's even less known than strace is by Linux devs. I've only learned the basics quite recently, and already used it to debug two hairy issues in few hours, that have already taken a man-month worth of time and would likely take another one. Being able to explore the recordings of the process tree evolution, see all file IO operations, their times and targets and relation to everything else happening in the system, all in both visual and tabular form - it gives you a perspective you otherwise can't get. (An unfortunate side effect is that the tough bugs, like multiprocess race conditions or silent crashes, all seem to be directed my way nowadays.) Unfortunately, xperf and WPA don't come pre-installed in most Windows distributions; the fastest way to get them is through UIforETW[0] - Google's UI for xperf that can automatically locate and install the correct version of Windows Performance Toolkit[1]. Bruce Dawson (of Google Chrome fame) wrote a lot about advanced usage of this tooling, see: https://randomascii.wordpress.com/2015/09/24/etw-central/ https://randomascii.wordpress.com/2015/09/24/etw-central/. -- [0] - https://github.com/google/UIforETW/releases https://github.com/google/UIforETW/releases - fetch the newest .zip package with binaries. [1] - Alternative approach, if you need xperf on a computer with no Internet access (happened to me the other day with a customer's VM), you need to download the Windows SDK ISO (e.g. https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/ https://developer.microsoft.com/en-us/windows/downloads/wind...), mount it, and grab the Windows Performance Toolkit installer from the mounted image.
- cinntaile 6y ago
  > (An unfortunate side effect is that the tough bugs, like multiprocess race conditions or silent crashes, all seem to be directed my way nowadays.) The trick is to never reveal your superpower! The article was a delight to read, it's definitely an area where I am lacking and it seems fairly difficult to pick up this skill through a curriculum.

yamrzou 6y ago

How does one go about learning these kinds of skills?

ruslan 6y ago
Pratice. strace and gdb are your friends in catching evasive bugs. Personally I would go exactly same route, though I won't modify the binary but coined an environment for it to work instead.
tonyarkles 6y ago
I, personally, learned gradually how to use them by refusing to accept a bug fix when I didn’t understand the bug I was fixing. Gdb was first, because when I learned C it was the only freely available option. Strace likely was next when I started encountering problems that happened too long into a program’s execution to successfully use breakpoints to debug. Valgrind was likely next, when I ran into bugs where the memory corruption happened a long time before the actual crash (think use-after-free or uninitialized values).
- gfxgirl 6y ago
  Every time I touch gdb I curse, "why isn't this easier to inspect code, watch variables, have multiple views, etc, like Visual Studio's debugger since 1995"
  - teddyh 6y ago
    There are numerous front-ends which does that.
b0afc375b5 6y ago
I was curious as well, so I'm currently reading this: https://dev.to/captainsafia/say-this-five-times-fast-strace-ptrace-dtrace-dtruss-3e1b https://dev.to/captainsafia/say-this-five-times-fast-strace-... where I stumbled upon and read the ezine: https://jvns.ca/blog/2015/04/14/strace-zine/ https://jvns.ca/blog/2015/04/14/strace-zine/
zbentley 6y ago
Speaking very personally and not axiomatically: by refusing to accept "fixes" that aren't fully understood. If that means rejecting the PR "stop using globals in $language because that triggers a 'slab allocation' (whatever that is) which stackoverflow says is bad", and instead insisting to your boss that you spend an extra week reading kernel source/stdlib source instead of moving on from the original 2-hour-scoped bug, so be it. The outcome is far more beneficial (for you and for the product/company) in the long term.

barrkel 6y ago

You won't have a good time trying to debug a JVM or Node JS runtime with gdb. There are too many threads, too many moving parts, and trying to stop the world to make sense of it is too invasive; you'll muck with timings and waits and timeouts and GC thresholds and all sorts, and your effort to figure out what's going on will have changed what's going on. And that's if gdb doesn't crash. IMO not enough people download and build the source. It's primitive, but adding logging or printfs to existing source, combined with strace if necessary, can be effective in a brutal way, where gdb - or any debugger - wouldn't get you far.

sgc 6y ago
I've always wondered if there shouldn't be a VSCode or other plugin that allows you to automatically throw in a bunch of printfs up a dependency chain to dump vars etc, and remove them just as easily.
- usefulcat 6y ago
  I'm not sure that would be as useful as it sounds. A big part of what you get from manually adding printfs is familiarity with the code you're instrumenting.
- user5994461 6y ago
  that's called a debugger, put a breakpoint somewhere, the application will pause right there and show you all the variables.
  - barrkel 6y ago
    Funny thing about debuggers is that they seem to encourage stepping - step in, step out, step over, single step, step to return. Stepping is an enormous waste of time with a debugger. Breakpoints are where it's at. Level up your breakpoints: logging breakpoints, conditional breakpoints, memory breakpoints (super handy when combined with deterministic memory allocation), syscall / library breakpoints - and soon you're going somewhere useful. Debugging is about formulating theories and proving them wrong, or right. Stepping doesn't scale; breakpoints can scale, if you know where to put them. I'm not sure that the gap between a debugger with breakpoints, and printfs in source, is that large, though. It's easier to write complex conditionals or log the state of a complex predicate in the source language than in whatever the debugger gives you - even if it gives you the native language, you typically have to write it all on a single line. And you need to dive deep into the source to understand where to put those breakpoints - or printfs. An underused debugging tool is an instrumenting profiler. The captured call graph is a useful clue for dynamic & indirect control flow in particular, without needing to decipher it from stepping or infer it from chasing source references and indirections. Things like generic message dispatchers and event callbacks are horrible to try and step through and aren't much better when eyeballing the source.

wdb 6y ago

I am not aware of strace/gdb/ldd working alternative for Windows. Still should look into strace looks really powerful

user5994461 6y ago
windows has visual studio, whose debugger is light years ahead of gdb. to capture syscall, there are procmon/regmon/sysmon from sysinternals, that are also light years ahead of anything available on linux and come with an amazing UI. for ldd I am not sure what it would be for, windows is not quite like linux when it comes to dynamic libraries. there is depends.exe from visual studio and there was a nice import analyzer tool I forgot the name.

TeMPOraL 6y ago

Seconding the post. The post focuses on strace & friends in the Linux/Mac world; for Windows, a powerful tool to look into is Event Tracing for Windows, which lets you trace events, syscalls and performance counters for everything that's running. I.e. a global trace of all (or at least most) processes. Two main tools to look into are xperf, the cmdline tool for configuring and running trace sessions, and Windows Performance Analyzer, a GUI app for exploring recorded traces (additionally, there's Windows Performance Recorder, which seems to be an official GUI for xperf, but I haven't used it myself yet). This is essentially a superpower that's even less known than strace is by Linux devs. I've only learned the basics quite recently, and already used it to debug two hairy issues in few hours, that have already taken a man-month worth of time and would likely take another one. Being able to explore the recordings of the process tree evolution, see all file IO operations, their times and targets and relation to everything else happening in the system, all in both visual and tabular form - it gives you a perspective you otherwise can't get. (An unfortunate side effect is that the tough bugs, like multiprocess race conditions or silent crashes, all seem to be directed my way nowadays.) Unfortunately, xperf and WPA don't come pre-installed in most Windows distributions; the fastest way to get them is through UIforETW[0] - Google's UI for xperf that can automatically locate and install the correct version of Windows Performance Toolkit[1]. Bruce Dawson (of Google Chrome fame) wrote a lot about advanced usage of this tooling, see: https://randomascii.wordpress.com/2015/09/24/etw-central/ https://randomascii.wordpress.com/2015/09/24/etw-central/. -- [0] - https://github.com/google/UIforETW/releases https://github.com/google/UIforETW/releases - fetch the newest .zip package with binaries. [1] - Alternative approach, if you need xperf on a computer with no Internet access (happened to me the other day with a customer's VM), you need to download the Windows SDK ISO (e.g. https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/ https://developer.microsoft.com/en-us/windows/downloads/wind...), mount it, and grab the Windows Performance Toolkit installer from the mounted image.

cinntaile 6y ago
> (An unfortunate side effect is that the tough bugs, like multiprocess race conditions or silent crashes, all seem to be directed my way nowadays.) The trick is to never reveal your superpower! The article was a delight to read, it's definitely an area where I am lacking and it seems fairly difficult to pick up this skill through a curriculum.