4 ms·

> Here’s my one simple rule for debugging: > Reproduce the issue. > . . . > I’m not sure that I’ve ever worked on a hard problem . . . I agree, the author h

> Here’s my one simple rule for debugging: > Reproduce the issue. > . . . > I’m not sure that I’ve ever worked on a hard problem . . . I agree, the author has probably not worked on hard problems. There are many situations where either a) reproducing the problem is extremely elusive, or b) reproduction is easy, but the cause is especially elusive, or c) organizational issues prevent access to the system, to source code, or to people willing to investigate and/or solve the issue. Some examples: For A, the elusive reproduction, I saw an issue where we had an executive escalation that their laptop would always blue screen shortly after boot up. Nobody could reproduce this issue. Telemetry showed nobody else had this issue. Changing hardware didn't fix it. Only this executive had the anti-Midas touch to cause the issue. Turned out the executive lived on a large parcel, and had exactly one WiFi network visible. Some code parsing that list of WiFi APs had an off-by-one error which caused a BSOD. A large portion of wireless technology (Bluetooth/Thread/WiFi/cellular) bugs fall into this category. For B, the easy to repro but still difficult, I've seen numerous bugs that cause stack corruption, random memory corruption, or trigger a hardware flaw that freezes or resets the system. These types of issues are terrible to debug, because either the logs aren't available (system comes down before the final moments), or because the culprit doesn't log anything and never harms themselves, only an innocent victim. Time-travel tracing is often the best solution, but is also often unavailable. Bisecting the code changes is sometimes little help in a busy codebase, since the culprit is often far away from their victims. Category C is also pretty common if you are integrating systems. Vendors will have closed source and be unable or unwilling to admit even the possibility of fault, help with an investigation, or commit to a fix. Partners will have ship blocking bugs in hardware that they just can't show you or share with you, but it must nonetheless get fixed. You will often end up shipping workarounds for errors in code you don't control, or carefully instrumenting code to uncover the partner's issues.

forrestthewoods 2y ago
> I agree, the author has probably not worked on hard problems. Conversely, if you focus on creating a repro you make problems easy. I’m a little surprised the OP doesn’t mention debuggers. For whatever reason many modern Linux programmers seem to have never and used a debugger and rely almost entirely on printf. Which is utterly bonkers to me! If you can repro and issue and trap it in a debugger then you’ve done 90% of the work.
- glandium 2y ago
  And if you can repro in rr, you've done 99% of the work most of the time.
- vardump 2y ago
  Debuggers are not great when you have a bug that occurs between multiple CPU cores, hardware device timing, etc. In those cases, a debugger just can't stop the world, so back to printf it is.
  - forrestthewoods 2y ago
    printf also mucks with sensitive timing issues. So it’s not great either. Also time traveling debuggers like WinDbg and rr are the ideal way to debug complex timing issues.
    - vardump 2y ago
      > printf also mucks with sensitive timing issues. So it’s not great either. True, but this can be worked around for example with lock free ring buffers etc. > Also time traveling debuggers like WinDbg and rr are the ideal way to debug complex timing issues. When applicable rr can be great indeed. Not sure how to apply TTD tools for example in kernel context. Nice to see Microsoft continues improving WinDbg. (Shame Visual Studio's debugger is so weak in comparison.)
  - Neikius 2y ago
    Well I am aware there are ring 0 debuggers out there. Also running in a VM could help. Question is is there anything useful around you could just grab (for linux)? It should be possible at last.
pjc50 2y ago
My favourite case of this was simply "touchscreen sometimes doesn't work". That took over a year before it even made it to us developers, being reported intermittently across a large number of point of sale systems. Sensibly, people were reluctant to pass on something that they couldn't reproduce, and it was often blamed on hardware issues or user error, such as damage from people operating the (resistive) touchscreen with a bunch of keys or the smooth metal Dallas key fob that was also used for log in. We eventually got a robot to reproduce some other issues with PIN pads (which for obvious reasons do not support any kind of software injection of input!), and got some time using the robot to press the touchscreen over and over. That way we were able to confirm it happening, and start the debug process. (anticlimactically, I can't quite remember what the resolution was, something to do with debounce logic and interrupt events)
- vardump 2y ago
  > something to do with debounce logic and interrupt events Oh, the classic, an unfiltered hardware button connected to a GPIO that generates an IRQ? What could go wrong... :-D For the uninitiated: https://en.wikipedia.org/wiki/Switch#Contact_bounce https://en.wikipedia.org/wiki/Switch#Contact_bounce

forrestthewoods 2y ago

> I agree, the author has probably not worked on hard problems. Conversely, if you focus on creating a repro you make problems easy. I’m a little surprised the OP doesn’t mention debuggers. For whatever reason many modern Linux programmers seem to have never and used a debugger and rely almost entirely on printf. Which is utterly bonkers to me! If you can repro and issue and trap it in a debugger then you’ve done 90% of the work.

glandium 2y ago
And if you can repro in rr, you've done 99% of the work most of the time.
vardump 2y ago
Debuggers are not great when you have a bug that occurs between multiple CPU cores, hardware device timing, etc. In those cases, a debugger just can't stop the world, so back to printf it is.
- forrestthewoods 2y ago
  printf also mucks with sensitive timing issues. So it’s not great either. Also time traveling debuggers like WinDbg and rr are the ideal way to debug complex timing issues.
  - vardump 2y ago
    > printf also mucks with sensitive timing issues. So it’s not great either. True, but this can be worked around for example with lock free ring buffers etc. > Also time traveling debuggers like WinDbg and rr are the ideal way to debug complex timing issues. When applicable rr can be great indeed. Not sure how to apply TTD tools for example in kernel context. Nice to see Microsoft continues improving WinDbg. (Shame Visual Studio's debugger is so weak in comparison.)
- Neikius 2y ago
  Well I am aware there are ring 0 debuggers out there. Also running in a VM could help. Question is is there anything useful around you could just grab (for linux)? It should be possible at last.

pjc50 2y ago

My favourite case of this was simply "touchscreen sometimes doesn't work". That took over a year before it even made it to us developers, being reported intermittently across a large number of point of sale systems. Sensibly, people were reluctant to pass on something that they couldn't reproduce, and it was often blamed on hardware issues or user error, such as damage from people operating the (resistive) touchscreen with a bunch of keys or the smooth metal Dallas key fob that was also used for log in. We eventually got a robot to reproduce some other issues with PIN pads (which for obvious reasons do not support any kind of software injection of input!), and got some time using the robot to press the touchscreen over and over. That way we were able to confirm it happening, and start the debug process. (anticlimactically, I can't quite remember what the resolution was, something to do with debounce logic and interrupt events)

vardump 2y ago
> something to do with debounce logic and interrupt events Oh, the classic, an unfiltered hardware button connected to a GPIO that generates an IRQ? What could go wrong... :-D For the uninitiated: https://en.wikipedia.org/wiki/Switch#Contact_bounce https://en.wikipedia.org/wiki/Switch#Contact_bounce