4 ms·

From the article: Here’s my one simple rule for debugging: Reproduce the issue. Unless it's one of those cursed things installed at the customer thousa

From the article: Here’s my one simple rule for debugging: Reproduce the issue. Unless it's one of those cursed things installed at the customer thousands of miles away that never happens back in the lab. Some things can be incredibly hard to debug and can depend on the craziest of things you'd never even consider possible. Like a thunderstorm causing voltage spikes that very subtly damage the equipment causing subtle failures a few months later. Sometimes that "software bug" turns out to be hardware in weird ways. Or issues like https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles – every person who's debugging weird issues should read that. Once you can actually reproduce the issue, you've often done 80-99+% of the work already.

virgilp 2y ago
It is true that for many (especially concurrency/distributed-systems-related) bugs, reproducing the issue might be the hardest part... but that's not always true. Long time ago, I was working on a C++ compiler for an embedded processor. Customer complains that when they turn on optimization, code fails. 100% reproducible, just put "-O3" and it fails, with "-O0" it works. Now, we're used to these bug reports, it's often a bug in the original software (like relying on undefined behaviour), but we manage to get the code, it looks good / we don't find anything strange. Can't run it, it only works on the customer board (which is their custom proprietary hardware, using our microprocessor). We look through the assembly, can't find any obvious optimization error that the compiler made. After much (remote) debugging, it turns out that it was a fault in their memory system.
- vardump 2y ago
  These kind of things happen all the time in embedded development. Tough to debug, because you can't initially know whether it's software or hardware.
vrighter 2y ago
had some hp microservers once, of which some, but not all, got occasional bit corruptions due to emf interference from themselves.
dexen 2y ago
>Unless it's one of those cursed things installed at the customer thousands of miles away that never happens back in the lab. I had a bug like that in my previous (telcom embedded dev) career. Ended up driving to the customer premises (luckily mere 200km), working two weeks on collecting traces for the repro, and a day on the patch. Once I figured out how to trace the repro, the rest was trivial - the bug was glaringly obvious in the trace. Which in hindsight means I didn't really need to drive there at all, I merely needed to properly implement tracing the repro, and send the built artifact to the customer. The problem would have been trivially solved if I had sufficient experience with tracing, or found a colleague with sufficient experience. However this one time the experience was bought & paid for with the trip.
- vardump 2y ago
  > Which in hindsight means I didn't really need to drive there at all, I merely needed to properly implement tracing the repro, and send the built artifact to the customer. In the alternate reality where you chose not to drive there, you'd now be complaining how it took 3 weeks to troubleshoot with the customer and how you should have really driven there instead.
metaphor 2y ago
> Unless it's one of those cursed things installed at the customer thousands of miles away that never happens back in the lab. > Once you can actually reproduce the issue, you've often done 80-99+% of the work already. War story... I spent over a year on-and-off between adjacent sprints circa peak COVID chasing an elusive intermittent bug that kept coming back after failing formal acceptance testing performed at a designated off-site location. Despite requesting that I be flown to this location to personally replicate the issue within days of the initial report (unable to replicate in local lab and familiar enough with the system's design to intuit the superficially unbounded depth of the rabbit hole) and every subsequent failed test report thereafter, PM persistently decided to allocate funds elsewhere and just kept kicking this product to the back of the pipeline. Nearing project end as the last backlogged issue in queue, I managed to isolate the root cause to a race condition in a firmware ISR of an embedded microcontroller buried 3 levels deep in mixed-signal hardware. This subroutine's execution path interacted with an async signal from an instrument that was occasionally ~10 ms slower between bursts than any of the references we had in our lab...which really shouldn't have mattered because the entire flow control setup was designed to be timing insensitive and executed by non-realtime host...except this one buried onion subroutine that really wasn't. Attaining local replication was an imperative, which required some creative speculation and a kludge hardware mock, but once I got to a state of repeatably mirroring the failure mode at will, the problem was effectively 95% solved; what followed was a quick subroutine rewrite, anti-pattern scrubdown review, host software update to account for new payload, and instructing off-site techs to field reprogram before the next scheduled test event. I'd never been happier to tag an issue resolved and finally merge an all but abandoned WIP branch. To this day, I still don't know with certainty what caused that catalyst instrument to occasionally perform ~10 ms slower relative to our references. In the end, we indirectly absorbed at least 50x the notional cost of that site visit when the dust finally settled.
- catskull 2y ago
  The deeper you get debugging, the more you have to reverse engineer, modify, even outright hack the target. Awesome war story, I know how good that must have felt.

virgilp 2y ago

It is true that for many (especially concurrency/distributed-systems-related) bugs, reproducing the issue might be the hardest part... but that's not always true. Long time ago, I was working on a C++ compiler for an embedded processor. Customer complains that when they turn on optimization, code fails. 100% reproducible, just put "-O3" and it fails, with "-O0" it works. Now, we're used to these bug reports, it's often a bug in the original software (like relying on undefined behaviour), but we manage to get the code, it looks good / we don't find anything strange. Can't run it, it only works on the customer board (which is their custom proprietary hardware, using our microprocessor). We look through the assembly, can't find any obvious optimization error that the compiler made. After much (remote) debugging, it turns out that it was a fault in their memory system.

vardump 2y ago
These kind of things happen all the time in embedded development. Tough to debug, because you can't initially know whether it's software or hardware.

vrighter 2y ago

had some hp microservers once, of which some, but not all, got occasional bit corruptions due to emf interference from themselves.

dexen 2y ago

>Unless it's one of those cursed things installed at the customer thousands of miles away that never happens back in the lab. I had a bug like that in my previous (telcom embedded dev) career. Ended up driving to the customer premises (luckily mere 200km), working two weeks on collecting traces for the repro, and a day on the patch. Once I figured out how to trace the repro, the rest was trivial - the bug was glaringly obvious in the trace. Which in hindsight means I didn't really need to drive there at all, I merely needed to properly implement tracing the repro, and send the built artifact to the customer. The problem would have been trivially solved if I had sufficient experience with tracing, or found a colleague with sufficient experience. However this one time the experience was bought & paid for with the trip.

vardump 2y ago
> Which in hindsight means I didn't really need to drive there at all, I merely needed to properly implement tracing the repro, and send the built artifact to the customer. In the alternate reality where you chose not to drive there, you'd now be complaining how it took 3 weeks to troubleshoot with the customer and how you should have really driven there instead.

metaphor 2y ago

> Unless it's one of those cursed things installed at the customer thousands of miles away that never happens back in the lab. > Once you can actually reproduce the issue, you've often done 80-99+% of the work already. War story... I spent over a year on-and-off between adjacent sprints circa peak COVID chasing an elusive intermittent bug that kept coming back after failing formal acceptance testing performed at a designated off-site location. Despite requesting that I be flown to this location to personally replicate the issue within days of the initial report (unable to replicate in local lab and familiar enough with the system's design to intuit the superficially unbounded depth of the rabbit hole) and every subsequent failed test report thereafter, PM persistently decided to allocate funds elsewhere and just kept kicking this product to the back of the pipeline. Nearing project end as the last backlogged issue in queue, I managed to isolate the root cause to a race condition in a firmware ISR of an embedded microcontroller buried 3 levels deep in mixed-signal hardware. This subroutine's execution path interacted with an async signal from an instrument that was occasionally ~10 ms slower between bursts than any of the references we had in our lab...which really shouldn't have mattered because the entire flow control setup was designed to be timing insensitive and executed by non-realtime host...except this one buried onion subroutine that really wasn't. Attaining local replication was an imperative, which required some creative speculation and a kludge hardware mock, but once I got to a state of repeatably mirroring the failure mode at will, the problem was effectively 95% solved; what followed was a quick subroutine rewrite, anti-pattern scrubdown review, host software update to account for new payload, and instructing off-site techs to field reprogram before the next scheduled test event. I'd never been happier to tag an issue resolved and finally merge an all but abandoned WIP branch. To this day, I still don't know with certainty what caused that catalyst instrument to occasionally perform ~10 ms slower relative to our references. In the end, we indirectly absorbed at least 50x the notional cost of that site visit when the dust finally settled.

catskull 2y ago
The deeper you get debugging, the more you have to reverse engineer, modify, even outright hack the target. Awesome war story, I know how good that must have felt.