4 ms·

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “

Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone. Though abs() returning negative numbers is hilarious.. “You had one job…” To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. I’m not just talking about concurrency issues either… The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. 2 days is cute though.

jakevoytko 2y ago
Author here! I debugged a fair number of those when I was a systems engineer in soft real time robotics systems, but none of them felt as bad in retrospect because you're just reading up on the system and mulling over it and eventually you get the answer in a shower thought. Maybe I just find the puzzle of them fun, I don't know why they don't feel quite so bad. This was just an exhausting 2-day brute-force grind where it turned out the damn compiler was broken.
- gertlex 2y ago
  I also came to the comments to weigh in on my perception of how rough this was, but instead will ask: Regarding "exhausting 2-day brute-force grind": is/was this just how you like to get things done, or was there external pressure of the "don't work on anything else" sort? I've never worked at a large company, and lots of descriptions of the way things get done are pretty foreign to me :). I am also used to being able to say "this isn't getting figured out today; probably going to be best if I work on something else for a bit, and sleep on it, too".
  - jakevoytko 2y ago
    The fatal error volume was so overwhelming that we didn't have any option but understanding the problem in perfect detail so that we could fix it if the problem was on our side, or avoid it if it was caused by something like our compiler or the browser. Our team also had a very grindy culture, so "I'm going to put in extra hours focusing exclusively on our top crash" was a pretty normalized behavior. After I left that team (and Google), most of my future teams have been more forgiving on pace for non-outages.
    - gertlex 2y ago
      That makes sense. Thanks for the extra info!
    - pas 2y ago
      Was there an option to roll back Google Docs to an earlier release? Or do shadow release A/B testing for a fraction of users? (And thanks for the war story!)
userbinator 2y ago
The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had. One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare ones which do appear are also extremely difficult to debug. The 2-hour, 2-day, and 2-week bugs have long been debugged out already.
- gmueckl 2y ago
  That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brave field technician in Finland in winter, inspecting a unit in an unheated space to even discover this particular bug because the units' internal temperatures were usually about 20C above ambient.
  - treyd 2y ago
    This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.
    - selimthegrim 2y ago
      I have seen this on Walgreens signs in suburban New Orleans, oddly enough.
    - shakna 2y ago
      The rollout is still happening, but the new resident water meters for Victoria, Australia come with a temperature fix. Prior to this year, they could only handle 0-127 degrees for the water temperature. Which used to be sensible, but there were some issues with pressurised water starting to be delivered to houses resulting in negative temperatures being reported, like -125C, which immediately has the water switch off to prevent icing problems. The software side also switched from COBOL to Ada. So that's kewl.
Terr_ 2y ago
This repro was a few times per day, but try fixing a Linux kernel panic when you don't even have C/C++ on your resume, and everyone who originally set stuff up has left... https://news.ycombinator.com/item?id=37859771 https://news.ycombinator.com/item?id=37859771 Point being that the difficulty of a fix can come from many possible places.
efortis 2y ago
Same here, we had an IE8 bug that prevented the initial voice over of the screen reader (JAWS). No dev could reproduce it because we all had DevTools open.
- smrq 2y ago
  I can't remember the actual bug now, but one of my early career memories was hunting down an IE7 issue by using bookmarklets to alert() values. (Did IE7 even have dev tools?)
  - camtarn 2y ago
    There was a downloadable developer toolbar for IE6 and IE7, and scripts could be debugged in the external Windows Script Debugger. The developer toolbar even told you which elements had the famous hasLayout attribute applied, which completely changed how it was rendered and interacted with other objects, which was invaluable.
- gsck 2y ago
  I had a similar issue, worked fine when I was testing it on my machine, but I had dev tools open to see any potential issues. Turns out IE8 doesn't define console until the devtools are open. That caused me to pull a few hairs out.
jffhn 2y ago
>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.
- eterm 2y ago
  You inspired me to check what .NET does in that situation. It throws an OverflowException: ("Negating the minimum value of a twos complement number is invalid.")
- adrian_b 2y ago
  Unchecked integer overflow strikes again.
- rhaps0dy 2y ago
  Oh no, Pytorch does the same thing: a = torch.tensor(-2*31, dtype=torch.int32) assert a == a.abs()
  - MawKKe 2y ago
    numpy as well. and tensorflow
- winstonbwillis 2y ago
  [dead]
- bobbylarrybobby 2y ago
  Rust does the same in release, although it panics in debug.
lukan 2y ago
"To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added." My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time). Feels like chasing ghosts.
- btschaegg 2y ago
  Terminology proposal: "Gremlins" :)
rowanG077 2y ago
I don't think taking how long something took to debug in number of days is at all interesting. Trivial bugs can take weeks to debug for a noob. Insanely hard bugs takes hours to debug for genius devs, maybe even without any reproducer, just by thinking about it.
dismalpedigree 2y ago
I always refer to them as “quantum bugs” because the act of observing the bug changes the bug. Absolutely infuriating. I like “heisenbug” better. Has a better ring to it.
steveBK123 2y ago
Yes ! I've dealt with complex issues that turned out to be vendor-swapped-hardware-woopsie which we spent over a month trying to solve in software before finally figuring it out. Part of it was difficulty of pinpointing the actual issue - fullness of drive vs throughput of writes. A lot of it was unfortunately organizational politics such that the system spanned two teams with different reporting lines that didn't cooperate well / had poor testing practices.
- voidifremoved 2y ago
  > A lot of it was unfortunately organizational politics The hardest bugs in my experience are those where your only source of vital information is a third party who is straight-up lying to you.
  - GianFabien 2y ago
    Sometimes it isn't outright lying. I have had the issues with hardware, API and SDK documentation being subtly different from the product as shipped. With hardware with a mixture of revisions, some conforming to doco and other differing and even their engineers not being clear about which is which.
yfontana 2y ago
I think the hardest bug I've had to work on was just plain irreproducible. Once we exhausted all other ideas, we just attributed it to some sort of bit flip. Not a very satisfying resolution, but kinda cool to have encountered such an issue at least once.
sesm 2y ago
For stuff like this we used in-memory ring buffer logger that printed the logs on request. And it didn't save the strings, just necessary data bits and a pointer to formatting function. Writing to this logger didn't affect any timings.
mystified5016 2y ago
In hardware, you regularly see behavior change when you probe the system. Your oscilloscope or LA probes affect the system just enough to make a marginal circuit work. It's absolutely maddening.
- fuzzfactor 2y ago
  The closer you get to natural science, eventually reliance on logical troubleshooting can be "illogical". The more abundant the undefined (mis)behavior, the more you're going to be tearing your hair out. Almost the kind of frustration where you're supposed to have a logic-based system, and it rears it ugly head and defies logic anyway :\
Adverblessly 2y ago
> To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.

jakevoytko 2y ago

Author here! I debugged a fair number of those when I was a systems engineer in soft real time robotics systems, but none of them felt as bad in retrospect because you're just reading up on the system and mulling over it and eventually you get the answer in a shower thought. Maybe I just find the puzzle of them fun, I don't know why they don't feel quite so bad. This was just an exhausting 2-day brute-force grind where it turned out the damn compiler was broken.

gertlex 2y ago
I also came to the comments to weigh in on my perception of how rough this was, but instead will ask: Regarding "exhausting 2-day brute-force grind": is/was this just how you like to get things done, or was there external pressure of the "don't work on anything else" sort? I've never worked at a large company, and lots of descriptions of the way things get done are pretty foreign to me :). I am also used to being able to say "this isn't getting figured out today; probably going to be best if I work on something else for a bit, and sleep on it, too".
- jakevoytko 2y ago
  The fatal error volume was so overwhelming that we didn't have any option but understanding the problem in perfect detail so that we could fix it if the problem was on our side, or avoid it if it was caused by something like our compiler or the browser. Our team also had a very grindy culture, so "I'm going to put in extra hours focusing exclusively on our top crash" was a pretty normalized behavior. After I left that team (and Google), most of my future teams have been more forgiving on pace for non-outages.
  - gertlex 2y ago
    That makes sense. Thanks for the extra info!
  - pas 2y ago
    Was there an option to roll back Google Docs to an earlier release? Or do shadow release A/B testing for a fraction of users? (And thanks for the war story!)

userbinator 2y ago

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently. The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had. One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare ones which do appear are also extremely difficult to debug. The 2-hour, 2-day, and 2-week bugs have long been debugged out already.

gmueckl 2y ago
That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago. The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brave field technician in Finland in winter, inspecting a unit in an unheated space to even discover this particular bug because the units' internal temperatures were usually about 20C above ambient.
- treyd 2y ago
  This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.
  - selimthegrim 2y ago
    I have seen this on Walgreens signs in suburban New Orleans, oddly enough.
  - shakna 2y ago
    The rollout is still happening, but the new resident water meters for Victoria, Australia come with a temperature fix. Prior to this year, they could only handle 0-127 degrees for the water temperature. Which used to be sensible, but there were some issues with pressurised water starting to be delivered to houses resulting in negative temperatures being reported, like -125C, which immediately has the water switch off to prevent icing problems. The software side also switched from COBOL to Ada. So that's kewl.

Terr_ 2y ago

This repro was a few times per day, but try fixing a Linux kernel panic when you don't even have C/C++ on your resume, and everyone who originally set stuff up has left... https://news.ycombinator.com/item?id=37859771 https://news.ycombinator.com/item?id=37859771 Point being that the difficulty of a fix can come from many possible places.

efortis 2y ago

Same here, we had an IE8 bug that prevented the initial voice over of the screen reader (JAWS). No dev could reproduce it because we all had DevTools open.

smrq 2y ago
I can't remember the actual bug now, but one of my early career memories was hunting down an IE7 issue by using bookmarklets to alert() values. (Did IE7 even have dev tools?)
- camtarn 2y ago
  There was a downloadable developer toolbar for IE6 and IE7, and scripts could be debugged in the external Windows Script Debugger. The developer toolbar even told you which elements had the famous hasLayout attribute applied, which completely changed how it was rendered and interacted with other objects, which was invaluable.
gsck 2y ago
I had a similar issue, worked fine when I was testing it on my machine, but I had dev tools open to see any potential issues. Turns out IE8 doesn't define console until the devtools are open. That caused me to pull a few hairs out.

jffhn 2y ago

>Though abs() returning negative numbers is hilarious. Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.

eterm 2y ago
You inspired me to check what .NET does in that situation. It throws an OverflowException: ("Negating the minimum value of a twos complement number is invalid.")
adrian_b 2y ago
Unchecked integer overflow strikes again.
rhaps0dy 2y ago
Oh no, Pytorch does the same thing: a = torch.tensor(-2*31, dtype=torch.int32) assert a == a.abs()
- MawKKe 2y ago
  numpy as well. and tensorflow
winstonbwillis 2y ago
[dead]
bobbylarrybobby 2y ago
Rust does the same in release, although it panics in debug.

lukan 2y ago

"To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added." My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time). Feels like chasing ghosts.

btschaegg 2y ago
Terminology proposal: "Gremlins" :)

rowanG077 2y ago

I don't think taking how long something took to debug in number of days is at all interesting. Trivial bugs can take weeks to debug for a noob. Insanely hard bugs takes hours to debug for genius devs, maybe even without any reproducer, just by thinking about it.

dismalpedigree 2y ago

I always refer to them as “quantum bugs” because the act of observing the bug changes the bug. Absolutely infuriating. I like “heisenbug” better. Has a better ring to it.

steveBK123 2y ago

Yes ! I've dealt with complex issues that turned out to be vendor-swapped-hardware-woopsie which we spent over a month trying to solve in software before finally figuring it out. Part of it was difficulty of pinpointing the actual issue - fullness of drive vs throughput of writes. A lot of it was unfortunately organizational politics such that the system spanned two teams with different reporting lines that didn't cooperate well / had poor testing practices.

voidifremoved 2y ago
> A lot of it was unfortunately organizational politics The hardest bugs in my experience are those where your only source of vital information is a third party who is straight-up lying to you.
- GianFabien 2y ago
  Sometimes it isn't outright lying. I have had the issues with hardware, API and SDK documentation being subtly different from the product as shipped. With hardware with a mixture of revisions, some conforming to doco and other differing and even their engineers not being clear about which is which.

yfontana 2y ago

I think the hardest bug I've had to work on was just plain irreproducible. Once we exhausted all other ideas, we just attributed it to some sort of bit flip. Not a very satisfying resolution, but kinda cool to have encountered such an issue at least once.

sesm 2y ago

For stuff like this we used in-memory ring buffer logger that printed the logs on request. And it didn't save the strings, just necessary data bits and a pointer to formatting function. Writing to this logger didn't affect any timings.

mystified5016 2y ago

In hardware, you regularly see behavior change when you probe the system. Your oscilloscope or LA probes affect the system just enough to make a marginal circuit work. It's absolutely maddening.

fuzzfactor 2y ago
The closer you get to natural science, eventually reliance on logical troubleshooting can be "illogical". The more abundant the undefined (mis)behavior, the more you're going to be tearing your hair out. Almost the kind of frustration where you're supposed to have a logic-based system, and it rears it ugly head and defies logic anyway :\

Adverblessly 2y ago

> To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added. A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.