11 ms·

I love the idea, but this line: > 1) no bug should take over 2 days Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, u

I love the idea, but this line: > 1) no bug should take over 2 days Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. Also, I tend to attack bugs by priority/severity, as opposed to difficulty. Some of the most serious bugs are often quite easy to find. Once I find the cause of a bug, the fix is usually just around the corner.

chii 1y ago
I find most bugs take less time to fix than it takes time to verify and reproduce.
- wahnfrieden 1y ago
  LLMs have helped me here the most. Adding copious detailed logging across the app on demand, then inspecting the logs to figure out the bug and even how to reproduce it.
  - ChrisMarshallNY 1y ago
    Yes. I often just copy the whole core dump, and feed it into the prompt.
    - Lionga 1y ago
      And this kids is how one bug got fixed and two more were created
      - ChrisMarshallNY 1y ago
        Nope. Good result == LLM + Experience. The LLM just reduces the overhead. That’s really what every “new paradigm” has ever done.
        enraged_camel 1y ago
        Also, robust test coverage helps prevent regressions.
      - Sohcahtoa82 1y ago
        There's a huge difference between using an LLM to assist you versus letting it just do all the work for you. Your implication that they're the same, and that the previous commenter let the LLM do the work, is lazy. ChrisMarshallNY only said they fed the dump into the LLM. They said nothing about using the LLM to write the fix.
    - criddell 1y ago
      This is something that I've been trying to improve at. I work on a Windows application and so I get crash dumps that I open with WinDbg and then I usually start looking for exceptions. Is this something an LLM could help with? What exactly do you mean when you say you feed a dump to the prompt?
      - ChrisMarshallNY 1y ago
        I literally copy the whole stack dump from the log, and paste it into the LLM (I find that ChatGPT does a better job than Claude), along with something along the lines of: > I am getting occasional crashes on my iOS 17 or above UIKit program. Given the following stack trace, what problem do think it might be? I will attach the source file, if I think I know the general area, along with any symptoms and steps to reproduce. One of the nice things about an LLM, is that it's difficult to overwhelm with too much information (unlike people). It will usually respond with a fairly detailed analysis. Usually, it has some good ideas to use as starting points. I don't think "I have a bug. Please fix it." would work, though. It's likely to try, but caveat emptor.
    - int_19h 1y ago
      I kinda wonder if at some point this is something we might use the LLM more directly for. As in, train them on raw binary dumps as input.
      - ChrisMarshallNY 11mo ago
        I wonder if we’ll be seeing tools that do this. I could see Apple or Microsoft, building it into their IDEs. But, as was noted elsewhere, I think it’s only useful as an advisor. I think a lot of folks look at LLMs as some kind of programmer replacement.
        wahnfrieden 11mo ago
        They are that too
        ChrisMarshallNY 11mo ago
        I still wouldn't trust them for a lot of stuff. Some of the code I get from Claude and ChatGPT is ... not so good.
        wahnfrieden 11mo ago
        I review it and I sometimes have it retry the same task 40+ times
        int_19h 11mo ago
        It's like an intern that is incapable of learning. But a very enthusiastic one.
  - bluGill 1y ago
    I did that once: logging ended up taking 80% of the CPU leaving not enough overhead for everything else the system should do. Now I am more careful to figure out what is worth logging at all, and also to make sure disabled logs are quickly bypassed.
    - dylan604 1y ago
      we've gotten into adding verbosity levels in logging where each logged event comes with an assigned level that only makes it to the log if it matches the requested log level. there are times when a full verbose output is just too damn much for day-to-day debugging, but is helpful when debugging the one feature. i used to think options like -vvv or -loglevel panic were just someone being funny, but they do work when necessary. -loglevel sane, -loglevel unsane, -loglevel insane would be my take but am aware that most people would roll their eyes so we're lame using ERROR, WARNING, INFO, VERBOSE
      - bluGill 1y ago
        On smaller projects that works. We have a complex system where individual logs can get the log level changed. Though this turns out too fine grained. I'm moving to every subsystem being controllable, but not the individual logs. I'm still not sure what the right answer is though - it always seems like there are 10,000 lines of unrelated useless logs to wade through before finding the useful one, but anytime I remove something that turns out to be the needed log for the very next bug report...
        1718627440 1y ago
        Use something like syslog, where everything is recorded and you can filter on display by subsystem and loglevel.
      - wahnfrieden 1y ago
        That's great when you have to maintain a large amount of logs for weeks, months, years. But I'm talking about adding and removing logs per dev task. There's really no need to have sophisticated log levels and maintaining them as the app evolves and grows, because the LLM can "instantly" add and remove the logging it needs per granular task. This is much faster for me than maintaining logs and carefully selecting log levels and managing how logs can be filtered. That only made sense to me when it took actual dev effort to add or remove these logs.
kykat 1y ago
Sometimes, a "bug" can be caused by nasty architecture with intertwined hacks. Particularly on games, where you can easily have event A that triggers B unless C is in X state... What I want to say is that I've seen what happens in a team with a history of quick fixes and inadequate architecture design to support the complex features. In that case, a proper bugfix could create significant rework and QA.
- ChrisMarshallNY 1y ago
  In that case, maybe having bug fixing be a two-step process (identify, then fix), might be sensible.
  - OhMeadhbh 1y ago
    I do this frequently. But sometimes identifying and/or fixing takes more than 2 days. But you hit on a point that seems to come up a lot. When a user story takes longer than the alloted points, I encourage my junior engineers to split it into two bugs. Exactly like what you say... One bug (or issue or story) describing what you did to typify the problem and another with a suggestion for what to do to fix it. There doesn't seem to be a lot of industry best practice about how to manage this, so we just do whatever seems best to communicate to other teams (and to ourselves later in time after we've forgotten about the bug) what happened and why. Bug fix times are probably a pareto distribution. The overwhelming majority will be identifiable within a fixed time box, but not all. So in addition to saying "no bug should take more than 2 days" I would add "if the bug takes more than 2 days, you really need to tell someone, something's going on." And one of the things I work VERY HARD to create is a sense of psychological safety so devs know they're not going to lose their bonus if they randomly picked a bug that was much more wicked than anyone thought.
    - ChrisMarshallNY 1y ago
      You sound like a great team leader. Wish there were more like you, out there.
    - ljm 1y ago
lapcat 1y ago
> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. This is explained later in the post. The 2 day hard limit is applied not to the estimate but rather to the actual work: "If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else."
- ChrisMarshallNY 1y ago
  Most of the work in finding/fixing bugs is reproducing them reliably enough to determine the root cause. Once I find a bug, the fix is often negligible. But I can get into a rabbithole, tracking down the root cause. I don’t know if I’ve ever spent more than a day, trying to pin down a bug, but I have walked away from rabbitholes, a couple of times. I hate doing that. Leaves an unscratchable itch.
triyambakam 1y ago
> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. Now I find that odd.
- ChrisMarshallNY 1y ago
  Yeah, I’m obviously a terrible programmer. Ya got me.
  - triyambakam 1y ago
    I just find it so oversimplified that I can't believe you're sincere. Like you have entirely no internal heuristic for even a coarse estimation of a few minutes, hours, or days? I would say you're not being very introspective or are just exaggerating.
    - com2kid 1y ago
      My team once encountered a bug that was due to a supplier misstating the delay timing needed for a memory chip. The timings we had in place worked, for most chips, but they failed for a small % of chips in the field. The failure was always exactly identical, the same memory address for corrupted, so it looked exactly like an invalid pointer access. It took multiple engineers months of investigating to finally track down the root cause.
      - triyambakam 1y ago
        But what was the original estimate? And even so I'm not saying it must be completely and always correct. I'm saying it seems wild to have no starting point, to simply give up.
        com2kid 1y ago
        Have you ever fixed random memory corruption in an OS without memory protection? Best case you trap on memory access to an address if your debugger supports it (ours didn't). Worst case you go through every pointer that is known to access nearby memory and go over the code very very carefully. Of course it doesn't have to be a nearby pointer, it can be any pointer anywhere in the code base causing the problem, you just hope it is a nearby pointer because the alternative is a needle in a haystack. I forget how we did find the root cause, I think someone may have just guessed bit flip in a pointer (vs overrun) and then un-bit-flipped every one of the possible bits one by one (not that many, only a few MB of memory so not many active bits for pointers...) and seen what was nearby (figuring what the originally intended address of the pointer was) and started investigating what pointer it was originally supposed to be. Then after confirming it was a bit flip you have to figure out why the hell a subset of your devices are reliably seeing the exact same bit flipped, once every few days. So to answer your question, you get a bug (memory is being corrupted), you do an initial investigation, and then provide an estimate. That estimate can very well be "no way to tell". The principal engineer on this particular project (Microsoft Band) had a strict 0 user impacting bugs rule. Accordingly, after one of my guys spend a couple weeks investigating, the principal engineer assigned one of the top firmware engineers in the world to track down this one bug and fix it. It took over a month.
PaulKeeble 1y ago
Sometimes you find the cause of the bug in 5 minutes because its precisely where you thought it was, sometimes its not there and you end up writing some extra logging to hopefully expose its cause in production after the next release because you can't reproduce as its transient. I don't know how to predict how long a bug will take to reproduce and track down and only once its understood do we know how long it takes to fix.
OhMeadhbh 1y ago
At Amazon we had a bug that was the result of a compiler bug and the behaviour of intel cores being mis-documented. It was intermittent and related to one core occasionally being allowed to access stale data in the cache. We debugged it with a logic analyzer, the commented nginx source and a copy of the C++ 11 spec. It took longer than 2 days to fix.
- auguzanellato 1y ago
  What kind of LA did you use to de bug an Intel core?
  - OhMeadhbh 1y ago
    The hardware team had some semi-custom thing from intel that spat out (no surprise) gigabytes of trace data per second. I remember much of the pain was in constructing a lab where we could drive a test system at reasonable loads to get the buggy behavior to emerge. It was intermittent so it took use a couple weeks to come up with theories, another couple days for testing and a week of analysis before we came up triggers that allowed us to capture the data that showed the bug. it was a bit of a production.
- ChrisMarshallNY 1y ago
  I’m old enough to have used ICEs to trace program execution. They were damn cool. I seriously doubt that something like that, exists outside of a TSMC or Intel lab, these days.
  - plq 1y ago
    ICE meaning in-circuit emulator in this instance, I assume?
    - ChrisMarshallNY 1y ago
      Yeah. Guess it’s kind of a loaded acronym, these days.
  - 1y ago
Uehreka 1y ago
> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. In my experience there are two types of low-priority bugs (high-priority bugs just have to be fixed immediately no matter how easy or hard they are). 1. The kind where I facepalm and go “yup, I know exactly what that is”, though sometimes it’s too low of a priority to do it right now, and it ends up sitting on the backlog forever. This is the kind of bug the author wants to sweep for, they can often be wiped out in big batches by temporarily making bug-hunting the priority every once in a while. 2. The kind where I go “Hmm, that’s weird, that really shouldn’t happen.” These can be easy and turn into a facepalm after an hour of searching, or they can turn out to be brain-broiling heisenbugs that eat up tons of time, and it’s difficult to figure out which. If you wipe out a ton of category 1 bugs then trying to sift through this category for easy wins can be a good use of time. And yeah, sometimes a category 1 bug turns out to be category 2, but that’s pretty unusual. This is definitely an area where the perfect is the enemy of the good, and I find this mental model to be pretty good.
- tonyedgecombe 1y ago
  >high-priority bugs just have to be fixed immediately no matter how easy or hard they are The fact that something is high priority doesn't make it less work.
  - ChrisMarshallNY 1y ago
    Or more. I often find the nastiest bugs are the quickest fixes. I have a "zero-crash" policy. Crashes are never acceptable. It's easy to enforce, because crashes are usually easy to find and fix. $> ThreadingProblems has entered the chat
j45 1y ago
Bugs taking less than 2 days are great to have as a target but will not be something that can be guaranteed.
- RossBencina 1y ago
  Next up: a new programming language or methodology that guarantees all bugs take less than two days to fix.
  - wiredfool 1y ago
    I think this, like many problems, can be reefactored into the halting problem. Which we know how to solve…. Right?
    - Viliam1234 11mo ago
      We definitely know how to create a Jira ticket for it, and the rest is developer's problem.
mat0 1y ago
you cannot know. that’s why the post elaborates saying (paraphrasing) “if you realize it’s taking longer, cut your losses and move on to something else”
sshine 1y ago
> unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one Race conditions in 3rd party services during / affected by very long builds and with poor metrics and almost no documentation. They only show up sometimes, and you have to wait for it to reoccur. Add to this a domain you’re not familiar with, and your ability to debug needs to be established first. Stack two or three of these on top of each other and you have days of figuring out what’s going on, mostly waiting for builds, speculating how to improve debug output. After resolving, don’t write any integration tests that might catch regressions, because you already spent enough time fixing it, and this needs to get replaced soon anyway (timeline: unknown).
marginalia_nu 1y ago
I think in general, bugs go unfixed in two scenarios: 1. The cause isn't immediately obvious. In this case, finding the problem is usually 90% of the work. Here it can't be known how long finding the problem is beforehand, though I don't think bailing because it's taking too long is a good idea. If anything, it's those really deep rabbit holes the real gremlins can hide. 2. The cause is immediately obvious, but is an architecture mistake, the fix is a shit-ton of work, breaks workflows, requires involving stakeholders, etc. Even in this case it can be hard to say how long it will take, especially if other people are involved and have to sign off on decisions. I suppose it can also happen in low-trust sweatshops where developers held on such a tight leash they aren't able to fix trivial bugs they find without first going through a bunch of jira rigmarole, which is sort of low key the vibe I got from the post.
pjc50 1y ago
I think the worst case I encountered was something like two years from first customer report to even fully confirming the bug, followed by about a month of increasingly detailed investigations, a robot, and an osciliscope. The initial description? "Touchscreen sometimes misses button presses".
- ChrisMarshallNY 1y ago
  Thanks. I love hearing stories like this.
  - pjc50 1y ago
    I'm no Raymond Chen, but sometimes I wish I'd kept notes on interesting bugs that I could take with me when I moved jobs. I've often been the go-to guy for weird shit that is happening that nobody else understands and requires cross-disciplinary insight. Other favourites include "Microsoft Structured Exception Handling sometimes doesn't catch segfaults", and "any two of these network devices work together but all three combined freak out".
ahoka 1y ago
Not sure why would you ever need to refactor for fixing a bug?
- ChrisMarshallNY 1y ago
  Oh, that's because a bug in requirements or specification is usually a killer. I have encountered areas where the basic design was wrong (often comes from rushing in, before taking the time to think things through, all the way). In these cases, we can either kludge a patch, or go back and make sure the design is fixed. The longer I've been working, the less often I need to go back and fix a busted design.
- nemetroid 1y ago
  A nice way to fix bugs is to make the buggy state impossible to represent. In cases where a bug was caused by some fundamental flaw in the original design, a redesign might be the only way to feel reasonably confident about the fix.
ZaoLahma 1y ago
> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. The longer I work as a software engineer, the rarer it is that I get to work with bugs that take only a day to fix.
- ChrisMarshallNY 1y ago
  I've found the opposite to be true, in my case.
  - ZaoLahma 1y ago
    For me the longer I work, the worse the bugs I work with become. Nowadays, after some 17 years in the business, it's pretty much always intermittently and rarely occurring race conditions of different flavors. They might result in different behaviors (crashes, missing or wrong data, ...), but at the core of it, it's almost always race conditions. The easy and quick to fix bugs never end up with me.
    - ChrisMarshallNY 1y ago
      The reward for good work, is more work. I tend to mostly work alone, these days (Chief Cook & Bottle-Washer). All bugs are mine.
      - sfink 1y ago
        What kind of kitchen are you working in where bugs are a concern??!
    - lll-o-lll 1y ago
      Yep. Non-determinism. Back in the day it was memory corruption caused by some race condition. By the time things have gone pop, you’re too far from the proximate cause to have useful logs or dumps. “Happens only once every 100k runs? Won’t fix”. That works until it doesn’t, then they come looking for the poor bastard that never fixes a bug in 2 days.
beberlei 1y ago
Its odd at first, but springs from economic principles, mainly sunk cost fallacy. If you invest 2 days of work and did not find the root cause of a bug, then you have the human desire to keep investing more work, because you already invested so much work. At that point however its best to re-evaluate and do something different instead, because it might have a bigger impact. Likelihood that after 2 days of not finding the problem, you wont find it after another 2 days is higher than starting over with another bug that on average you find the problem earlier.
- lan321 1y ago
  This sounds incorrect. You didn't find it but you're gaining domain knowledge and excluding options, hopefully narrowing down the cause. It's not like you're just chucking random garbage at Jenkins. Of course, if it's a difficult bug and you can just say 'fuck it' and bury it in the backlog forever that's fine, but in my experience the very complex ones don't get discovered or worked on at all unless it's absolutely critical or a customer complains.
mobeigi 1y ago
I believe the idea is to pick small items that you'd likely be able to solve quickly. You don't know for sure but you can usually take a good guess at which tasks are quick.
khannn 1y ago
I had a job that required estimation on bug tickets. It's honestly amazing how they didn't realize that I'd take my actual estimate, then multiply it by 4, then use the extra time to work on my other bug tickets that the 4x multiplier wasn't good enough for.
- mewpmewp2 1y ago
  That's just you hedging, they don't really need to know that. As long as if you are hedging accurately in the big picture, that's all that matters. They need estimates to be able to make decisions on what should be done and what not. You could tell them that 25% chance it's going to take 2 hours or less, 50% chance it's going to take 4 hours or less, 75% chance it's going to take 8 hours or less, 99% it's going to take 16 hours or less, to be accurate, but communication wise you'll win out if you just call items like those 10 hours or similar intuitively. Intuitively you feel that 10 hours seems safe with those probabilities (which are intuitive experience based too). So you probably would say 10 hours, unless something really unexpected (the 1%) happens. Btw in reality with above probabilities the actual average would be 5h - 6h with 1% tasks potentially failing, but even your intuitive probability estimations could be off so you likely want to say 10h. But anyhow that's why story points are mostly used as well, because if you say hours they will naturally think it's more fixed estimation. Hours would be fine if everyone understood naturally that it implies a certain statistical average of time + reasonable buffer it would take over a large amount of similar tasks.
- georgemcbay 1y ago
  Are you sure they didn't realize it...? Virtually everywhere I've ever worked has had an unwritten but widely understood informal policy of placing a multiple on predicted effort for both new code/features and bug fixing to account for Hofstadter's law.
  - khannn 11mo ago
    There are a lot of scrum masters who have never actually programmed
brightball 1y ago
In my experience, the vast majority of bugs are quick fixes that are easy to isolate or potentially even have a stack trace associated with them. There will always be those “only happens on the 3rd Tuesday every 6 months” issues that are more complicated but…if you can get all the small stuff out of the way it’s much easier to dedicate some time to the more complicated ones. Maximizing the value of time is the real key to focusing on quicker fixes. If nobody can make a case why one is more important than other, then the best use of your time is the fastest fix.
muixoozie 1y ago
I worked for a company that.. Used msql sever a lot and we would run into a heisenbug every few months that would crash our self hosted msql server cluster or it would become unresponsive. I'm not a database person so I'm probably butchering the description here. From our POV progress would stop and require manual intervention (on call). Back and forth went on with MS and our DBAs for YEARS pouring over logs or whatever they do.. Honestly never thought it would be fixed. Then one time it happened and we caught all the data going into the commit and realized it would 100% reproduce the crash. Only if we restored the database to a specific state and with this specific commit it would crash MS SQL Server. NDAs were signed and I took machete to our code base to create a minimal repro binary that could deserialize our data store and commit / crash MS SQL sever. Made a nice powershell script to wrap it and repro the issue fast and guess what? Within a month they fixed it. Was never clear on what exactly the problem was on their end.. I got buffer overflow vibes, but that's a guess.
- newtwilly 1y ago
  Wow, that's pretty epic and satisfying
- DanielHB 1y ago
  I once ran into a bug where our server code would crash only on a specific version of the Linux Kernel under a specific version of the OpenJDK that our client had. At least it would crash at startup but it was some good 2 weeks of troubleshooting because we couldn't change the target environment we were deploying on. At least it crashed at startup, if it was random it would have been hell.
yxhuvud 1y ago
I've seen people spending 4 months on a hard to replicate segfault.
w0m 1y ago
> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. oh sweet sweet summer child...
QuiEgo 1y ago
As someone who works with hardware, hard to repo bugs can take months to track down. Your code, the compiler, or the hardware itself (which is often a complex ball of IP from dozens of manufacturers held together with a NoC) could all be a problem. The extra fun bugs are when a bug is due to problems in two or three of them combining together in the perfect storm to make a mega bug that is impossible to reproduce in isolation.
- QuiEgo 1y ago
  Random example: I once worked on a debug where you were not allowed to send zero length packets due to a known HW bug. Okay fine, work around in SW. Turns out there was an HW eviction timer that was disabled. It was connected to a counter that counted sys clk ticks. Turns out it was not disabled entirely properly due to SW bug, so once every 2^32 ticks, it would trigger an evection, and if the queue happened to be empty, it would send a ZLP, which triggered the first bug (hard hang the system in a way that breaks the debugger). There were dozens of ways that could hard hang the system, this was just one. Good luck debugging that in two days.
  - jeffreygoesto 1y ago
    We had one where data, interpreted as address (simple C typo before static analysis was common) fell into an unmapped memory region and the PCI controller stalled trying to get a response, thereby also halting the internal debugging logic and JTAG just stopped forever (PPC603 core). Each time you'd hit the bug, the debugger was thrown off.
huherto 1y ago
I do agree that you should be able to fix most bugs in 2 days or less. If you have many bugs taking longer to fix, it may be an indication that you may have systemic issues. (e.g design, architectural, tooling, environment access, test infrastructure, etc)
- bluGill 1y ago
  Sure, but you never know if this next bug is another fix it in 1 hour, or it will take months to figure out. I have had a few "The is not spelled 'Teh'" bugs that it takes longer to find the code in question with grep than to fix, but most are a not that obvious and so you don't know if there are 2 hours left or not until 2 hours latter when you know you found something or are still looking. (or unless you think you fixed it and the time to verify the test is about 2 hours, but then only if your fix worked)
AbstractH24 1y ago
> Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. Learning how to better estimate how long tasks take is one of my biggest goals. And one I've yet to even figure out how to master
peepee1982 1y ago
Yep. Also, sometimes you figure out a bug and in the process you find a whole bunch of new ones that the first bug just never let surface.
jorvi 1y ago
Yeah, "no bug should take over 2 days" tells me you've never had a race condition in your codebase.
- deleted 1y ago
  [deleted]
- ChrisMarshallNY 1y ago
  I'm sure that you're right. I'm likely a bad, inexperienced engineer. There's a lot of us, out here.
  - jorvi 1y ago
    I'm sure your sarcasm is right. You're likely a good, godlike engineer that would fix even the most intractable race conditions within 48 hours. There's a lot of you, out there.
    - ChrisMarshallNY 1y ago
      Have a great day!
JJMcJ 1y ago
It's like remodeling. The drywall comes down. Do you just put up a new sheet or do you need to reframe one wall of the house?
thfuran 1y ago
>I can’t imagine spending more than a day on one. You mean starting after it has been properly tracked down? It can often take a whole lot of time to go from "this behavior is incorrect sometimes" to "and here's what need to change".
- ChrisMarshallNY 1y ago
  Depends. If it takes a long time to track down, then it should either be sidelined, or the design needs to be revisited. I have found that really deep bugs are the result of bad design, on my part, and applying "band-aid" fixes often just kicks the can down the road, for a reckoning (that is now just a bit worse), later. If it is not super-serious (small performance issues, for instance; which can involve moving a lot of cheese), I can often schedule a design review for a time when it's less critical, and maybe set up an exploration branch. People keep bringing up threading and race conditions, which are legitimately nasty bugs. In my experience, they are often the result of bad design, on my part. It's been my experience that "thread everything" can be a recipe for disaster. The OS/SDK will often do internal threading, and I can actually make things worse, by running my own threads. I try to design stuff that will work fine, in any thread, which gives me the option to sequester it into a new thread, at a later time (I just did exactly that, a few days ago, in a Watch app), but don't immediately do that.
  - bagacrap 1y ago
    > If it takes a long time to track down, then it should either be sidelined, or the design needs to be revisited. I don't get this. Either you give up on the bug after a day, or you throw out the entire codebase and start over? Sure, if the bug is low severity and I don't have a reproduction, I will ignore it. But there are bad bugs that are not understood and can take a lot more than a day to look into, such as by adding telemetry to help track it down. Yes, it is usually the case that tracking it down is harder than fixing. But there are also cases where the larger system makes some broad assumptions which are not true, and fixing is tricky. It is not usually an option to throw out the entire system and start over each time this happens in a project.
dockd 1y ago
How is this for a rule of thumb: the time it takes to fix a bug is directly related to the age of the software.
- ChrisMarshallNY 1y ago
  That's also a "It Depends™" thing. Really old software can be referred to as "Mature," as opposed to "Decrepit." It can be extremely well-documented, and well-understood. Many times, there are tools that grow up, alongside the main code. I wrote stuff that was still in use, 25 years later, because the folks that took it over, did a really good job of maintaining it.
michaelbuckbee 1y ago
Something I often find are "categorical" bugs where it's really 3 or 4 different bugs in a trench coat all presenting as a single issue.
claw-el 1y ago
> Also, I tend to attack bugs by priority/severity, as opposed to difficulty. This is one part that is rarely properly implemented. We have our bug bash days too, but I noticed after the fact that maybe 1/3 of the bugs we solved is on a feature we are thinking of deprecating soon due to low usage. How can we attack bugs better by priority?
Aurornis 1y ago
All of the buggy software projects I've been employed to work on have had some version of this rule. Usually it's implicit, rather than explicit: Nobody tells you to limit work on bugs to 1-2 days, but if you spend an entire week debugging something difficult and don't accumulate any story points in Jira, a cadre of project manager, program managers, and other manager titles you didn't even know existed will descend upon you and ask why you're dragging the velocity down. Lesson learned: Next time, avoid the hard bugs and give up early if something isn't going to turn into story points for hidden charts that are viewed by more people than you ever thought.
- kccqzy 1y ago
  I hate this kind of management culture that misuses story points. Story points are supposed to take into account difficulty. So if you spend an entire week debugging a difficult bug, you should’ve accumulated about the same amount of story points as colleagues debugging ten easy bugs.
  - oldestofsports 1y ago
    Every one have different approaches to story points and every one thinks their way is ”the right way”. In the end they just turn into an abstraction layer for man hours.
  - int_19h 1y ago
    Just about everything about Agile as it is actually practiced IRL by most workplaces is "misuses X". At some point one can't help but wonder: if almost everyone is "misusing" it, then maybe it's a problem with the methodology itself, and the people for whom it works would have worked just as well organically without it?
- aeternum 1y ago
  It's the right lesson because the difficulty of the bug often depends on the dev. For example it might take one dev weeks to figure out that a hang due to a sleep(.001) call within asyncio whereas another can identify it with a glance at the code.
oldestofsports 1y ago
> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. I understood it as the whole point of the 2 day hard limit - you start working on a bug that turn out to be bigger than expected, so you write down your findings and move on to the next one.
cvoss 1y ago
The article addresses your concerns directly. > In one of our early fixits, someone picked up what looked like a straightforward bug. It should have been a few hours, maybe half a day. But it turned into a rabbit hole. Dependencies on other systems, unexpected edge cases, code that hadn’t been touched in years. > They spent the entire fixit week on it. And then the entire week after fixit trying to finish it. What started as a bug fix turned into a mini project. The work was valuable! But they missed the whole point of a fixit. No closing bugs throughout the week. No momentum. No dopamine hits from shipping fixes. Just one long slog. > That’s why we have the 2-day hard limit now. If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else. The limit isn’t about the work being worthless - it’s about keeping fixit feeling like fixit.
lkbm 1y ago
A big reason we did a "fix week" at my old job was to deal with all the simple, low priority issues. Sure, there were high severity bugs, but they would get prioritized during normal work, whereas fix week was to prevent death of a thousand cuts. Kinda trivial things that just accumulate and make the site look and feel janky. Some things turn out to be surprisingly complex, but you can very often know that the simple thing is simple.
SergeAx 11mo ago
> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. I think what they mean is that after 2 days of working on bug you stop it regardless the result, leaving paper trail behind for the next person.

chii 1y ago

I find most bugs take less time to fix than it takes time to verify and reproduce.

wahnfrieden 1y ago
LLMs have helped me here the most. Adding copious detailed logging across the app on demand, then inspecting the logs to figure out the bug and even how to reproduce it.
- ChrisMarshallNY 1y ago
  Yes. I often just copy the whole core dump, and feed it into the prompt.
  - Lionga 1y ago
    And this kids is how one bug got fixed and two more were created
    - ChrisMarshallNY 1y ago
      Nope. Good result == LLM + Experience. The LLM just reduces the overhead. That’s really what every “new paradigm” has ever done.
      - enraged_camel 1y ago
        Also, robust test coverage helps prevent regressions.
    - Sohcahtoa82 1y ago
      There's a huge difference between using an LLM to assist you versus letting it just do all the work for you. Your implication that they're the same, and that the previous commenter let the LLM do the work, is lazy. ChrisMarshallNY only said they fed the dump into the LLM. They said nothing about using the LLM to write the fix.
  - criddell 1y ago
    This is something that I've been trying to improve at. I work on a Windows application and so I get crash dumps that I open with WinDbg and then I usually start looking for exceptions. Is this something an LLM could help with? What exactly do you mean when you say you feed a dump to the prompt?
    - ChrisMarshallNY 1y ago
      I literally copy the whole stack dump from the log, and paste it into the LLM (I find that ChatGPT does a better job than Claude), along with something along the lines of: > I am getting occasional crashes on my iOS 17 or above UIKit program. Given the following stack trace, what problem do think it might be? I will attach the source file, if I think I know the general area, along with any symptoms and steps to reproduce. One of the nice things about an LLM, is that it's difficult to overwhelm with too much information (unlike people). It will usually respond with a fairly detailed analysis. Usually, it has some good ideas to use as starting points. I don't think "I have a bug. Please fix it." would work, though. It's likely to try, but caveat emptor.
  - int_19h 1y ago
    I kinda wonder if at some point this is something we might use the LLM more directly for. As in, train them on raw binary dumps as input.
    - ChrisMarshallNY 11mo ago
      I wonder if we’ll be seeing tools that do this. I could see Apple or Microsoft, building it into their IDEs. But, as was noted elsewhere, I think it’s only useful as an advisor. I think a lot of folks look at LLMs as some kind of programmer replacement.
      - wahnfrieden 11mo ago
        They are that too
        ChrisMarshallNY 11mo ago
        I still wouldn't trust them for a lot of stuff. Some of the code I get from Claude and ChatGPT is ... not so good.
        wahnfrieden 11mo ago
        I review it and I sometimes have it retry the same task 40+ times
        int_19h 11mo ago
        It's like an intern that is incapable of learning. But a very enthusiastic one.
- bluGill 1y ago
  I did that once: logging ended up taking 80% of the CPU leaving not enough overhead for everything else the system should do. Now I am more careful to figure out what is worth logging at all, and also to make sure disabled logs are quickly bypassed.
  - dylan604 1y ago
    we've gotten into adding verbosity levels in logging where each logged event comes with an assigned level that only makes it to the log if it matches the requested log level. there are times when a full verbose output is just too damn much for day-to-day debugging, but is helpful when debugging the one feature. i used to think options like -vvv or -loglevel panic were just someone being funny, but they do work when necessary. -loglevel sane, -loglevel unsane, -loglevel insane would be my take but am aware that most people would roll their eyes so we're lame using ERROR, WARNING, INFO, VERBOSE
    - bluGill 1y ago
      On smaller projects that works. We have a complex system where individual logs can get the log level changed. Though this turns out too fine grained. I'm moving to every subsystem being controllable, but not the individual logs. I'm still not sure what the right answer is though - it always seems like there are 10,000 lines of unrelated useless logs to wade through before finding the useful one, but anytime I remove something that turns out to be the needed log for the very next bug report...
      - 1718627440 1y ago
        Use something like syslog, where everything is recorded and you can filter on display by subsystem and loglevel.
    - wahnfrieden 1y ago
      That's great when you have to maintain a large amount of logs for weeks, months, years. But I'm talking about adding and removing logs per dev task. There's really no need to have sophisticated log levels and maintaining them as the app evolves and grows, because the LLM can "instantly" add and remove the logging it needs per granular task. This is much faster for me than maintaining logs and carefully selecting log levels and managing how logs can be filtered. That only made sense to me when it took actual dev effort to add or remove these logs.

kykat 1y ago

Sometimes, a "bug" can be caused by nasty architecture with intertwined hacks. Particularly on games, where you can easily have event A that triggers B unless C is in X state... What I want to say is that I've seen what happens in a team with a history of quick fixes and inadequate architecture design to support the complex features. In that case, a proper bugfix could create significant rework and QA.

ChrisMarshallNY 1y ago
In that case, maybe having bug fixing be a two-step process (identify, then fix), might be sensible.
- OhMeadhbh 1y ago
  I do this frequently. But sometimes identifying and/or fixing takes more than 2 days. But you hit on a point that seems to come up a lot. When a user story takes longer than the alloted points, I encourage my junior engineers to split it into two bugs. Exactly like what you say... One bug (or issue or story) describing what you did to typify the problem and another with a suggestion for what to do to fix it. There doesn't seem to be a lot of industry best practice about how to manage this, so we just do whatever seems best to communicate to other teams (and to ourselves later in time after we've forgotten about the bug) what happened and why. Bug fix times are probably a pareto distribution. The overwhelming majority will be identifiable within a fixed time box, but not all. So in addition to saying "no bug should take more than 2 days" I would add "if the bug takes more than 2 days, you really need to tell someone, something's going on." And one of the things I work VERY HARD to create is a sense of psychological safety so devs know they're not going to lose their bonus if they randomly picked a bug that was much more wicked than anyone thought.
  - ChrisMarshallNY 1y ago
    You sound like a great team leader. Wish there were more like you, out there.
  - ljm 1y ago

lapcat 1y ago

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. This is explained later in the post. The 2 day hard limit is applied not to the estimate but rather to the actual work: "If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else."

ChrisMarshallNY 1y ago
Most of the work in finding/fixing bugs is reproducing them reliably enough to determine the root cause. Once I find a bug, the fix is often negligible. But I can get into a rabbithole, tracking down the root cause. I don’t know if I’ve ever spent more than a day, trying to pin down a bug, but I have walked away from rabbitholes, a couple of times. I hate doing that. Leaves an unscratchable itch.

triyambakam 1y ago

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. Now I find that odd.

ChrisMarshallNY 1y ago
Yeah, I’m obviously a terrible programmer. Ya got me.
- triyambakam 1y ago
  I just find it so oversimplified that I can't believe you're sincere. Like you have entirely no internal heuristic for even a coarse estimation of a few minutes, hours, or days? I would say you're not being very introspective or are just exaggerating.
  - com2kid 1y ago
    My team once encountered a bug that was due to a supplier misstating the delay timing needed for a memory chip. The timings we had in place worked, for most chips, but they failed for a small % of chips in the field. The failure was always exactly identical, the same memory address for corrupted, so it looked exactly like an invalid pointer access. It took multiple engineers months of investigating to finally track down the root cause.
    - triyambakam 1y ago
      But what was the original estimate? And even so I'm not saying it must be completely and always correct. I'm saying it seems wild to have no starting point, to simply give up.
      - com2kid 1y ago
        Have you ever fixed random memory corruption in an OS without memory protection? Best case you trap on memory access to an address if your debugger supports it (ours didn't). Worst case you go through every pointer that is known to access nearby memory and go over the code very very carefully. Of course it doesn't have to be a nearby pointer, it can be any pointer anywhere in the code base causing the problem, you just hope it is a nearby pointer because the alternative is a needle in a haystack. I forget how we did find the root cause, I think someone may have just guessed bit flip in a pointer (vs overrun) and then un-bit-flipped every one of the possible bits one by one (not that many, only a few MB of memory so not many active bits for pointers...) and seen what was nearby (figuring what the originally intended address of the pointer was) and started investigating what pointer it was originally supposed to be. Then after confirming it was a bit flip you have to figure out why the hell a subset of your devices are reliably seeing the exact same bit flipped, once every few days. So to answer your question, you get a bug (memory is being corrupted), you do an initial investigation, and then provide an estimate. That estimate can very well be "no way to tell". The principal engineer on this particular project (Microsoft Band) had a strict 0 user impacting bugs rule. Accordingly, after one of my guys spend a couple weeks investigating, the principal engineer assigned one of the top firmware engineers in the world to track down this one bug and fix it. It took over a month.

PaulKeeble 1y ago

Sometimes you find the cause of the bug in 5 minutes because its precisely where you thought it was, sometimes its not there and you end up writing some extra logging to hopefully expose its cause in production after the next release because you can't reproduce as its transient. I don't know how to predict how long a bug will take to reproduce and track down and only once its understood do we know how long it takes to fix.

OhMeadhbh 1y ago

At Amazon we had a bug that was the result of a compiler bug and the behaviour of intel cores being mis-documented. It was intermittent and related to one core occasionally being allowed to access stale data in the cache. We debugged it with a logic analyzer, the commented nginx source and a copy of the C++ 11 spec. It took longer than 2 days to fix.

auguzanellato 1y ago
What kind of LA did you use to de bug an Intel core?
- OhMeadhbh 1y ago
  The hardware team had some semi-custom thing from intel that spat out (no surprise) gigabytes of trace data per second. I remember much of the pain was in constructing a lab where we could drive a test system at reasonable loads to get the buggy behavior to emerge. It was intermittent so it took use a couple weeks to come up with theories, another couple days for testing and a week of analysis before we came up triggers that allowed us to capture the data that showed the bug. it was a bit of a production.
ChrisMarshallNY 1y ago
I’m old enough to have used ICEs to trace program execution. They were damn cool. I seriously doubt that something like that, exists outside of a TSMC or Intel lab, these days.
- plq 1y ago
  ICE meaning in-circuit emulator in this instance, I assume?
  - ChrisMarshallNY 1y ago
    Yeah. Guess it’s kind of a loaded acronym, these days.
- 1y ago

Uehreka 1y ago

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. In my experience there are two types of low-priority bugs (high-priority bugs just have to be fixed immediately no matter how easy or hard they are). 1. The kind where I facepalm and go “yup, I know exactly what that is”, though sometimes it’s too low of a priority to do it right now, and it ends up sitting on the backlog forever. This is the kind of bug the author wants to sweep for, they can often be wiped out in big batches by temporarily making bug-hunting the priority every once in a while. 2. The kind where I go “Hmm, that’s weird, that really shouldn’t happen.” These can be easy and turn into a facepalm after an hour of searching, or they can turn out to be brain-broiling heisenbugs that eat up tons of time, and it’s difficult to figure out which. If you wipe out a ton of category 1 bugs then trying to sift through this category for easy wins can be a good use of time. And yeah, sometimes a category 1 bug turns out to be category 2, but that’s pretty unusual. This is definitely an area where the perfect is the enemy of the good, and I find this mental model to be pretty good.

tonyedgecombe 1y ago
>high-priority bugs just have to be fixed immediately no matter how easy or hard they are The fact that something is high priority doesn't make it less work.
- ChrisMarshallNY 1y ago
  Or more. I often find the nastiest bugs are the quickest fixes. I have a "zero-crash" policy. Crashes are never acceptable. It's easy to enforce, because crashes are usually easy to find and fix. $> ThreadingProblems has entered the chat

j45 1y ago

Bugs taking less than 2 days are great to have as a target but will not be something that can be guaranteed.

RossBencina 1y ago
Next up: a new programming language or methodology that guarantees all bugs take less than two days to fix.
- wiredfool 1y ago
  I think this, like many problems, can be reefactored into the halting problem. Which we know how to solve…. Right?
  - Viliam1234 11mo ago
    We definitely know how to create a Jira ticket for it, and the rest is developer's problem.

mat0 1y ago

you cannot know. that’s why the post elaborates saying (paraphrasing) “if you realize it’s taking longer, cut your losses and move on to something else”

sshine 1y ago

> unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one Race conditions in 3rd party services during / affected by very long builds and with poor metrics and almost no documentation. They only show up sometimes, and you have to wait for it to reoccur. Add to this a domain you’re not familiar with, and your ability to debug needs to be established first. Stack two or three of these on top of each other and you have days of figuring out what’s going on, mostly waiting for builds, speculating how to improve debug output. After resolving, don’t write any integration tests that might catch regressions, because you already spent enough time fixing it, and this needs to get replaced soon anyway (timeline: unknown).

marginalia_nu 1y ago

I think in general, bugs go unfixed in two scenarios: 1. The cause isn't immediately obvious. In this case, finding the problem is usually 90% of the work. Here it can't be known how long finding the problem is beforehand, though I don't think bailing because it's taking too long is a good idea. If anything, it's those really deep rabbit holes the real gremlins can hide. 2. The cause is immediately obvious, but is an architecture mistake, the fix is a shit-ton of work, breaks workflows, requires involving stakeholders, etc. Even in this case it can be hard to say how long it will take, especially if other people are involved and have to sign off on decisions. I suppose it can also happen in low-trust sweatshops where developers held on such a tight leash they aren't able to fix trivial bugs they find without first going through a bunch of jira rigmarole, which is sort of low key the vibe I got from the post.

pjc50 1y ago

I think the worst case I encountered was something like two years from first customer report to even fully confirming the bug, followed by about a month of increasingly detailed investigations, a robot, and an osciliscope. The initial description? "Touchscreen sometimes misses button presses".

ChrisMarshallNY 1y ago
Thanks. I love hearing stories like this.
- pjc50 1y ago
  I'm no Raymond Chen, but sometimes I wish I'd kept notes on interesting bugs that I could take with me when I moved jobs. I've often been the go-to guy for weird shit that is happening that nobody else understands and requires cross-disciplinary insight. Other favourites include "Microsoft Structured Exception Handling sometimes doesn't catch segfaults", and "any two of these network devices work together but all three combined freak out".

ahoka 1y ago

Not sure why would you ever need to refactor for fixing a bug?

ChrisMarshallNY 1y ago
Oh, that's because a bug in requirements or specification is usually a killer. I have encountered areas where the basic design was wrong (often comes from rushing in, before taking the time to think things through, all the way). In these cases, we can either kludge a patch, or go back and make sure the design is fixed. The longer I've been working, the less often I need to go back and fix a busted design.
nemetroid 1y ago
A nice way to fix bugs is to make the buggy state impossible to represent. In cases where a bug was caused by some fundamental flaw in the original design, a redesign might be the only way to feel reasonably confident about the fix.

ZaoLahma 1y ago

> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. The longer I work as a software engineer, the rarer it is that I get to work with bugs that take only a day to fix.

ChrisMarshallNY 1y ago
I've found the opposite to be true, in my case.
- ZaoLahma 1y ago
  For me the longer I work, the worse the bugs I work with become. Nowadays, after some 17 years in the business, it's pretty much always intermittently and rarely occurring race conditions of different flavors. They might result in different behaviors (crashes, missing or wrong data, ...), but at the core of it, it's almost always race conditions. The easy and quick to fix bugs never end up with me.
  - ChrisMarshallNY 1y ago
    The reward for good work, is more work. I tend to mostly work alone, these days (Chief Cook & Bottle-Washer). All bugs are mine.
    - sfink 1y ago
      What kind of kitchen are you working in where bugs are a concern??!
  - lll-o-lll 1y ago
    Yep. Non-determinism. Back in the day it was memory corruption caused by some race condition. By the time things have gone pop, you’re too far from the proximate cause to have useful logs or dumps. “Happens only once every 100k runs? Won’t fix”. That works until it doesn’t, then they come looking for the poor bastard that never fixes a bug in 2 days.

beberlei 1y ago

Its odd at first, but springs from economic principles, mainly sunk cost fallacy. If you invest 2 days of work and did not find the root cause of a bug, then you have the human desire to keep investing more work, because you already invested so much work. At that point however its best to re-evaluate and do something different instead, because it might have a bigger impact. Likelihood that after 2 days of not finding the problem, you wont find it after another 2 days is higher than starting over with another bug that on average you find the problem earlier.

lan321 1y ago
This sounds incorrect. You didn't find it but you're gaining domain knowledge and excluding options, hopefully narrowing down the cause. It's not like you're just chucking random garbage at Jenkins. Of course, if it's a difficult bug and you can just say 'fuck it' and bury it in the backlog forever that's fine, but in my experience the very complex ones don't get discovered or worked on at all unless it's absolutely critical or a customer complains.

mobeigi 1y ago

I believe the idea is to pick small items that you'd likely be able to solve quickly. You don't know for sure but you can usually take a good guess at which tasks are quick.

khannn 1y ago

I had a job that required estimation on bug tickets. It's honestly amazing how they didn't realize that I'd take my actual estimate, then multiply it by 4, then use the extra time to work on my other bug tickets that the 4x multiplier wasn't good enough for.

mewpmewp2 1y ago
That's just you hedging, they don't really need to know that. As long as if you are hedging accurately in the big picture, that's all that matters. They need estimates to be able to make decisions on what should be done and what not. You could tell them that 25% chance it's going to take 2 hours or less, 50% chance it's going to take 4 hours or less, 75% chance it's going to take 8 hours or less, 99% it's going to take 16 hours or less, to be accurate, but communication wise you'll win out if you just call items like those 10 hours or similar intuitively. Intuitively you feel that 10 hours seems safe with those probabilities (which are intuitive experience based too). So you probably would say 10 hours, unless something really unexpected (the 1%) happens. Btw in reality with above probabilities the actual average would be 5h - 6h with 1% tasks potentially failing, but even your intuitive probability estimations could be off so you likely want to say 10h. But anyhow that's why story points are mostly used as well, because if you say hours they will naturally think it's more fixed estimation. Hours would be fine if everyone understood naturally that it implies a certain statistical average of time + reasonable buffer it would take over a large amount of similar tasks.
georgemcbay 1y ago
Are you sure they didn't realize it...? Virtually everywhere I've ever worked has had an unwritten but widely understood informal policy of placing a multiple on predicted effort for both new code/features and bug fixing to account for Hofstadter's law.
- khannn 11mo ago
  There are a lot of scrum masters who have never actually programmed

brightball 1y ago

In my experience, the vast majority of bugs are quick fixes that are easy to isolate or potentially even have a stack trace associated with them. There will always be those “only happens on the 3rd Tuesday every 6 months” issues that are more complicated but…if you can get all the small stuff out of the way it’s much easier to dedicate some time to the more complicated ones. Maximizing the value of time is the real key to focusing on quicker fixes. If nobody can make a case why one is more important than other, then the best use of your time is the fastest fix.

muixoozie 1y ago

I worked for a company that.. Used msql sever a lot and we would run into a heisenbug every few months that would crash our self hosted msql server cluster or it would become unresponsive. I'm not a database person so I'm probably butchering the description here. From our POV progress would stop and require manual intervention (on call). Back and forth went on with MS and our DBAs for YEARS pouring over logs or whatever they do.. Honestly never thought it would be fixed. Then one time it happened and we caught all the data going into the commit and realized it would 100% reproduce the crash. Only if we restored the database to a specific state and with this specific commit it would crash MS SQL Server. NDAs were signed and I took machete to our code base to create a minimal repro binary that could deserialize our data store and commit / crash MS SQL sever. Made a nice powershell script to wrap it and repro the issue fast and guess what? Within a month they fixed it. Was never clear on what exactly the problem was on their end.. I got buffer overflow vibes, but that's a guess.

newtwilly 1y ago
Wow, that's pretty epic and satisfying
DanielHB 1y ago
I once ran into a bug where our server code would crash only on a specific version of the Linux Kernel under a specific version of the OpenJDK that our client had. At least it would crash at startup but it was some good 2 weeks of troubleshooting because we couldn't change the target environment we were deploying on. At least it crashed at startup, if it was random it would have been hell.

yxhuvud 1y ago

I've seen people spending 4 months on a hard to replicate segfault.

w0m 1y ago

> That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. oh sweet sweet summer child...

QuiEgo 1y ago

As someone who works with hardware, hard to repo bugs can take months to track down. Your code, the compiler, or the hardware itself (which is often a complex ball of IP from dozens of manufacturers held together with a NoC) could all be a problem. The extra fun bugs are when a bug is due to problems in two or three of them combining together in the perfect storm to make a mega bug that is impossible to reproduce in isolation.

QuiEgo 1y ago
Random example: I once worked on a debug where you were not allowed to send zero length packets due to a known HW bug. Okay fine, work around in SW. Turns out there was an HW eviction timer that was disabled. It was connected to a counter that counted sys clk ticks. Turns out it was not disabled entirely properly due to SW bug, so once every 2^32 ticks, it would trigger an evection, and if the queue happened to be empty, it would send a ZLP, which triggered the first bug (hard hang the system in a way that breaks the debugger). There were dozens of ways that could hard hang the system, this was just one. Good luck debugging that in two days.
- jeffreygoesto 1y ago
  We had one where data, interpreted as address (simple C typo before static analysis was common) fell into an unmapped memory region and the PCI controller stalled trying to get a response, thereby also halting the internal debugging logic and JTAG just stopped forever (PPC603 core). Each time you'd hit the bug, the debugger was thrown off.

huherto 1y ago

I do agree that you should be able to fix most bugs in 2 days or less. If you have many bugs taking longer to fix, it may be an indication that you may have systemic issues. (e.g design, architectural, tooling, environment access, test infrastructure, etc)

bluGill 1y ago
Sure, but you never know if this next bug is another fix it in 1 hour, or it will take months to figure out. I have had a few "The is not spelled 'Teh'" bugs that it takes longer to find the code in question with grep than to fix, but most are a not that obvious and so you don't know if there are 2 hours left or not until 2 hours latter when you know you found something or are still looking. (or unless you think you fixed it and the time to verify the test is about 2 hours, but then only if your fix worked)

AbstractH24 1y ago

> Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. Learning how to better estimate how long tasks take is one of my biggest goals. And one I've yet to even figure out how to master

peepee1982 1y ago

Yep. Also, sometimes you figure out a bug and in the process you find a whole bunch of new ones that the first bug just never let surface.

jorvi 1y ago

Yeah, "no bug should take over 2 days" tells me you've never had a race condition in your codebase.

deleted 1y ago
[deleted]
ChrisMarshallNY 1y ago
I'm sure that you're right. I'm likely a bad, inexperienced engineer. There's a lot of us, out here.
- jorvi 1y ago
  I'm sure your sarcasm is right. You're likely a good, godlike engineer that would fix even the most intractable race conditions within 48 hours. There's a lot of you, out there.
  - ChrisMarshallNY 1y ago
    Have a great day!

JJMcJ 1y ago

It's like remodeling. The drywall comes down. Do you just put up a new sheet or do you need to reframe one wall of the house?

thfuran 1y ago

>I can’t imagine spending more than a day on one. You mean starting after it has been properly tracked down? It can often take a whole lot of time to go from "this behavior is incorrect sometimes" to "and here's what need to change".

ChrisMarshallNY 1y ago
Depends. If it takes a long time to track down, then it should either be sidelined, or the design needs to be revisited. I have found that really deep bugs are the result of bad design, on my part, and applying "band-aid" fixes often just kicks the can down the road, for a reckoning (that is now just a bit worse), later. If it is not super-serious (small performance issues, for instance; which can involve moving a lot of cheese), I can often schedule a design review for a time when it's less critical, and maybe set up an exploration branch. People keep bringing up threading and race conditions, which are legitimately nasty bugs. In my experience, they are often the result of bad design, on my part. It's been my experience that "thread everything" can be a recipe for disaster. The OS/SDK will often do internal threading, and I can actually make things worse, by running my own threads. I try to design stuff that will work fine, in any thread, which gives me the option to sequester it into a new thread, at a later time (I just did exactly that, a few days ago, in a Watch app), but don't immediately do that.
- bagacrap 1y ago
  > If it takes a long time to track down, then it should either be sidelined, or the design needs to be revisited. I don't get this. Either you give up on the bug after a day, or you throw out the entire codebase and start over? Sure, if the bug is low severity and I don't have a reproduction, I will ignore it. But there are bad bugs that are not understood and can take a lot more than a day to look into, such as by adding telemetry to help track it down. Yes, it is usually the case that tracking it down is harder than fixing. But there are also cases where the larger system makes some broad assumptions which are not true, and fixing is tricky. It is not usually an option to throw out the entire system and start over each time this happens in a project.

dockd 1y ago

How is this for a rule of thumb: the time it takes to fix a bug is directly related to the age of the software.

ChrisMarshallNY 1y ago
That's also a "It Depends™" thing. Really old software can be referred to as "Mature," as opposed to "Decrepit." It can be extremely well-documented, and well-understood. Many times, there are tools that grow up, alongside the main code. I wrote stuff that was still in use, 25 years later, because the folks that took it over, did a really good job of maintaining it.

michaelbuckbee 1y ago

Something I often find are "categorical" bugs where it's really 3 or 4 different bugs in a trench coat all presenting as a single issue.

claw-el 1y ago

> Also, I tend to attack bugs by priority/severity, as opposed to difficulty. This is one part that is rarely properly implemented. We have our bug bash days too, but I noticed after the fact that maybe 1/3 of the bugs we solved is on a feature we are thinking of deprecating soon due to low usage. How can we attack bugs better by priority?

Aurornis 1y ago

All of the buggy software projects I've been employed to work on have had some version of this rule. Usually it's implicit, rather than explicit: Nobody tells you to limit work on bugs to 1-2 days, but if you spend an entire week debugging something difficult and don't accumulate any story points in Jira, a cadre of project manager, program managers, and other manager titles you didn't even know existed will descend upon you and ask why you're dragging the velocity down. Lesson learned: Next time, avoid the hard bugs and give up early if something isn't going to turn into story points for hidden charts that are viewed by more people than you ever thought.

kccqzy 1y ago
I hate this kind of management culture that misuses story points. Story points are supposed to take into account difficulty. So if you spend an entire week debugging a difficult bug, you should’ve accumulated about the same amount of story points as colleagues debugging ten easy bugs.
- oldestofsports 1y ago
  Every one have different approaches to story points and every one thinks their way is ”the right way”. In the end they just turn into an abstraction layer for man hours.
- int_19h 1y ago
  Just about everything about Agile as it is actually practiced IRL by most workplaces is "misuses X". At some point one can't help but wonder: if almost everyone is "misusing" it, then maybe it's a problem with the methodology itself, and the people for whom it works would have worked just as well organically without it?
aeternum 1y ago
It's the right lesson because the difficulty of the bug often depends on the dev. For example it might take one dev weeks to figure out that a hang due to a sleep(.001) call within asyncio whereas another can identify it with a glance at the code.

oldestofsports 1y ago

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. I understood it as the whole point of the 2 day hard limit - you start working on a bug that turn out to be bigger than expected, so you write down your findings and move on to the next one.

cvoss 1y ago

The article addresses your concerns directly. > In one of our early fixits, someone picked up what looked like a straightforward bug. It should have been a few hours, maybe half a day. But it turned into a rabbit hole. Dependencies on other systems, unexpected edge cases, code that hadn’t been touched in years. > They spent the entire fixit week on it. And then the entire week after fixit trying to finish it. What started as a bug fix turned into a mini project. The work was valuable! But they missed the whole point of a fixit. No closing bugs throughout the week. No momentum. No dopamine hits from shipping fixes. Just one long slog. > That’s why we have the 2-day hard limit now. If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else. The limit isn’t about the work being worthless - it’s about keeping fixit feeling like fixit.

lkbm 1y ago

A big reason we did a "fix week" at my old job was to deal with all the simple, low priority issues. Sure, there were high severity bugs, but they would get prioritized during normal work, whereas fix week was to prevent death of a thousand cuts. Kinda trivial things that just accumulate and make the site look and feel janky. Some things turn out to be surprisingly complex, but you can very often know that the simple thing is simple.

SergeAx 11mo ago

> It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. I think what they mean is that after 2 days of working on bug you stop it regardless the result, leaving paper trail behind for the next person.