11 ms·

We stopped roadmap work for a week and fixed bugs

284 points by lalitmaganti 11mo ago

inhumantsar 11mo ago
I firmly believe that this sort of fixit week is as much of an anti-pattern as all-features-all-the-time. Ensuring engineers have the agency and the space to fix things and refactor as part of the normal process pays serious dividends in the long run. eg: My last company's system was layer after layer built on top of the semi-technical founder's MVP. The total focus on features meant engineers worked solo most of the time and gave them few opportunities to coordinate and standardize. The result was a mess. Logic smeared across every layer, modules or microservices with overlapping responsibilities writing to the same tables and columns. Mass logging all at the error or info level. It was difficult to understand, harder to trace, and nearly every new feature started off with "well first we need to get out of this corner we find ourselves painted into". When I compare that experience with some other environments I've been in where engineering had more autonomy at the day-to-day level, it's clear to me that this company should have been able to move at least as quickly with half the engineers if they were given the space to coordinate ahead of a new feature and occasionally take the time to refactor things that got spaghettified over time.
- lalitmaganti 11mo ago
  As I pointed out in the "criticisms" section, I don't see fixit weeks as a replacement for good technical hygiene. To be clear, engineers have a lot of autonomy in my team to do what they want. People can and do fix things as they come up and are encouraged to refactor and pay down technical debt as part of their day to day work. It's more that even with this autonomy fixits bugs are underappreciated by everyone, even engineers. Having a week where we can address the balance does wonders.
  - inhumantsar 11mo ago
    yeah that's fair, didn't mean to argue against them entirely. I've just been in too many environments where fixits or cooldowns or whatever were used in place of good hygiene.
julianlam 11mo ago
We did this ages ago at our company (back then we were making silly Facebook games, remember those?) It was by far the most fun, productive, and fulfilling week. It went on to shape the course of our development strategy when I started my own company. Regularly work on tech debt and actively applaud it when others do it too.
xnx 11mo ago
I've never understood why bugs get treated differently from new features. If there was a bug, the old feature was never completed. The time cost and benefits should be considered equally.
- superxpro12 11mo ago
  until we develop a way for MBA's with spreadsheets to quantify profit/loss w.r.t. bugs, it will never be valued.
  - lapcat 11mo ago
    The solution is to never hire an MBA.
    - pixl97 11mo ago
      'Why are we getting bought out by a company that cut corners and hired MBAs, and then fired?'
- sb8244 11mo ago
  If the bug affects 1 customer and the feature affects the rest, is the old feature complete? It's not binary.
  - zelphirkalt 11mo ago
    Yet engineers are pushed to give unknowable estimates of points and when things take "longer" (did you notice that shift right there?) they are either overdue, taking too long, or they don't, and to say: "It takes as long as it takes." is not accepted by middle management.
    - sb8244 11mo ago
      That's a strawman. It is not really related to the main point and I'm not sure of the point you're trying to make (maybe that tension exists?) Obviously things take as long as they take. I've always been an educator of this back to the business leadership. In my experience, most business people truly have no freaking clue how a product gets built and code gets shipped. Giving proactive updates (meaning not the day it was expected to be done according to last update) are important part of a professionals working life. There's always a tension between business and engineers. Engineers just generally don't do that well with tension and try to minimize it, or complain about it.
entropie 11mo ago
I wanted to take a look at some of these bug fixes, and one of the linked ones [1] seems more like a feature to me. So maybe it should be the week of "low priority" issues, or something like that. I don't mean to sound negative, I think it's a great idea. I do something like this at home from time to time. Just spend a day repairing and fixing things. Everything that has accumulated. 1: https://github.com/google/perfetto/issues/154 https://github.com/google/perfetto/issues/154
- mulquin 11mo ago
  To be fair, the blog post does not explicitly say anywhere that the week was for bug fixes only.
ls-a 11mo ago
189 bugs in one week. How many employees quit after that?
- asdfman123 11mo ago
  They said they only pick bugs that take 2 days to fix. Places where you can move fast and actually do things are actually far better places to work for. I mean the ones were you can show up, do 5 hours of really good work, and then slack off/leave a little early.
  - kykat 11mo ago
    Too bad many places care more about how long you stay warming the seat than how useful the work done actually is.
  - ls-a 11mo ago
    Nothing takes 2 days to fix. Those are definitely not bugs, like someone else mentioned
    - toast0 11mo ago
      You haven't seen the same kind of bugs I have, I guess. This kind of thing takes more than 2 days to fix, unless you're really good. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217637 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217637 Or this one https://security.stackexchange.com/questions/104845/dhe-rsa-pubkey-length-in-tls-1-2 https://security.stackexchange.com/questions/104845/dhe-rsa-... I can find more of these that I've run into if I look. I've had tricky bugs in my team's code too, but those don't result in public artifacts, and I'm responsible for all the code that runs on my server, regardless of who wrote it... And I also can't crash client code, regardless of who wrote it, even if my code just follows the RFC.
      - ls-a 11mo ago
        That's what I'm saying. Nothing takes 2 days to fix meaning it takes more time
BurningFrog 11mo ago
This is weird to me... The way I learned the trade, and usually worked, is that bug fixing always comes first! You don't work on new features until the old ones work as they should. This worked well for the teams I was on. Having a (AFAYK) bug free code base is incredibly useful!!
- deleted 11mo ago
  [deleted]
- Celeo 11mo ago
  Depending on the size of the team/org/company, working on anything other than the next feature is a hard sell to PM/PO/PgM/management.
  - BurningFrog 11mo ago
    That's what I hear. I've had some mix of luck and skill in finding these jobs. Working with people you've worked with before helps with knowing what you're in for. I also don't really ask anyone, I just fix any bugs I find. That may not work in all organizations :)
    - zelphirkalt 11mo ago
      micro-managing middle manager: "Are all your other sprint tasks finished?" code reviewing coworker: "This shouldn't be done on this branch!" (OK, at least this is easy to fix by doing it on a separate branch.)
    - ramon156 11mo ago
      I can guarantee you this doesn't work in our team! you didn't make a ticket, so the PM has no idea what you're doing! Yes, a ticket takes 2 seconds. it also puts me off my focus :P but i guess measuring is more important than achieving
  - NegativeK 11mo ago
    I've had to inform leadership that stability is a feature, just like anything else, and that you can't just expect it to happen without giving it time. One leader kind of listened. Sort of. I'm pretty sure I was lucky.
hastily3114 11mo ago
We do this too sometimes and I love it. When I work on my own projects I always stop and refactor/fix problems before adding any new features. I wish companies would see the value in doing this Also love the humble brag. "I've just closed my 12th bug" and later "12 was maximum number of bugs closed by one person"
ChrisMarshallNY 11mo ago
I love the idea, but this line: > 1) no bug should take over 2 days Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. Also, I tend to attack bugs by priority/severity, as opposed to difficulty. Some of the most serious bugs are often quite easy to find. Once I find the cause of a bug, the fix is usually just around the corner.
- chii 11mo ago
  I find most bugs take less time to fix than it takes time to verify and reproduce.
  - wahnfrieden 11mo ago
    LLMs have helped me here the most. Adding copious detailed logging across the app on demand, then inspecting the logs to figure out the bug and even how to reproduce it.
    - ChrisMarshallNY 11mo ago
      Yes. I often just copy the whole core dump, and feed it into the prompt.
      - Lionga 11mo ago
        And this kids is how one bug got fixed and two more were created
        ChrisMarshallNY 11mo ago
        Nope. Good result == LLM + Experience. The LLM just reduces the overhead. That’s really what every “new paradigm” has ever done.
jchrisa 11mo ago
I just had a majorly fun time addressing tech debt, deleting about 15k lines-of-code from a codebase that now has ~45k lines of implementation, and 50k lines of tests. This was made possible by moving from a homegrown auth system to Clerk, as well as consolidating some Cloudflare workers, and other basic stuff. Not as fun as creating the tech debt in the first place, but much more satisfying. Open source repo if you like to read this sort of thing: https://github.com/VibesDIY/vibes.diy/pull/582 https://github.com/VibesDIY/vibes.diy/pull/582
- wredcoll 11mo ago
  I would be weirdly happy to have a role whose entire job was literally just deleting code. It is extremely satisfying.
  - cindyllm 11mo ago
    [dead]
- jchrisa 11mo ago
  Damn I just deleted another 29k loc https://github.com/VibesDIY/vibes.diy/pull/635 https://github.com/VibesDIY/vibes.diy/pull/635 Code slows you down, always worth cleaning up. Yes, the business case is aligned with both the past bloat, and the current cleanup.
tait1 11mo ago
We’ve done little mini competitions like this at my company, and it’s always great for morale. Celebrating tiny wins in a light, semi-competitive way goes a long way for elevating camaraderie. Love it!
kangs 11mo ago
hello b/Googler :)
flakiness 11mo ago
FYI, this article describes how traditional Google fixit was conducted: https://mike-bland.com/2011/10/04/fixits.html https://mike-bland.com/2011/10/04/fixits.html
Ethan312 11mo ago
Focused bug-fixing weeks like this really help improve product quality and team morale. It’s impressive to see the impact when everyone pitches in on these smaller but important issues that often get overlooked.
Cedricgc 11mo ago
One nice thing if you work on the B2B software side - end of year is generally slow in terms of new deals. Definitely a good idea to schedule bug bashes, refactors, and general tech debt payments with greater buy in from the business
Galxeagle 11mo ago
In my experience, having a fixit week on the calendar encourages teams to just defer what otherwise could be done relatively easily at first report. ("ah we'll get to it in fixit week"). Sometimes it's a PM justifying putting their feature ahead of product quality, other times it's because a dev thinks they're lining up work for an anticipated new hire's onboarding. It's even hinted at in the article ('All year round, we encourage everyone to tag bugs as “good fixit candidates” as they encounter them.') My preferred approach is to explicitly plan in 'keep the lights on' capacity into the quarter/sprint/etc in much the same way that oncall/incident handling is budgeted for. With the right guidelines, it gives the air cover for an engineer to justify spending the time to fix it right away and builds a culture of constantly making small tweaks. That said, I totally resonate with the culture aspect - I think I'd just expand the scope of the week-long event to include enhancements and POCs like a quasi hackathon
OhMeadhbh 11mo ago
How did you not get fired?
neilv 11mo ago
> We also have a “points system” for bugs and a leaderboard showing how many points people have. [...] It’s a simple structure, but it works surprisingly well. What good and bad experiences have people had with software development metrics leaderboards?
- watty 11mo ago
  I've never had a good experience with individual metrics leaderboards. On one team we had a JIRA story point tracker shown on a tv by a clueless exec. Devs did everything they could to game the system and tasks that required uncertainty (hard tasks) went undone. I believe it contributed to the cog culture that caused an exodus of developers. However, I love the idea of an occasional team based leaderboard for an event. I've held bug and security hackathons with teams of 3-5 and have had no problem with them.
riwsky 11mo ago
So much of the tech debt work scheduling feels like a coordination or cover problem. We’re overdue for a federal “Tech Debt Week” holiday once a year, and just save people all the hand-wringing of how when or how much. If big tech brands can keep affording to celebrate April fools jokes, they can afford to celebrate this.
siliconc0w 11mo ago
I'm a bit torn on Fix-it weeks. They are nice but many bugs simply aren't worth fixing. Generally, if they were worth fixing - they would have been fixed. I do appreciate though that certain people, often very good detail oriented engineers, find large backlogs incredibly frustrating so I support fix-it weeks even if there isn't clear business ROI.
- forgotoldacc 11mo ago
  > Generally, if they were worth fixing - they would have been fixed. ??? Basically any major software product accumulates a few issues over time. There's always a "we can fix that later" mindset and it all piles up. MacOS and Windows are both buggy messes. I think I speak for the vast majority of people when I say that I'd prefer they have a fix-it year and just get rid of all the issues instead of trying to rush new features out the door. Maybe rushing out features is good for more money now, but someday there'll be a straw that breaks the camel's back and they'll need to devote a lot of time to fix things or their products will be so bad that people will move to other options.
  - foxygen 11mo ago
    Oh boy, I’d trade one(or easily 2/3) major MacOs version for a year worth of bug fixes in a heartbeat.
    - Barbing 11mo ago
      You got it per Gurman: >For iOS 27 and next year’s other major operating system updates — including macOS 27 — the company is focused on improving the software’s quality and underlying performance. -via Bloomberg today
      - baq 11mo ago
        I’ll believe it when I see it, but holy quality Batman I want to believe.
      - Lionga 11mo ago
captainkrtek 11mo ago
A company I worked at also did this, though there was no limits. Some folks would choose to spend the whole week working on a larger refactor, for example, I unified all of our redis usage to use a single modern library compared to the mess of 3 libraries of various ages across our codebase. This was relatively easy, but tedious, and required some new tests/etc. Overall, I think this kind of thing is very positive for the health of building software, and morale to show that it is a priority to actually address these things.
j45 11mo ago
Fixing bugs before new code can shed interesting lights on how a dev team can become more effective.
troad 11mo ago
It's fairly telling of the state of the software industry that the exotic craft of 'fixing bugs' is apparently worth a LinkedIn-style self-promotional blog post. I don't mean to be too harsh on the author. They mean well. But I am saddened by the wider context, where a dev posts 'we fix bugs occasionally' and everyone is thrilled, because the idea of ensuring software continues to work well over time is now as alien to software dev as the idea of fair dealing is to used car salesmen.
- pjmlp 11mo ago
  That is why I stand on the side of better law for company responsibilities. We as industry have taught people that broken products is acceptable. In any other industry, unless people are from the start getting something they know is broken or low quality, flea market, 1 euro shop, or similar, they will return the product, ask for the money back, sue the company whatever.
  - k4rli 11mo ago
    Imagining that the software will be shipped with hardware, that has no internet access and therefore cumbersome firmware upgrades, might be helpful. Avoiding shipping critical bugs is actually critical so bricking the hardware is undesirable. Example: (aftermarket) car headunit.
    - zeroCalories 11mo ago
      This type of testing is incredibly expensive and you'll have a startup run circles around you, assuming a startup could even exist when the YC investment needs to stretch 4x as far for the same product. The real solution is to have individual software developers be licensed and personally liable for the damage their work does. Write horrible bugs? A licencing board will review your work. Make a calculated risk that damages someone? Company sued by the user, developer sued by the company. This correctly balances incentives between software quality and productivity, and has the added benefit of culling low quality workers.
      - 11mo ago
heyitsdaad 11mo ago
False sense of accomplishment. Doing what you want to do instead of what you should doing (hint: you should be busy making money). Inability to triage and live with imperfections. Not prioritizing business and democratizing decision making.
- ocimbote 11mo ago
  You criticize the initiative because you judge it doesn't have impact on the product or business. I would challenge the assumption with the claim that a sense of acconplishment, of decision-making and of completion are strong retention and productivity enhancers. Therefore, they're absolutely, albeit indirectly, impacting product and business.
- snovv_crash 11mo ago
  Just because you can't measure the loss of customers who are turned off by your buggy product doesn't mean they don't exist.
PeterStuer 11mo ago
Confused about the meaning of "bug" used in this artcle. It seems to be more about feature requests, nice to haves and polish rather than actual errors in edge cases. Also explains the casual mention of "estimation" on fixes. A real bug fix is even more hard to estimate than already brittle feature estimates.
boxed 11mo ago
I did this with my entire employment at a company I worked with. Or rather, I should say I made it a point to ignore the roadmap and do what was right for the company by optimizing for value for customers and the team. Fixit weeks is a band aid, and we also tried it. The real fix is being a good boss and trusting your coworkers to do their jobs.
stevoski 11mo ago
I’m a strong believer in “fix bugs first” - especially in the modern age of “always be deploying” web apps. (I run a small SaaS product - a micro-SaaS as some call it.) We’ll stop work on a new feature to fix a newly reported bug, even if it is a minor problem affecting just one person. Once you have been following a “fix bugs first” approach for a while, the newly discovered bugs tend to be few, and straight forward to reproduce and fix. This is not necessarily the best approach from a business perspective. But from the perspective of being proud of what we do, of making high quality software, and treating our customers well, it is a great approach. Oh, and customers love it when the bug they reported is fixed within hours or days.
- ivolimmen 11mo ago
  Would love to work on a project with this as a rule but I am working on a project that was build before me with 1.2 million lines of code, 15 years old, really old frameworks; I don't think we could add features if we did this.
  - chamomeal 11mo ago
    Same. The legacy project that powers all of our revenue-making projects at work is a gargantuan hulking php monster of the worst code I’ve ever seen. A lot of the internal behaviors ARE bugs that have been worked around, and become part of the arbitrary piles of logic that somehow serve customer needs. My own understanding of bugs in general has definitely changed.
- stevoski 11mo ago
  I wrote up my thoughts on this into a longer post: https://killthehippo.com/posts/fix-bugs-or-add-new-features https://killthehippo.com/posts/fix-bugs-or-add-new-features
jll29 11mo ago
From the report, it sounds like a good thing, for the product and the team morale. Strangely the math looks such that they could hire nearly 1 FTE engineer that works full time only on "little issues" (40 weeks, given that people have vacations and public holidays and sick time that's a full year's work at 100%), and then the small issues could be addressed immediately, modulo the good vibes created by dedicating the whole group to one cause for one week. Of course nobody would approve that role...
- joker99 11mo ago
  The unkind world we live in would see this role being abused quickly and a person not lasting long in this role. For one, in the wrong team, it might lead to devs just doing 80% of the work and leaving the rest to the janitor. And the janitor might get fed up with having to fix the buggy code of their colleagues. I wonder if the janitor role could be rotated weekly or so? Then everyone could also reap the benefits of this role too, I can imagine this being a good thing for anyone in terms of motivation. Fixing stuff triggers a different positive response than building stuff
  - lkbm 11mo ago
    I've had on-call roles where the idea was that in your spare time you worked on bug fixes (specifically the ones causing the most on-call noise). Unfortunately, that on-call was so overwhelmed that you were lucky to be able to handle all the alerts/crises, let alone having spare time to fix the root causes.
eviks 11mo ago
> closed a feature request from 2021! > It’s a classic fixit issue: a small improvement that never bubbled to the priority list. It took me one day to implement. One day for something that sat there for four years > The benefits of fixits > For the product: craftsmanship and care sorry, but this is not care when the priority system is so broken that it requires a full suspension, but only once a quarter > A hallmark of any good product is attention to detail: That's precisely the issue, taking 4 years to bring attention to detail, and only outside the main priority system. Now, don't get me wrong, a fixit is better than nothing and having 4 year bugs turn into 40 year ones, it's just that this is not a testament of craftsmanship/care/attention to detail
- lalitmaganti 11mo ago
  > this is not care when the priority system is so broken that it requires a full suspension I'm not sure I understand this line. The whole point of the fixit is to address the bugs which are considered "low priority" because they only appear in a edge case or are not quite 100% perfectly polished but still matter over the long tail of people using the product. Or do you propose that every issue like this needs to be fixed before doing anything else?
pmontra 11mo ago
About stopping and fixing problems, did anybody have had this kind of experience? 1. Working on Feature A, stopped by management or by the customer because we need Feature B as soon as possible. 2. Working on Feature B, stopped because there is Emergency C in production due to something that you warned the customer about months ago but there was no time to stop, analyze and fix. 3. Deployed a workaround and created issue D to fix it properly. 4. Postponed issue D because the workaround is deemed to be enough, resumed Feature B. 5. Stopped Feature B again because either Emergency E or new higher priority Feature F. At this point you can't remember what that original Feature A was about and you get a feeling that you're about to forget Feature B too. 6. Working on whatever the new thing is, you are interrupted by Emergency G that happened because that workaround at step 3 was only a workaround, as you correctly assessed, but again, no time to implement the proper fix D so you hack a new workaround. Maybe add another couple of iterations but at this time every party are angry or at least unhappy of each other party. You have a feeling that the work of the last two or three months on every single feature has been wasted because you could not deliver any one of them. That means that the customer wasted the money they paid you. Their problem, but it can't be good for their business so your problem too. The current state of the production system is "buggy and full of workarounds" and it's going to get worse. So you think that the customer would have been wiser to pause and fix all the nastier bugs before starting Feature A. We could have had a system running smoothly, no emergencies, and everybody happier. But no, so one starts thinking that maybe the best course of action is changing company or customer.
- abroszka33 11mo ago
  > did anybody have had this kind of experience? Yes, the issue is not you, it's a toxic workplace. Leave as soon as you can.
- pjc50 11mo ago
  This is not uncommon but I've mostly managed to avoid it, because it's a management failure. There is a delicate process of "managing the customer" so that they get a result they will eventually be satisfied with, rather than just saying yes to whatever the last phone call was.
- 11mo ago
alkonaut 11mo ago
We once did this for a massive product with 3 releases per year: took a whole cycle to do zero features, and just fix bugs. Internal customers who usually stepped over themselves to get their latest feature in the program, were accepting it. But we had to announce it early. Otherwise the usual consensus is that customers would rather take 1 feature together with 10 new bugs, than -5 bugs and no new features.
klabetron 11mo ago
I introduced this to my old company years ago and called it Big Block of Cheese Day after the West Wing episode [1]. We mostly focused on very minor bugs that affected a tiny bit of our user base in edgey edge cases but littered our error logs. (This was years ago at a, back then, relatively immature tech company.) It had the same spirit as a hackathon. [1] https://westwing.fandom.com/wiki/Big_Block_of_Cheese_Day https://westwing.fandom.com/wiki/Big_Block_of_Cheese_Day
- philipallstar 11mo ago
  When the world learned the wrong reason for why the Mercator Projection was adopted.
fsniper 11mo ago
I feel odd about "bug fixing" to be a special occasion than being the work. Features need to be added, so do bugs need to be fixed. Making it a special occasion makes it feel like some very low priority "grunt work" that requires a hard push to be looked at.
etamponi 11mo ago
Ex-Meta employee here. I worked at reality labs, perhaps in other orgs the situation is different. At Meta we did "fix-it weeks", more or less every quarter. At the beginning I was thrilled: leadership that actually cares about fixing bugs! Then reality hit: it's the worst possible decision for code and software quality. Basically this turned into: you are allowed to land all the possible crap you want. Then you have one week to "fix all the bugs". Guess what: most of the time we couldn't even fix a single bug because we were drown in tech debt.
- demaga 11mo ago
  From the post: > That’s not to say we don’t fix important bugs during regular work; we absolutely do. But fixits recognize that there should be a place for handling the “this is slightly annoying but never quite urgent enough” class of problems. So in their case, fixit week is mostly about smaller bugs, quality of life improvements and developer experience.
  - gregoriol 11mo ago
    It must be part of the normal process. If the normal process leaves things like this to "some other time", one should start by fixing the process.
    - IgorPartola 11mo ago
      Say you are working on a banking system. You ship a login form, it is deployed, used by tons of people. Six months later you are mid-sprint on the final leg of a project that will hook your bank into the new FedNow system. There are dozens of departments working together to coordinate deploying this new setup as large amounts of money will be moved through it. You are elbows deep in the context of your part of this and the system cannot go live without it. Twice a day you are getting QA feedback and need to make prompt updates to your code so the work doesn’t stall. This is when the report comes in that your login form update from six months ago does not work on mobile Opera if you disable JavaScript. The fix isn’t obvious and will require research, potentially many hours or even days of testing and since it is a login form you will need the QA team to test it after you find another developer on your team to do a code review for you. What exactly would you do in this case? Pull resources from a major project that has the full attention of the C suite to accommodate some tin foil Luddite a few weeks sooner or classify this as lower priority?
invalidusernam3 11mo ago
I like the idea of this, but why not just have some time per week/sprint for bugs? At my company we prioritise features, but we also take some bug tickets every sprint (sometimes loads of bug tickets if there aren't many new features ready for dev), and generally one engineer is on "prod support" which means tackling bugs as they get reported
- stuartjohnson12 11mo ago
  Because marginal work is only marginally rewarded. Spending one week and coming back to whoever with a nice piece of paper saying we fixed 60 bugs will earn a lot more rope from non-technical folk than fixing 3 bugs per week - the latter just looks like cleaning up your incompetence.
  - LocalPCGuy 11mo ago
    I suspect I'm preaching to the choir, but that is a communication issue and a sign the "rewards system" is out of whack, not a "reason" not to push for regular maintenance/tech debt/bug cleanup work. It should be understood that there WILL be bugs, that is NOT a sign of incompetence, and so cleaning them up should be an ongoing task so they do not linger and collect (and potentially get worse by compounding with other bugs).
radiator 11mo ago
It is good to fix bugs, but in my team we need neither the "points system” for bugs nor the leaderboard showing how many points people have. We are against quantifying.
Cthulhu_ 11mo ago
We did a bug hunt once at a previous employer, just stop regular work, open the website, and look for issues. We found over a hundred in a day. Stopping your regular work and actively work with your product is a healthy practice. Facebook did (does?) do a thing where once a week they'd throttle the internet so everyone had to experience what things are like for their average users.
lalitmaganti 11mo ago
Author here! Really glad to have sparked a lively discussion in the comments. Since there is so many threads since I last looked at this post, making one top level comment to provide some thoughts: 1) I agree that estimating a bug's complexity upfront is an error prone process. This is exactly why I say in the post that we encourage everyone to "feel out" non trivial issues and if it feels like the scope is expanding too much (after a few hours of investigation), to just pick something else after writing up their findings on the bug. 2) I use the word "bug" to refer to more traditional bugs ("X is wrong in product") but also feature requests ("I wish X feature worked differently"). This is just a companyism that maybe I should have called out in the post! 3) There's definitely a risk the fixit week turns into just "let's wait to fix bugs until that week". This is why our fixits are especially for small bugs which won't be fixed otherwise - it's not a replacement for technical hygiene (i.e. refactoring code, removing dead code, improving abstractions) nor a replacement for fixing big/important issues in a timely manner.
- danielbarla 11mo ago
  Very interesting post, thank you! I'd also be curious to know the following: how many new errors or regressions were caused by the bug fixes?
  - lalitmaganti 11mo ago
    Since the fixit just finished on Friday, I don't have hard numbers from this one I'm afraid :) Historically though, I would guess maybe 5-10% end up needing some followup fix which is itself usually smaller than the original (maybe a typo in some documentation or some edge case we spot when it hits prod etc). The smaller the original fixes, the less likely you are to need followups so another reason to prefer working mainly on them!
    - danielbarla 11mo ago
      I think 5-10% is pretty good, it probably means that the codebase is mostly understandable and maintainable. I have definitely worked on some which were full of little traps and landmines just waiting for eager do-gooders to step on, which was sadly a self-fulfilling prophecy for the app.
brightball 11mo ago
Getting ready to do a December “Bug Smash” based on the model in the book Shape Up. Whole team has been eagerly awaiting it for months.
- briandrum 11mo ago
  Shape Up was my first thought too. I just left a team where I introduced cycles of six weeks of feature development and two weeks of bug fixing, tech debt, and anything else the developers decided to tackle. It depends on the stage and size of your team and company of course, but for us the result was more predictable delivery and happier, more-engaged developers. For anyone curious to learn more: https://basecamp.com/shapeup/2.2-chapter-08#cool-down https://basecamp.com/shapeup/2.2-chapter-08#cool-down
nottorp 11mo ago
So normally they don't fix bugs before adding feature bloat?
a4isms 11mo ago
In the early days of Hacker News, and maybe even before Hacker News when Reddit didn't have subreddits... OG blogger Joel Spolsky posited the "Joel Test," twelve simple yes/no questions that defined a certain reasonable-by-today's-standards local optimum for shipping software: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s... Some seem ridiculously obvious today, but weren't standard 25 years ago. Seriously! At the turn of the century, not everyone used a bug database or ticket tracker. Lots of places had complicated builds to production, with error-prone manual steps. But question five is still relevant today: Do you fix bugs before writing new code?
molly0 11mo ago
We had a quarter, where each Monday we spent most of the day fixing bugs. It greatly improved the product.
tracker1 11mo ago
I've been pushing for things like this for years... Having every 3rd or 4th sprint being dev initiatives and bugs... Or having a long/short sprint cycle where short sprints are for bugs mostly... Basically every 3rd week is for meetings and bug work so you get a solid 2 weeks with reduced meetings. It's hard to convince upper managers of the utility though.
yujzgzc 11mo ago
There are really two kinds of "small bugs". 1) Things that have existed in your product for decades and haven't been major strategic issues. 2) Things that arose recently in the wake of launches. This can be because it's hard to fix every corner case, or because of individuals throwing sloppy code over the wall to look like they "ship fast". I try to hold the team to fix bugs (2) quickly while their memory is fresh as it points to unwanted regressions. The bugs in (1) are more interesting. It's a bit sad that teams kinda have to "sneak that work in" with fixit weeks. I have known of products large enough to be able to A/B test the effects of a quarter's worth of "small fixes", and finding significant gains in key product metrics. That changed management's attitude with respect to "small fixes" - when you have a ton of them, they can produce meaningful impact worthy of strategic consideration, not just a week of giving the dev team free rein to scratch their itch.
knallfrosch 11mo ago
It's concerning that noone was able to fix these bugs as an "aside". With an average of 4 bugs fixed in 5 days and 150 bugs, we can assume 50 bugs with less than one days's effort were just lying around with noone daring to touch them.
deleted 11mo ago
[deleted]
hermitcrab 11mo ago
Alternative title: "We only fix bugs for one week every quarter".
q2dg 11mo ago
Systemd should do this too
caycep 11mo ago
granted, I feel like fixing bugs should be pre-allocated to the road map adequately, vs. spending 1 giant cycle every 10 years catching up on bug fixes a la snow leopard (cough cough apple)
franciscator 11mo ago
Good done, otherwise technical debt will have stopped you
dgunay 11mo ago
An ex-employer of mine had a regular cycle: 1. Build features at all costs 2. Eventually a high profile client has a major issue during an event, costing them a ton of goodwill 3. Leadership pauses everything and the company only works on bugfixes and tech debt for a week or two I onboarded during step 3. I should have taken that as a warning that that's how the company operated. If your company doesn't make time for bugfixes and getting out of its own way, that culture is hard to change.
dzonga 11mo ago
I do wonder though if some of those google products were not part of google but independent co's what would happen ?
bears123 11mo ago
Teams that implement this, or similar, exercise: how do you handle PR Reviews for fixits, if at all? I'd like to implement, but at a smaller team (8 devs, 3 whom approve PRs) the volume would be so high that the 3 senior devs would likely spend all their time reviewing.
- LocalPCGuy 11mo ago
  In the spirit of that exercise, the fixes should not take an excessive amount of time to review. If they are, it's likely either the scope of the fix is too large for that kind of exercise, or the PR review process is too in-depth. I would also question why only 3 of 8 devs approve PRs. Even if that can't change more broadly all of the time, this kind of exercise seems like a perfect time to allow everyone to review PRs - two fold benefit, more fixes are reviewed and gives experience reviewing to others that don't get to do that regularly. So yes, definitely still do PRs, and if that is problematic, consider whether that is an indication the PR process may itself need to be reviewed.
internet101010 11mo ago
Microsoft should do this for an entire year. Windows 11 is still a bug-ridden heap of trash.
codingdave 11mo ago
> Then the week before fixit, each subteam goes through these bugs and sizes them: I advocate to never size/score bugs. Instead, if your process demands scores, call everything a 2 because over the course of all the bugs, that will be your average. You'll knock out 10 small ones and then get stuck on a big one. Bug-fixing efforts should be more Kanban than Scrum. Prioritize the most important/damaging/whatever ones, do them in order, and keep doing them until they are done or you run out of time.
fourseventy 11mo ago
I like Linears approach to bugs: https://linear.app/now/zero-bugs-policy https://linear.app/now/zero-bugs-policy
stevage 11mo ago
Surprising that no bug should take more than 2 days, yet most developers fixed only 4 bugs in 5 days.
deeptishukla22 11mo ago
Linear has a similar policy - https://linear.app/now/zero-bugs-policy https://linear.app/now/zero-bugs-policy

284 points by lalitmaganti 11mo ago

I firmly believe that this sort of fixit week is as much of an anti-pattern as all-features-all-the-time. Ensuring engineers have the agency and the space to fix things and refactor as part of the normal process pays serious dividends in the long run. eg: My last company's system was layer after layer built on top of the semi-technical founder's MVP. The total focus on features meant engineers worked solo most of the time and gave them few opportunities to coordinate and standardize. The result was a mess. Logic smeared across every layer, modules or microservices with overlapping responsibilities writing to the same tables and columns. Mass logging all at the error or info level. It was difficult to understand, harder to trace, and nearly every new feature started off with "well first we need to get out of this corner we find ourselves painted into". When I compare that experience with some other environments I've been in where engineering had more autonomy at the day-to-day level, it's clear to me that this company should have been able to move at least as quickly with half the engineers if they were given the space to coordinate ahead of a new feature and occasionally take the time to refactor things that got spaghettified over time.

lalitmaganti 11mo ago
As I pointed out in the "criticisms" section, I don't see fixit weeks as a replacement for good technical hygiene. To be clear, engineers have a lot of autonomy in my team to do what they want. People can and do fix things as they come up and are encouraged to refactor and pay down technical debt as part of their day to day work. It's more that even with this autonomy fixits bugs are underappreciated by everyone, even engineers. Having a week where we can address the balance does wonders.
- inhumantsar 11mo ago
  yeah that's fair, didn't mean to argue against them entirely. I've just been in too many environments where fixits or cooldowns or whatever were used in place of good hygiene.

julianlam 11mo ago

We did this ages ago at our company (back then we were making silly Facebook games, remember those?) It was by far the most fun, productive, and fulfilling week. It went on to shape the course of our development strategy when I started my own company. Regularly work on tech debt and actively applaud it when others do it too.

xnx 11mo ago

I've never understood why bugs get treated differently from new features. If there was a bug, the old feature was never completed. The time cost and benefits should be considered equally.

superxpro12 11mo ago
until we develop a way for MBA's with spreadsheets to quantify profit/loss w.r.t. bugs, it will never be valued.
- lapcat 11mo ago
  The solution is to never hire an MBA.
  - pixl97 11mo ago
    'Why are we getting bought out by a company that cut corners and hired MBAs, and then fired?'
sb8244 11mo ago
If the bug affects 1 customer and the feature affects the rest, is the old feature complete? It's not binary.
- zelphirkalt 11mo ago
  Yet engineers are pushed to give unknowable estimates of points and when things take "longer" (did you notice that shift right there?) they are either overdue, taking too long, or they don't, and to say: "It takes as long as it takes." is not accepted by middle management.
  - sb8244 11mo ago
    That's a strawman. It is not really related to the main point and I'm not sure of the point you're trying to make (maybe that tension exists?) Obviously things take as long as they take. I've always been an educator of this back to the business leadership. In my experience, most business people truly have no freaking clue how a product gets built and code gets shipped. Giving proactive updates (meaning not the day it was expected to be done according to last update) are important part of a professionals working life. There's always a tension between business and engineers. Engineers just generally don't do that well with tension and try to minimize it, or complain about it.

entropie 11mo ago

I wanted to take a look at some of these bug fixes, and one of the linked ones [1] seems more like a feature to me. So maybe it should be the week of "low priority" issues, or something like that. I don't mean to sound negative, I think it's a great idea. I do something like this at home from time to time. Just spend a day repairing and fixing things. Everything that has accumulated. 1: https://github.com/google/perfetto/issues/154 https://github.com/google/perfetto/issues/154

mulquin 11mo ago
To be fair, the blog post does not explicitly say anywhere that the week was for bug fixes only.

ls-a 11mo ago

189 bugs in one week. How many employees quit after that?

asdfman123 11mo ago
They said they only pick bugs that take 2 days to fix. Places where you can move fast and actually do things are actually far better places to work for. I mean the ones were you can show up, do 5 hours of really good work, and then slack off/leave a little early.
- kykat 11mo ago
  Too bad many places care more about how long you stay warming the seat than how useful the work done actually is.
- ls-a 11mo ago
  Nothing takes 2 days to fix. Those are definitely not bugs, like someone else mentioned
  - toast0 11mo ago
    You haven't seen the same kind of bugs I have, I guess. This kind of thing takes more than 2 days to fix, unless you're really good. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217637 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217637 Or this one https://security.stackexchange.com/questions/104845/dhe-rsa-pubkey-length-in-tls-1-2 https://security.stackexchange.com/questions/104845/dhe-rsa-... I can find more of these that I've run into if I look. I've had tricky bugs in my team's code too, but those don't result in public artifacts, and I'm responsible for all the code that runs on my server, regardless of who wrote it... And I also can't crash client code, regardless of who wrote it, even if my code just follows the RFC.
    - ls-a 11mo ago
      That's what I'm saying. Nothing takes 2 days to fix meaning it takes more time

BurningFrog 11mo ago

This is weird to me... The way I learned the trade, and usually worked, is that bug fixing always comes first! You don't work on new features until the old ones work as they should. This worked well for the teams I was on. Having a (AFAYK) bug free code base is incredibly useful!!

deleted 11mo ago
[deleted]
Celeo 11mo ago
Depending on the size of the team/org/company, working on anything other than the next feature is a hard sell to PM/PO/PgM/management.
- BurningFrog 11mo ago
  That's what I hear. I've had some mix of luck and skill in finding these jobs. Working with people you've worked with before helps with knowing what you're in for. I also don't really ask anyone, I just fix any bugs I find. That may not work in all organizations :)
  - zelphirkalt 11mo ago
    micro-managing middle manager: "Are all your other sprint tasks finished?" code reviewing coworker: "This shouldn't be done on this branch!" (OK, at least this is easy to fix by doing it on a separate branch.)
  - ramon156 11mo ago
    I can guarantee you this doesn't work in our team! you didn't make a ticket, so the PM has no idea what you're doing! Yes, a ticket takes 2 seconds. it also puts me off my focus :P but i guess measuring is more important than achieving
- NegativeK 11mo ago
  I've had to inform leadership that stability is a feature, just like anything else, and that you can't just expect it to happen without giving it time. One leader kind of listened. Sort of. I'm pretty sure I was lucky.

hastily3114 11mo ago

We do this too sometimes and I love it. When I work on my own projects I always stop and refactor/fix problems before adding any new features. I wish companies would see the value in doing this Also love the humble brag. "I've just closed my 12th bug" and later "12 was maximum number of bugs closed by one person"

ChrisMarshallNY 11mo ago

I love the idea, but this line: > 1) no bug should take over 2 days Is odd. It’s virtually impossible for me to estimate how long it will take to fix a bug, until the job is done. That said, unless fixing a bug requires a significant refactor/rewrite, I can’t imagine spending more than a day on one. Also, I tend to attack bugs by priority/severity, as opposed to difficulty. Some of the most serious bugs are often quite easy to find. Once I find the cause of a bug, the fix is usually just around the corner.

chii 11mo ago
I find most bugs take less time to fix than it takes time to verify and reproduce.
- wahnfrieden 11mo ago
  LLMs have helped me here the most. Adding copious detailed logging across the app on demand, then inspecting the logs to figure out the bug and even how to reproduce it.
  - ChrisMarshallNY 11mo ago
    Yes. I often just copy the whole core dump, and feed it into the prompt.
    - Lionga 11mo ago
      And this kids is how one bug got fixed and two more were created
      - ChrisMarshallNY 11mo ago
        Nope. Good result == LLM + Experience. The LLM just reduces the overhead. That’s really what every “new paradigm” has ever done.

jchrisa 11mo ago

I just had a majorly fun time addressing tech debt, deleting about 15k lines-of-code from a codebase that now has ~45k lines of implementation, and 50k lines of tests. This was made possible by moving from a homegrown auth system to Clerk, as well as consolidating some Cloudflare workers, and other basic stuff. Not as fun as creating the tech debt in the first place, but much more satisfying. Open source repo if you like to read this sort of thing: https://github.com/VibesDIY/vibes.diy/pull/582 https://github.com/VibesDIY/vibes.diy/pull/582

wredcoll 11mo ago
I would be weirdly happy to have a role whose entire job was literally just deleting code. It is extremely satisfying.
- cindyllm 11mo ago
  [dead]
jchrisa 11mo ago
Damn I just deleted another 29k loc https://github.com/VibesDIY/vibes.diy/pull/635 https://github.com/VibesDIY/vibes.diy/pull/635 Code slows you down, always worth cleaning up. Yes, the business case is aligned with both the past bloat, and the current cleanup.

tait1 11mo ago

We’ve done little mini competitions like this at my company, and it’s always great for morale. Celebrating tiny wins in a light, semi-competitive way goes a long way for elevating camaraderie. Love it!

kangs 11mo ago

hello b/Googler :)

flakiness 11mo ago

FYI, this article describes how traditional Google fixit was conducted: https://mike-bland.com/2011/10/04/fixits.html https://mike-bland.com/2011/10/04/fixits.html

Ethan312 11mo ago

Focused bug-fixing weeks like this really help improve product quality and team morale. It’s impressive to see the impact when everyone pitches in on these smaller but important issues that often get overlooked.

Cedricgc 11mo ago

One nice thing if you work on the B2B software side - end of year is generally slow in terms of new deals. Definitely a good idea to schedule bug bashes, refactors, and general tech debt payments with greater buy in from the business

Galxeagle 11mo ago

In my experience, having a fixit week on the calendar encourages teams to just defer what otherwise could be done relatively easily at first report. ("ah we'll get to it in fixit week"). Sometimes it's a PM justifying putting their feature ahead of product quality, other times it's because a dev thinks they're lining up work for an anticipated new hire's onboarding. It's even hinted at in the article ('All year round, we encourage everyone to tag bugs as “good fixit candidates” as they encounter them.') My preferred approach is to explicitly plan in 'keep the lights on' capacity into the quarter/sprint/etc in much the same way that oncall/incident handling is budgeted for. With the right guidelines, it gives the air cover for an engineer to justify spending the time to fix it right away and builds a culture of constantly making small tweaks. That said, I totally resonate with the culture aspect - I think I'd just expand the scope of the week-long event to include enhancements and POCs like a quasi hackathon

OhMeadhbh 11mo ago

How did you not get fired?

neilv 11mo ago

> We also have a “points system” for bugs and a leaderboard showing how many points people have. [...] It’s a simple structure, but it works surprisingly well. What good and bad experiences have people had with software development metrics leaderboards?

watty 11mo ago
I've never had a good experience with individual metrics leaderboards. On one team we had a JIRA story point tracker shown on a tv by a clueless exec. Devs did everything they could to game the system and tasks that required uncertainty (hard tasks) went undone. I believe it contributed to the cog culture that caused an exodus of developers. However, I love the idea of an occasional team based leaderboard for an event. I've held bug and security hackathons with teams of 3-5 and have had no problem with them.

riwsky 11mo ago

So much of the tech debt work scheduling feels like a coordination or cover problem. We’re overdue for a federal “Tech Debt Week” holiday once a year, and just save people all the hand-wringing of how when or how much. If big tech brands can keep affording to celebrate April fools jokes, they can afford to celebrate this.

siliconc0w 11mo ago

I'm a bit torn on Fix-it weeks. They are nice but many bugs simply aren't worth fixing. Generally, if they were worth fixing - they would have been fixed. I do appreciate though that certain people, often very good detail oriented engineers, find large backlogs incredibly frustrating so I support fix-it weeks even if there isn't clear business ROI.

forgotoldacc 11mo ago
> Generally, if they were worth fixing - they would have been fixed. ??? Basically any major software product accumulates a few issues over time. There's always a "we can fix that later" mindset and it all piles up. MacOS and Windows are both buggy messes. I think I speak for the vast majority of people when I say that I'd prefer they have a fix-it year and just get rid of all the issues instead of trying to rush new features out the door. Maybe rushing out features is good for more money now, but someday there'll be a straw that breaks the camel's back and they'll need to devote a lot of time to fix things or their products will be so bad that people will move to other options.
- foxygen 11mo ago
  Oh boy, I’d trade one(or easily 2/3) major MacOs version for a year worth of bug fixes in a heartbeat.
  - Barbing 11mo ago
    You got it per Gurman: >For iOS 27 and next year’s other major operating system updates — including macOS 27 — the company is focused on improving the software’s quality and underlying performance. -via Bloomberg today
    - baq 11mo ago
      I’ll believe it when I see it, but holy quality Batman I want to believe.
    - Lionga 11mo ago

captainkrtek 11mo ago

A company I worked at also did this, though there was no limits. Some folks would choose to spend the whole week working on a larger refactor, for example, I unified all of our redis usage to use a single modern library compared to the mess of 3 libraries of various ages across our codebase. This was relatively easy, but tedious, and required some new tests/etc. Overall, I think this kind of thing is very positive for the health of building software, and morale to show that it is a priority to actually address these things.

j45 11mo ago

Fixing bugs before new code can shed interesting lights on how a dev team can become more effective.

troad 11mo ago

It's fairly telling of the state of the software industry that the exotic craft of 'fixing bugs' is apparently worth a LinkedIn-style self-promotional blog post. I don't mean to be too harsh on the author. They mean well. But I am saddened by the wider context, where a dev posts 'we fix bugs occasionally' and everyone is thrilled, because the idea of ensuring software continues to work well over time is now as alien to software dev as the idea of fair dealing is to used car salesmen.

pjmlp 11mo ago
That is why I stand on the side of better law for company responsibilities. We as industry have taught people that broken products is acceptable. In any other industry, unless people are from the start getting something they know is broken or low quality, flea market, 1 euro shop, or similar, they will return the product, ask for the money back, sue the company whatever.
- k4rli 11mo ago
  Imagining that the software will be shipped with hardware, that has no internet access and therefore cumbersome firmware upgrades, might be helpful. Avoiding shipping critical bugs is actually critical so bricking the hardware is undesirable. Example: (aftermarket) car headunit.
  - zeroCalories 11mo ago
    This type of testing is incredibly expensive and you'll have a startup run circles around you, assuming a startup could even exist when the YC investment needs to stretch 4x as far for the same product. The real solution is to have individual software developers be licensed and personally liable for the damage their work does. Write horrible bugs? A licencing board will review your work. Make a calculated risk that damages someone? Company sued by the user, developer sued by the company. This correctly balances incentives between software quality and productivity, and has the added benefit of culling low quality workers.
    - 11mo ago

heyitsdaad 11mo ago

False sense of accomplishment. Doing what you want to do instead of what you should doing (hint: you should be busy making money). Inability to triage and live with imperfections. Not prioritizing business and democratizing decision making.

ocimbote 11mo ago
You criticize the initiative because you judge it doesn't have impact on the product or business. I would challenge the assumption with the claim that a sense of acconplishment, of decision-making and of completion are strong retention and productivity enhancers. Therefore, they're absolutely, albeit indirectly, impacting product and business.
snovv_crash 11mo ago
Just because you can't measure the loss of customers who are turned off by your buggy product doesn't mean they don't exist.

PeterStuer 11mo ago

Confused about the meaning of "bug" used in this artcle. It seems to be more about feature requests, nice to haves and polish rather than actual errors in edge cases. Also explains the casual mention of "estimation" on fixes. A real bug fix is even more hard to estimate than already brittle feature estimates.

boxed 11mo ago

I did this with my entire employment at a company I worked with. Or rather, I should say I made it a point to ignore the roadmap and do what was right for the company by optimizing for value for customers and the team. Fixit weeks is a band aid, and we also tried it. The real fix is being a good boss and trusting your coworkers to do their jobs.

stevoski 11mo ago

I’m a strong believer in “fix bugs first” - especially in the modern age of “always be deploying” web apps. (I run a small SaaS product - a micro-SaaS as some call it.) We’ll stop work on a new feature to fix a newly reported bug, even if it is a minor problem affecting just one person. Once you have been following a “fix bugs first” approach for a while, the newly discovered bugs tend to be few, and straight forward to reproduce and fix. This is not necessarily the best approach from a business perspective. But from the perspective of being proud of what we do, of making high quality software, and treating our customers well, it is a great approach. Oh, and customers love it when the bug they reported is fixed within hours or days.

ivolimmen 11mo ago
Would love to work on a project with this as a rule but I am working on a project that was build before me with 1.2 million lines of code, 15 years old, really old frameworks; I don't think we could add features if we did this.
- chamomeal 11mo ago
  Same. The legacy project that powers all of our revenue-making projects at work is a gargantuan hulking php monster of the worst code I’ve ever seen. A lot of the internal behaviors ARE bugs that have been worked around, and become part of the arbitrary piles of logic that somehow serve customer needs. My own understanding of bugs in general has definitely changed.
stevoski 11mo ago
I wrote up my thoughts on this into a longer post: https://killthehippo.com/posts/fix-bugs-or-add-new-features https://killthehippo.com/posts/fix-bugs-or-add-new-features

jll29 11mo ago

From the report, it sounds like a good thing, for the product and the team morale. Strangely the math looks such that they could hire nearly 1 FTE engineer that works full time only on "little issues" (40 weeks, given that people have vacations and public holidays and sick time that's a full year's work at 100%), and then the small issues could be addressed immediately, modulo the good vibes created by dedicating the whole group to one cause for one week. Of course nobody would approve that role...

joker99 11mo ago
The unkind world we live in would see this role being abused quickly and a person not lasting long in this role. For one, in the wrong team, it might lead to devs just doing 80% of the work and leaving the rest to the janitor. And the janitor might get fed up with having to fix the buggy code of their colleagues. I wonder if the janitor role could be rotated weekly or so? Then everyone could also reap the benefits of this role too, I can imagine this being a good thing for anyone in terms of motivation. Fixing stuff triggers a different positive response than building stuff
- lkbm 11mo ago
  I've had on-call roles where the idea was that in your spare time you worked on bug fixes (specifically the ones causing the most on-call noise). Unfortunately, that on-call was so overwhelmed that you were lucky to be able to handle all the alerts/crises, let alone having spare time to fix the root causes.

eviks 11mo ago

> closed a feature request from 2021! > It’s a classic fixit issue: a small improvement that never bubbled to the priority list. It took me one day to implement. One day for something that sat there for four years > The benefits of fixits > For the product: craftsmanship and care sorry, but this is not care when the priority system is so broken that it requires a full suspension, but only once a quarter > A hallmark of any good product is attention to detail: That's precisely the issue, taking 4 years to bring attention to detail, and only outside the main priority system. Now, don't get me wrong, a fixit is better than nothing and having 4 year bugs turn into 40 year ones, it's just that this is not a testament of craftsmanship/care/attention to detail

lalitmaganti 11mo ago
> this is not care when the priority system is so broken that it requires a full suspension I'm not sure I understand this line. The whole point of the fixit is to address the bugs which are considered "low priority" because they only appear in a edge case or are not quite 100% perfectly polished but still matter over the long tail of people using the product. Or do you propose that every issue like this needs to be fixed before doing anything else?

pmontra 11mo ago

About stopping and fixing problems, did anybody have had this kind of experience? 1. Working on Feature A, stopped by management or by the customer because we need Feature B as soon as possible. 2. Working on Feature B, stopped because there is Emergency C in production due to something that you warned the customer about months ago but there was no time to stop, analyze and fix. 3. Deployed a workaround and created issue D to fix it properly. 4. Postponed issue D because the workaround is deemed to be enough, resumed Feature B. 5. Stopped Feature B again because either Emergency E or new higher priority Feature F. At this point you can't remember what that original Feature A was about and you get a feeling that you're about to forget Feature B too. 6. Working on whatever the new thing is, you are interrupted by Emergency G that happened because that workaround at step 3 was only a workaround, as you correctly assessed, but again, no time to implement the proper fix D so you hack a new workaround. Maybe add another couple of iterations but at this time every party are angry or at least unhappy of each other party. You have a feeling that the work of the last two or three months on every single feature has been wasted because you could not deliver any one of them. That means that the customer wasted the money they paid you. Their problem, but it can't be good for their business so your problem too. The current state of the production system is "buggy and full of workarounds" and it's going to get worse. So you think that the customer would have been wiser to pause and fix all the nastier bugs before starting Feature A. We could have had a system running smoothly, no emergencies, and everybody happier. But no, so one starts thinking that maybe the best course of action is changing company or customer.

abroszka33 11mo ago
> did anybody have had this kind of experience? Yes, the issue is not you, it's a toxic workplace. Leave as soon as you can.
pjc50 11mo ago
This is not uncommon but I've mostly managed to avoid it, because it's a management failure. There is a delicate process of "managing the customer" so that they get a result they will eventually be satisfied with, rather than just saying yes to whatever the last phone call was.
11mo ago

alkonaut 11mo ago

We once did this for a massive product with 3 releases per year: took a whole cycle to do zero features, and just fix bugs. Internal customers who usually stepped over themselves to get their latest feature in the program, were accepting it. But we had to announce it early. Otherwise the usual consensus is that customers would rather take 1 feature together with 10 new bugs, than -5 bugs and no new features.

klabetron 11mo ago

I introduced this to my old company years ago and called it Big Block of Cheese Day after the West Wing episode [1]. We mostly focused on very minor bugs that affected a tiny bit of our user base in edgey edge cases but littered our error logs. (This was years ago at a, back then, relatively immature tech company.) It had the same spirit as a hackathon. [1] https://westwing.fandom.com/wiki/Big_Block_of_Cheese_Day https://westwing.fandom.com/wiki/Big_Block_of_Cheese_Day

philipallstar 11mo ago
When the world learned the wrong reason for why the Mercator Projection was adopted.

fsniper 11mo ago

I feel odd about "bug fixing" to be a special occasion than being the work. Features need to be added, so do bugs need to be fixed. Making it a special occasion makes it feel like some very low priority "grunt work" that requires a hard push to be looked at.

etamponi 11mo ago

Ex-Meta employee here. I worked at reality labs, perhaps in other orgs the situation is different. At Meta we did "fix-it weeks", more or less every quarter. At the beginning I was thrilled: leadership that actually cares about fixing bugs! Then reality hit: it's the worst possible decision for code and software quality. Basically this turned into: you are allowed to land all the possible crap you want. Then you have one week to "fix all the bugs". Guess what: most of the time we couldn't even fix a single bug because we were drown in tech debt.

demaga 11mo ago
From the post: > That’s not to say we don’t fix important bugs during regular work; we absolutely do. But fixits recognize that there should be a place for handling the “this is slightly annoying but never quite urgent enough” class of problems. So in their case, fixit week is mostly about smaller bugs, quality of life improvements and developer experience.
- gregoriol 11mo ago
  It must be part of the normal process. If the normal process leaves things like this to "some other time", one should start by fixing the process.
  - IgorPartola 11mo ago
    Say you are working on a banking system. You ship a login form, it is deployed, used by tons of people. Six months later you are mid-sprint on the final leg of a project that will hook your bank into the new FedNow system. There are dozens of departments working together to coordinate deploying this new setup as large amounts of money will be moved through it. You are elbows deep in the context of your part of this and the system cannot go live without it. Twice a day you are getting QA feedback and need to make prompt updates to your code so the work doesn’t stall. This is when the report comes in that your login form update from six months ago does not work on mobile Opera if you disable JavaScript. The fix isn’t obvious and will require research, potentially many hours or even days of testing and since it is a login form you will need the QA team to test it after you find another developer on your team to do a code review for you. What exactly would you do in this case? Pull resources from a major project that has the full attention of the C suite to accommodate some tin foil Luddite a few weeks sooner or classify this as lower priority?

invalidusernam3 11mo ago

I like the idea of this, but why not just have some time per week/sprint for bugs? At my company we prioritise features, but we also take some bug tickets every sprint (sometimes loads of bug tickets if there aren't many new features ready for dev), and generally one engineer is on "prod support" which means tackling bugs as they get reported

stuartjohnson12 11mo ago
Because marginal work is only marginally rewarded. Spending one week and coming back to whoever with a nice piece of paper saying we fixed 60 bugs will earn a lot more rope from non-technical folk than fixing 3 bugs per week - the latter just looks like cleaning up your incompetence.
- LocalPCGuy 11mo ago
  I suspect I'm preaching to the choir, but that is a communication issue and a sign the "rewards system" is out of whack, not a "reason" not to push for regular maintenance/tech debt/bug cleanup work. It should be understood that there WILL be bugs, that is NOT a sign of incompetence, and so cleaning them up should be an ongoing task so they do not linger and collect (and potentially get worse by compounding with other bugs).

radiator 11mo ago

It is good to fix bugs, but in my team we need neither the "points system” for bugs nor the leaderboard showing how many points people have. We are against quantifying.

Cthulhu_ 11mo ago

We did a bug hunt once at a previous employer, just stop regular work, open the website, and look for issues. We found over a hundred in a day. Stopping your regular work and actively work with your product is a healthy practice. Facebook did (does?) do a thing where once a week they'd throttle the internet so everyone had to experience what things are like for their average users.

lalitmaganti 11mo ago

Author here! Really glad to have sparked a lively discussion in the comments. Since there is so many threads since I last looked at this post, making one top level comment to provide some thoughts: 1) I agree that estimating a bug's complexity upfront is an error prone process. This is exactly why I say in the post that we encourage everyone to "feel out" non trivial issues and if it feels like the scope is expanding too much (after a few hours of investigation), to just pick something else after writing up their findings on the bug. 2) I use the word "bug" to refer to more traditional bugs ("X is wrong in product") but also feature requests ("I wish X feature worked differently"). This is just a companyism that maybe I should have called out in the post! 3) There's definitely a risk the fixit week turns into just "let's wait to fix bugs until that week". This is why our fixits are especially for small bugs which won't be fixed otherwise - it's not a replacement for technical hygiene (i.e. refactoring code, removing dead code, improving abstractions) nor a replacement for fixing big/important issues in a timely manner.

danielbarla 11mo ago
Very interesting post, thank you! I'd also be curious to know the following: how many new errors or regressions were caused by the bug fixes?
- lalitmaganti 11mo ago
  Since the fixit just finished on Friday, I don't have hard numbers from this one I'm afraid :) Historically though, I would guess maybe 5-10% end up needing some followup fix which is itself usually smaller than the original (maybe a typo in some documentation or some edge case we spot when it hits prod etc). The smaller the original fixes, the less likely you are to need followups so another reason to prefer working mainly on them!
  - danielbarla 11mo ago
    I think 5-10% is pretty good, it probably means that the codebase is mostly understandable and maintainable. I have definitely worked on some which were full of little traps and landmines just waiting for eager do-gooders to step on, which was sadly a self-fulfilling prophecy for the app.

brightball 11mo ago

Getting ready to do a December “Bug Smash” based on the model in the book Shape Up. Whole team has been eagerly awaiting it for months.

briandrum 11mo ago
Shape Up was my first thought too. I just left a team where I introduced cycles of six weeks of feature development and two weeks of bug fixing, tech debt, and anything else the developers decided to tackle. It depends on the stage and size of your team and company of course, but for us the result was more predictable delivery and happier, more-engaged developers. For anyone curious to learn more: https://basecamp.com/shapeup/2.2-chapter-08#cool-down https://basecamp.com/shapeup/2.2-chapter-08#cool-down

nottorp 11mo ago

So normally they don't fix bugs before adding feature bloat?

a4isms 11mo ago

In the early days of Hacker News, and maybe even before Hacker News when Reddit didn't have subreddits... OG blogger Joel Spolsky posited the "Joel Test," twelve simple yes/no questions that defined a certain reasonable-by-today's-standards local optimum for shipping software: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s... Some seem ridiculously obvious today, but weren't standard 25 years ago. Seriously! At the turn of the century, not everyone used a bug database or ticket tracker. Lots of places had complicated builds to production, with error-prone manual steps. But question five is still relevant today: Do you fix bugs before writing new code?

molly0 11mo ago

We had a quarter, where each Monday we spent most of the day fixing bugs. It greatly improved the product.

tracker1 11mo ago

I've been pushing for things like this for years... Having every 3rd or 4th sprint being dev initiatives and bugs... Or having a long/short sprint cycle where short sprints are for bugs mostly... Basically every 3rd week is for meetings and bug work so you get a solid 2 weeks with reduced meetings. It's hard to convince upper managers of the utility though.

yujzgzc 11mo ago

There are really two kinds of "small bugs". 1) Things that have existed in your product for decades and haven't been major strategic issues. 2) Things that arose recently in the wake of launches. This can be because it's hard to fix every corner case, or because of individuals throwing sloppy code over the wall to look like they "ship fast". I try to hold the team to fix bugs (2) quickly while their memory is fresh as it points to unwanted regressions. The bugs in (1) are more interesting. It's a bit sad that teams kinda have to "sneak that work in" with fixit weeks. I have known of products large enough to be able to A/B test the effects of a quarter's worth of "small fixes", and finding significant gains in key product metrics. That changed management's attitude with respect to "small fixes" - when you have a ton of them, they can produce meaningful impact worthy of strategic consideration, not just a week of giving the dev team free rein to scratch their itch.

knallfrosch 11mo ago

It's concerning that noone was able to fix these bugs as an "aside". With an average of 4 bugs fixed in 5 days and 150 bugs, we can assume 50 bugs with less than one days's effort were just lying around with noone daring to touch them.

deleted 11mo ago

[deleted]

hermitcrab 11mo ago

Alternative title: "We only fix bugs for one week every quarter".

q2dg 11mo ago

Systemd should do this too

caycep 11mo ago

granted, I feel like fixing bugs should be pre-allocated to the road map adequately, vs. spending 1 giant cycle every 10 years catching up on bug fixes a la snow leopard (cough cough apple)

franciscator 11mo ago

Good done, otherwise technical debt will have stopped you

dgunay 11mo ago

An ex-employer of mine had a regular cycle: 1. Build features at all costs 2. Eventually a high profile client has a major issue during an event, costing them a ton of goodwill 3. Leadership pauses everything and the company only works on bugfixes and tech debt for a week or two I onboarded during step 3. I should have taken that as a warning that that's how the company operated. If your company doesn't make time for bugfixes and getting out of its own way, that culture is hard to change.

dzonga 11mo ago

I do wonder though if some of those google products were not part of google but independent co's what would happen ?

bears123 11mo ago

Teams that implement this, or similar, exercise: how do you handle PR Reviews for fixits, if at all? I'd like to implement, but at a smaller team (8 devs, 3 whom approve PRs) the volume would be so high that the 3 senior devs would likely spend all their time reviewing.

LocalPCGuy 11mo ago
In the spirit of that exercise, the fixes should not take an excessive amount of time to review. If they are, it's likely either the scope of the fix is too large for that kind of exercise, or the PR review process is too in-depth. I would also question why only 3 of 8 devs approve PRs. Even if that can't change more broadly all of the time, this kind of exercise seems like a perfect time to allow everyone to review PRs - two fold benefit, more fixes are reviewed and gives experience reviewing to others that don't get to do that regularly. So yes, definitely still do PRs, and if that is problematic, consider whether that is an indication the PR process may itself need to be reviewed.

internet101010 11mo ago

Microsoft should do this for an entire year. Windows 11 is still a bug-ridden heap of trash.

codingdave 11mo ago

> Then the week before fixit, each subteam goes through these bugs and sizes them: I advocate to never size/score bugs. Instead, if your process demands scores, call everything a 2 because over the course of all the bugs, that will be your average. You'll knock out 10 small ones and then get stuck on a big one. Bug-fixing efforts should be more Kanban than Scrum. Prioritize the most important/damaging/whatever ones, do them in order, and keep doing them until they are done or you run out of time.

fourseventy 11mo ago

I like Linears approach to bugs: https://linear.app/now/zero-bugs-policy https://linear.app/now/zero-bugs-policy

stevage 11mo ago

Surprising that no bug should take more than 2 days, yet most developers fixed only 4 bugs in 5 days.

deeptishukla22 11mo ago

Linear has a similar policy - https://linear.app/now/zero-bugs-policy https://linear.app/now/zero-bugs-policy