34 ms·

We invested 10% to pay back tech debt

• romanhn 4y ago
  This was a failure of product management and, doubly so, engineering management. A PM working in software absolutely needs to understand the concept of technical debt. Not necessarily to know where the debt is on their own, butand to work closely with engineer
• asmyers1793 4y ago
  I'm starting to check out grit.io for technical debt help
• hanifbbz 4y ago
  Story from a job I had couple of years ago. At the time I took it for granted but I haven’t seen something quite like it before or after.
• cratermoon 4y ago
  I've seen it said that in any ongoing work on a large and complex software product, about 20% of the time is spent on upkeep just to keep the system from degrading further. Probably your team was already spending a portion of their time shoring up the worst parts. That's typical, and why the build up of problems slows down feature work. I always plan on spending some percent of my time on work I know I'll have to do before I can start adding the new features. Regardless of how I'm asked to estimate, I keep in mind the cleanup work in accounting for the total effort. Unfortunately there are still a lot of PMs and folks, including some programmers, who aren't aware of or don't understand the need for structural maintenance. Those are the teams I regularly see start out going very fast but within a few months get bogged down in having to work in the awful system they built.
  • no_wizard 4y ago
    Story old as time. Management typically prioritizes features over other types of work. If your ENG dept has weak leadership it becomes daunting to fight for the 20% required for maintenance of the system. Then at some unknown point things slow down in a noticeable way and everyone scrambles to fix the slow down without fixing the culture that lead to it, and that’s if you’re lucky. Somehow in management eyes rewrites are more palatable than refactors despite costing 10x more
    • xvilka 4y ago
      > Somehow in management eyes rewrites are more palatable than refactors despite costing 10x more Because many of them are either incompetents or expect to run to another project or company by the time things start falling apart.
    • DangitBobby 4y ago
      There's also a psychological component for the devs. Asking me to fix your trash heap? Bye! Asking me to rewrite your trash heap? Sounds fun. In all of these discussions about reinventing/rewriting/reimplementing already solved x, people forget that other people aren't interested in maintaining something where the fun, education, and impact was already had by the people who came first! They want to have their own fun, education, or impact. Pretty much every conversation that starts with an angry "this has been a solved problem since such and such genius from the 70's did y" is born from misunderstanding what drives people to do anything.
    • atom_arranger 4y ago
      I usually tend to do the refactoring without asking permission. Just spend 20% of my time on it, get it done, definitely pays off. I tend to be pretty productive generally and with this refactoring I'm not doing something like saying "give me a week to refactor from X to Y", instead I just do a little bit every day or two, so I've never had much problem with this from management. If you're delivering features quickly it shouldn't matter if you spend some of your time making sure you'll be able to continue doing so in the future.
      • zomgwat 4y ago
        Exactly this. I also don’t ask permission because I consider this type of refactoring an implicit and fundamental responsibility.
    • dragonwriter 4y ago
      > Somehow in management eyes rewrites are more palatable than refactors despite costing 10x more In part, it is resume-driven development. Rewrites are major projects, and running them (and the associated scale and budget) looks good on a management resume, and provides a nice accomplishment item. In part, its the short-term incentive to be seen to do more with less, and move on before the consequences bite (this is also a kind of resume-driven motivation, though its more resume-driven development avoidance.) You defer work on anything but visible features, and do those in the quickest/cheapest possible way that will work in the short term, and use the credit for efficiency to move up and out before the deferred maintenance catches up. When it does, someone else gets to do a resume-enhancing big replacement project, since the state of the app makes fixing in place seem impractical. (Ironically, even with deferred maintenance, incremental remediation would probably be quicker, more efficient, and less prone to major timeline and budget-busting surprises, but when it has gotten bad enough, even the technical people who might recognize that will back management’s desire for a complete replacement because they don’t want to have to deal with the legacy mess.)
  • rqtwteye 4y ago
    I always tell the young devs that they shouldn't ask for dedicated refactoring time but it's just part of professional development. You have to trust that using good practices consistently will help you in the long term and actually speed up development.
    • DangitBobby 4y ago
      Sometimes people learn and grow and want to apply that to old code that they wrote. Are they given time to do that?
      • cratermoon 4y ago
        If they are having to touch that old code to add new features, fix bugs, migrate to a new context (say, OS version), or improve the performance/memory usage/responsiveness, then they should be improving it. Whether or not the developers are "given" time, making the change will take the time. That time can either be because the system is bad and making changes is hard, or because the system gets improved to make the changes possible.
• WniAggTCerH 4y ago
  [dead]
• andrewstuart 4y ago
  I might be doing it wrong but I constantly pay down technical debt to almost zero. The reason I do this is because when I am working on an area of system it takes alot of thinking to understand it, thus I try to get as much done as possible in that context, because re-learning the context later is very expensive in terms of time. I'm pretty certain that means new features are implemented alot more easily. Hard to quantify though. I can say I'm always thanking "yesterday me" for finishing the job and cleaning up whatever "today me" is working on. It's possible for me to do this because I do most of my programming on my own. I don't think this approach is suitable for most software engineering processes because they tend to deprioritise technical debt.
  • bobkazamakis 4y ago
    I agree with this, but it also requires some skilled automation or very well-defined boundaries; otherwise you spend all of your time maintaining and not enough time implementing features.
  • abc_lisper 4y ago
    That’s hardwon wisdom! I think it’s possible to do this elsewhere if you keep it simple, restrict yourself to a feature set that one understands well.
  • andy_ppp 4y ago
    I call this “fighting the good fight”. Hardly anywhere will prioritise making your life better, I say don’t tell them and sneak in tidy ups and refactoring as part of every task. Essentially a codebase is either getting better or worse over time, I prefer to work towards improvement rather than ossification. Having said that in some places and on some days (and working with some people) I don’t always have perfect energy for the fight.
    • omgwtf1000 4y ago
      This is the approach I take too. I’ll refactor code / UI cleanup in the area I’m working in. Sometimes QA complains if they review commits, but I’ve never had to revert changes because of it.
• nisa 4y ago
  Good article and a good inspiration to debate this topic. Honestly I already get in a bad mood when I read this conversation with the PM - this mentality "We need to timebox that activity so it does not swallow the time that should go to features and bug fixes." - depending on the tech debt and state of the codebase devs are wasting enormous amounts of time working around stupid issues (i.e. tech debt) and even a little bit of team effort might help - there is maybe some company out there where someone argues for rewriting the almost bug free haskell code to use higher kinded types and lenses because at the moment it's not so looking well but I guess in reality often there is just a lot of shitty cruft in java/javascript land that a few weeks of thinking and fixing (in the team, with lot's of discussions) improves much more and gives more time for features than avoiding that.
  • zdragnar 4y ago
    This is one area where velocity tracking actually helps PMs understand what you're talking about. If you follow the "boy scout" rule of leaving every file a little better every time you touch it, you will be slowly cleaning up debt while also slowing down velocity. If that is not an acceptable trade-off to the PM, then they need a proper estimate for what the cleanup will take- that's the armor they have to shield you from shit raining down from higher levels when someone wants to know why the new feature timeline is being delayed. If your tech debt is so bad it can't be incrementally improved, then it can usually be reframed as building a new version of feature X when really you're just rewriting it. Sell it as a performance / stability improvement and voila, it can be prioritized appropriately. If you have tried all of the above and your work priorities are still being dictated, either the company is in a tight spot (contracts require features on a tight timeline) and you need to muddle through until the next phase, or the company isn't regarding you as a professional and it is time to move on. No skilled tradesman worth their salt would let management tell them not to clean the shop because that takes time away from building things. The professional dictates the work practices.
    • deleted 4y ago
      [deleted]
  • megablast 4y ago
• torben-friis 4y ago
  I’ve seen “x% time for tech debt“ rules in several companies and sadly it didn’t work too well. Since the problem was the culture of continually pushing half baked features in the first place, the rule was quickly corrupted: people would design a good system, throw anything that’s not required for a POC into the tech debt backlog and deliver a barely functioning version. “This is a technical debt task” was used to prevent everything that wasnt new Features taking time of the other 90% of the sprint. Basically, if you assign a block of time to quality, you risk people taking that as an excuse to not focus on quality outside that block.
  • neilv 4y ago
    I've also seen this sprint-task-fake-completed later cleanup misattributed to "refactoring", which I've started calling "fudge-factoring". IMHO, at the point you have engineers misrepresenting that something is done, even if you're not in an application domain in which engineers go to jail for that, you take a step back and fix your engineering/product culture. That might include nuking your current methodology from orbit, just to reinforce that organization needs to learn a new way of thinking. (Not just "don't do that" on one symptom, when whatever culture weakness that caused the fudging is still in play.)
    • josephg 4y ago
      Yep. Or in other words, you (usually) can’t solve cultural problems with technical solutions. If your team culture doesn’t value long term thinking, it’ll take more than a schedule change to introduce that mentality. In the story in this post, one important thing that happened was that the team (with management) came together to acknowledge the tech debt problem, acknowledge they want to solve it and talk through potential solutions. This moves the problem from something individuals care about to something the team as a whole acknowledges it care about (a shift from individual knowledge to common knowledge). Once everyone agrees this is a problem, individual engineers will know they’re acting against the will of the tribe by writing lazy pull requests. And being reprimanded by their coworkers for doing so will have much more weight.
      • 4y ago
• DangitBobby 4y ago
  This environment sounds stifling. Very glad most of the time there's no one trying to inject tickets between me and fixing broken shit.
• GuB-42 4y ago
  Before clicking on the link, I expected failure. Turned out it is a success story, and I am a bit disappointed. I have seen "code cleanup" that made things worse countless times. Especially if the idea is to dedicate time just for that, instead of doing it as you go. I expected the article to be about one of these failures. The article describes 7 points, and suspiciously nothing goes wrong. It is rarely how it happens, there is typically some trial and error involved, and knowing what failed is as important if not more so than the final solution that works for you.
• gumby 4y ago
  IMHO this was the best part of the (interesting) post: > Having dealt with tech debt in a collaborative manner, enabled us to do the “regular work” faster because we had a better collective understanding of the code, and the code was cleaner to work with. One could argue this is just a positive effect of mob programming, but the lack of a concrete agenda also helped the autonomy that unlocked creativity.
• floatinglotus 4y ago
  You’ll never end up with any technical debt if you don’t ship.
• vb-8448 4y ago
  > Decreasing MTBF (mean time between failure): the software fails more often and there are increasingly more incidents. Is this actually true? In my experience, in the long term the failure rate tends to zero
• zmmmmm 4y ago
  The types of tech debt described don't really match well to the real problem ones I see. These are major decisions deferred / delayed that never got rectified and the cost has risen exponentially to address them. To get them done requires a total stop-the-world type effort with simultaneous migrations across multiple teams / services / products.
• llsf 4y ago
  I had started the "Documentation afternoon" on Fridays in my team. And everyone seemed to like it, to wind down the week with uninterrupted time for documentation (email catch up, expenses, etc.)
• ramesh31 4y ago
  Maybe I'm an un-organized mess. But there's no such thing as "x%" of my time. Theres's just time. And I'm either focused on doing something or I'm not. Management loves to think that they can parcel out time in that way. But it's a complete fantasy.
  • xyzelement 4y ago
    I used to feel that way but now I appreciate the usefulness of "% of time" concept for aligning individual or team effort. For example, if my boss and I agree that I should be focused 80% on new business development and 20% on keeping existing customers happy, it doesn't mean I allocate every hour or every day this way, but it does mean that I pay a lot more attention to A vs B, but don't neglect B altogether. That's a useful way to agree on what is important in your context so your work can overall reflect that.
• llamaLord 4y ago
  As a PM, can I just say that your PM is fucking useless! Your PM is meant to be commercially minded, meaning they should understand the concept of compound interest. Every new feature you build, on average, increases the value of your product linearly. Every major piece of tech debt incurs reduced efficiency that compounds. It is extremely common in older systems for the incremental value of resolving tech-debt to be HIGHER than the incremental value of shipping "another feature".
  • jshen 4y ago
    “Every new feature you build, on average, increases the value of your product linearly.” Why do you believe this? It doesn’t match my experience and I don’t think it’s a good way to think about it.
    • ZephyrBlu 4y ago
      Can you provide an explanation as to why it doesn't match your experience? There might very occasionally be a "killer feature", but that seems far more uncommon than features incrementally increasing the value of the product.
      • jshen 4y ago
        For starters, a lot of features add negative value. For most things, people don’t want everything and the kitchen sync thrown it. It confuses them and makes the product hard to use. If your model were true, any dummy off the street could achieve linear growth in a product by just continually adding more features. That’s not how it is. Second is what you hit on. Some features add exponential or greater value. This means that the value of new features does not fit a bell curve and reasoning about it with averages is a mistake.
    • someweirdperson 4y ago
      Where there are dependencies between features, effort increases more than linearly (even more than polynomial). If usefulness increases only linearly, that puts an upper bound on the meaningful size of an implementation.
• echohack5 4y ago
  Having been in this space for 25 years now, here's my take: Tech Debt is almost always a few things in disguise: 1. Product Failure - The person with the final say on the product has poor taste. They poorly understand the tradeoffs their tech stack gives and how it influences their product. And most importantly they hate their customers and don't understand their needs. 2. Marketing and Sales dominance - The product organization might be competent but they have been driven out of the decision making venues in the company. So marketers and sales make the decisions and hand them down to product to deal with, resulting in tone deft decisions even as the company continues to make good money, the product itself erodes, until competitors arrive and instantly wipe out the business. 3. success and shift - The product, which was initially a success, has stagnated. The product itself was so successful that it created an entire new market with competitors that resulted in commodifying the product. Now the company is frantically looking for a pivot to keep growing the business, resulting in the original successful product drying up even faster. 4. Leadership void - The product was made by a strong, pioneering leader. They might not even consider themselves such a leader, but after they move on, the product fails without their support. The replacement leader might not even be bad at running the business for a time, but eventually focusing on EBITA alone won't inspire people, and the product will erode through churn. 5. Press Release driven development - The company operates by making a moat around its original core offering and they have a semi-monopoly in their space. So the only way to drive more revenue is to sell services that cost more to already existing customers. As a result, teams build products that are made to drive hype cycles and press releases -- once the product is shipped, the major players get promotions and move on to the next exciting product instead of supporting the now-shipped product.
  • Yabood 4y ago
    All valid points, but let's not forget that development teams are a major contributor too.
    • sublinear 4y ago
      I think this is highly underrated. The ideal is when a team's experience actually matches the project they're assigned. They need to be up to the task. This requires management to be at least more experienced than the teams they manage and to make good hiring and placement decisions. This is not just number of years in the industry, but the average number of years spent at any single company. They need to have seen the long tail of maintenance in the development lifecycle on projects they were wholly responsible for.
• deleted 4y ago
  [deleted]
• Scubabear68 4y ago
  I would argue that if they were able to pay down the tech debt that quickly and easily, then most likely the debt wasn’t as bad as described to begin with.
• hooande 4y ago
  > "You see, the leadership did not care about the code quality as long as the stories were delivered on time." My problem with the concept of technical debt is that "code quality" is subjective, and more often than not it translates to "I don't like the structure of this code because I didn't write it". What matters to the business is that the code works. The owners of the business do not care if the developers enjoy working with the code or if they find it to be well written. I hear all the time that bad code slows down development, increases bugs, etc. And that can be true in a number of cases. But most of the time it's just complaining. If a developer takes the time to re-factor the code base, it's very likely that whoever takes over from them in a few years will have the same complaints about technical debt and want to re-factor all over again. The best way to ensure everyone's job and increase compensation is to deliver features that customers really want. Doesn't matter what the code looks like, as long as the customers are happy. Re-factoring is important to developers and to maintaining an organization. But it can wait until there aren't pressing feature demands.
  • mistrial9 4y ago
    no mention of language(s), product(s) complexity, age of code, generic code hygiene topics..
  • mkoubaa 4y ago
    There's always the risk that the people tasked with paying down the technical debt simply drive the code into a different corner. Excellent technical leadership is extremely rare and can make a huge difference.
  • posix86 4y ago
    Agreed until the very last sentence. Refactoring should never happen for its own sake. You should never make code beautiful because it's beautiful. It should be nice to the extent, as you correctly said, that it facilitates creating value, in form of a running program, not as source code. This means that refactoring is worth it exactly if the total time spent to refactor the code is less than the time you save implementing future features after the refactoring. I'm at the start of my career as a software engineer but it seems to me a good rule for refactoring is that it should only ever be done if the advantage can be evaluated in the form of concrete new features being easier to implement. If you refactor the code so that existing features would've felt nicer to implement, or are nicer to read, you've already played yourself, because maybe you get a high of ecstasy because the code that was ugly before is now clean, but as soon as the next feature comes around that you didn't anticipate, it was all for naught, and you're writing ugly code again.
• hashmap 4y ago
  How often do products go under because of unresolved tech debt? Is it many? When I first got into software I spent a lot of time on refactoring and cleaning up code. As time goes on I find myself doing that less and less, and just focusing on work that produces meaningful results. Fixing tech debt, from what I can tell, is less meaningful than conventional wisdom says it is. No software will be perfect. Eventually it will die and something will replace it. I think knowing what "good enough" is, is perhaps the more important capability. Larger fruit hangs lower than tech debt the vast majority of the time.
  • jahewson 4y ago
    When a software startup fails it’s rare that we ever get to find out the real reason for it. The straw that broke the camels back gets the blame - the recession, some hackers, Google, whatever. Nobody goes to the newspaper saying hey, we neglected every possible aspect of care and maintenance over the years, put a bunch of interns on fixing issues using spreadsheets and raw SQL and bled our customers dry over a decade until they all left. Usually the bigco that got suckered into acquiring them gets the blame and the founders are on a yacht in the Caribbean. Tech debt won’t kill a company but it will strip away its immune system leaving it vulnerable to the slightest hiccup. Then the deadly hiccup gets the blame.
    • Nomentatus 4y ago
      I've known a case way back where deliberate tech debt of a sort was built up by not switching to a more modern compiler for a prototype. The idea was to then do a hopefully-quick rewrite on a more modern setup to obtain the minimal viable product (not a term at that time.) But Microsoft (who I think probably got wind of this product) announced a very similar product (that never appeared as such, and did not appear at all for at least a decade. The problem was harder than it looked.) Our financial backers immediately and fully pulled out, verbal contracts notwithstanding. With no more cash, the company was dead. A few years later others created just part of this product, and raked in very nice profits for a long time. If the same prototype had been built on the more modern compiler, that would have been the minimum viable product. The more modern compiler was one able to use more memory as the 286 cpu allowed.
• snissn 4y ago
  This strikes me as very smart - The first rule is not to create tech debt in the first place. The PR (Pull Request) that creates tech debt should come paired with the issue to deal with it.
• theptip 4y ago
  I like this approach, and have found it useful. One other trick I used, is to have a “tech debt week” at the end of every quarter. 12 weeks of coding per month, then a week for the managers/PMs to evaluate the last Q, and plan the next one. At the same time, while there is this awkward 1wk window where the plan is in flux, the engineering teams can focus on polishing their tools and attacking tech debt that might take more than a day every sprint to make progress on. Of course, this planning window is probably too short a window for big companies, and too much planning overhead for one-team startups. But for 20-30 person engineering org this cadence worked well.
  • kevsim 4y ago
    This is somewhat similar to the Shape Up methodology from Basecamp [0] where there's a 2 week "cool down" at the end of every development cycle. So 6 weeks of feature development, followed by 2 weeks of bug fixing, refactoring, etc. I've never done it in practice so I don't know how well it really works and there are some other parts of Shape Up that I disagree with strongly. 0: https://basecamp.com/shapeup https://basecamp.com/shapeup
    • theptip 4y ago
      If you can afford that level of slack, I think you will build a very happy, long-term-focused org. Worth noting that they build and open-source frameworks like Rails, Hotwire; if you want to polish and share, you need extra bandwidth vs. just building internal-facing products.
  • theptip 4y ago
    ^ should of course be “12 weeks of coding per Q”
• xyzelement 4y ago
  The "X% for work engineers want to do" approach always strikes me as a poor substitute for communication and sober evaluation of the value of work. If I am a PM, and my team's velocity is badly bogged down by some tech debt, then the right allocation to fixing it is 100% (fix the shit so we can go fast forever.) On the other hand, if some "tech debt" doesn't actually impact team velocity/clients (eg, some code is "bad" but it's in a part of the system that's never touched) the right allocation is 0%. There's ultimately the only thing that matters - getting value out to the customers. Tech debt only matters to the extent that it gets in the way of that, so prioritizing it vs features is easy because at the end it's still about "what do the clients get, when" The X% approach seems to happen when engineering and product fail to have that conversation, fail to understand each other, so they have to just get a flat allocation each. Suboptimal.
  • Infinitesimus 4y ago
    Thanks for sharing your perspective. Some fun sprinkles of complications > and my team's velocity is badly bogged down by some tech debt, then the right allocation to fixing it is 100% (fix the shit so we can go fast forever.) That's impossible. Software, like all things in nature, evolves and degrades over time and there is unlikely a magic 100% fix. Something can always be improved. > On the other hand, if some "tech debt" doesn't actually impact team velocity/clients (eg, some code is "bad" but it's in a part of the system that's never touched) the right allocation is 0%. Tech debt is also a great way to get your executing development team down to 0%. Yes, some people leave organizations because the code quality is bad and no one cares to improve it and it makes everything just a tad worse. This can compound over time. One day, the thing that is never touched will need to be touched and no one will be able to understand it in a timely manner, leading to other failures and re-prioritization. > The X% approach seems to happen when engineering and product fail to have that conversation, fail to understand each other, so they have to just get a flat allocation each. Agreed, a constant conversation is a good starting point but eventually, there will always be a bias in one direction. As you demonstrated, your bias is "There's ultimately the only thing that matters - getting value out to the customers". This makes you more likely over time to prioritize shipping to the customer over invisible improvements. The feedback loop continues until things breakdown. If you're in doubt, go ask your EM/TL partners if there isn't some technical debt they think is more important to address in the next 3 months than whatever you're currently trying to ship. Technology organizations are quite complex systems but i appreciate your attempt at simplifying.
• logicallee 4y ago
  Hey I have a question for everyone here. Is it possible to go camping without being obligated to build a house at the same site?
  • jbuhbjlnjbn 4y ago
    Not if you are going to rent your tent as a house afterwards.
    • logicallee 4y ago
      Got it: when you accept money for letting someone sleep in your tent you are $1 million in debt since you are now obligated to purchase that plot of land (which may not even be for sale at any price), find a licensed architect, design a house, get building permits, find contractors and build it. Plus you are infinite amount in debt since it takes infinite money to build a time machine so that your initial customer can sleep in appropriate lodgings retroactively. Often code with technical debt has paying customers, and going back in time to get them what you imagine they should eventually get is part of the financial equation, right? Thus technical debt always includes building a time machine so that your initial customers can use the product they should have gotten from the backend they should have gotten it from, back when they should have gotten it. In other news, the world's richest person is actually the world's poorest: Elon Musk built a tent in people's minds and now he has to deliver on it. His technical debt is a hotel on Mars. Since technical debt is real, that makes him the world's poorest person by a huge margin. Also, since technical debt is real you can deduct it from your taxes just like any other debt, right? I mean we wouldn't want the IRS to be unaware that you owe more than 100% of the time and money you invested in building a solution so far, since it could lead them to want to tax you based on what you earned, which just plain isn't fair. They should only tax any remaining money after you have engineered a veritable Taj Mahal of code quality such that there isn't any remaining conceivable improvement. After all, you know what another word is for conceivable improvement: technical debt.
• rubyist5eva 4y ago
  You don’t have dedicated time for technical debt. You make fixing technical debt part of your estimates for new features.
• bob1029 4y ago
  I think using financial terminology to describe technology problems is a mistake. There is no such thing as tech "debt". Or, if there is, it's entirely subjective. I've seen things described as a "6+ month major rewrite" that another developer could address with careful, incremental enhancements over a matter of weeks.
• carl_sandland 4y ago
  There's no such thing as 'technical debt', just call it crap code (or a 'hack') and face the impacts of having to live with it. Its been my experience that these things get logged to Jira as 'medium' or 'low' and then of course never get looked at again, and if they ever get opened up its likely the code has moved on. This leads to the following idea; 1. code the MVP the customer accepts (customer is happy) 2. go ahead and create all the debt tickets (makes you feel professional) 3. every xmas just delete all non-high tickets older than 12 months. If the code is shit enough, it will die a natural death (most of today's wiring type code has very short TTL anyways) Similar to "if a tree falls in the forest...", "If a customer doesn't notice...". We do so many 'invisible', 'hard' things in our line of work, it's almost an impossibly thankless road to build quality into our systems.
• gerdesj 4y ago
  Tech debt is simply another debt that a company or org needs to deal with. It's far cooler for a tech company to worry about their funky no vowels thing than say the rent. My boring little IT company in the UK owns its premises - roughly an acre in a town in Somerset with a 19m long edge red brick two floor building. The property costed us about £240,000 - the mortgage is cheaper than the rent we paid on part of a converted stables really out in the Styx. Now with a mortgage there is the fine print. In the UK it is normal for a bank to require a "debenture". Your Loan to Value is below a percentage then the bank can intervene at any point and take over, which is what at least one bank in the UK did to try and shore up their finances when it all went south in 2007ish. There is another thing called an "Overage Clause". That's where the vendor wants to make a hold over future profits. So we bought our place from the NHS and they sought an overage in perpetuity (for ever) on any profits we might make on selling the place. We negotiated on 10 years. That expires this year. With it, we would have to hand over 50% of any profits from sale. Running a company is not rocket science but it can be tricky. My little firm will never be a unicorn or even a mare with a cornet on its nose. I don't care.
• r3trohack3r 4y ago
  I really wish we would stop calling it technical debt. Every team/org I’ve worked in with “tech debt” issues has had very tactical problems that could have been communicated, invested in, and solved. But instead the org talked about “tech debt” - an immeasurable boogeyman that anyone outside of the engineering org has no grasp of and, most importantly, management up the chain to the C-Suite have decreasing mental models of investment/pay-off the further up you get. Teams saying “tech debt” are perpetually under funded and under appreciated. Instead, speak a language your management chain understands. * These specific services have outgrown their architecture and back pressure keeps outgrowing their current scale, we need to invest in a more reactive architecture. It’s going to cost 3 teams 1Q and we will prevent N outages based on historical data. * In 2023, engineers far fingered the deployment of these services N times causing various levels of service outages, one made the news, we need to invest in guardrails in our CI/CD to prevent that. It’ll cost one team 2Q and we will prevent N outages. * We had 4 employees across our engineering org quit last quarter because holding the pager burned them out, we need to stand up a tiger team that can help kick our metrics into shape. Speak a language your management understands. Speak in terms of delivering features (feature velocity), reliability (outages), employee retention, hiring through resume driven development, etc. You’ll find you’re negotiating in a positive sum game if you do this. You give me 1 unit of investment for this problem this quarter and I’ll give you 1.3 units of return next quarter. And maybe there are greater returns elsewhere so you aren’t making a competitive bid and that is okay, or maybe your management will invest in you and you just signed yourself up to deliver 1.3 units. But don’t handwave and ask for budget.
  • 0xbadcafebee 4y ago
    > Speak a language your management understands Yes! And if they don't care about velocity and reliability, don't tell them that. If they've been going on about hiring and employee retention, tell them how this tech debt thing is going to have such a huge change that you can turn it into a kickass conference talk and hire more 10x programmers, which is a bigger value to them than "I made the app more maintainable". They don't care much what your team does; just give them something they want to buy.
  • sdiupIGPWEfh 4y ago
• fuzzieozzie 4y ago
  I have communicated tech debt very effectively as cleaning a house - "If you do not regularly clean up you're going to end up in a world of hurt!" Warning: your house might end up looking like a hoarder's house!
• deleted 4y ago
  [deleted]
• namuol 4y ago
  Dedicated budgets for tech debt only serve to justify decisions to cut corners in the first place. It’s not a prioritization issue or a matter of smarter sequencing. It’s a cultural problem.
  • xtracto 4y ago
    Right on. Shitty developers are going to make shitty software. Every City Mayor wants to build their bridge in 1 week. However, you don't see Civil Engineers "cutting corners" and leaving "debt" when designing and building those bridges. There's a minimum quality that they will sign off. No matter how loud their bosses shout. When I lead Software Dev teams, I always told developers not to compromise certain quality factors. If someone asked them "can it be done faster" and they answered "yes, but X thing would suffer" they are giving a door to for sloppiness, and the Business side doesn't have a way to understand the implications of X. They only hear "yes". On the flip side, if an developer says "it will take 2 weeks", there's nothing else non-technical people can do but sit down and wait. The only ones that can really challenge that are other technical people, and hopefully if Business "escalates" the developer to his manager's manager (up to the CTO), she will be backed up by the technical higher ups, if she makes sense. I would sit down with people trying to shorten development time of some things. There are times where it is sensible to get some debt, but it is a decision that shouldn't be done by Jr or even Mid level engineers.
    • namuol 4y ago
      Agree with everything else you said but take some issue with this: > Shitty developers are going to make shitty software. That’s really not the point I was trying to make. It’s more of a trap to blame ICs on the ground in most situations when they’re usually only following existing practices. All this will do is demoralize your engineers. Mismanagement in the form of poor staffing choices (especially moving tenured and experienced engineers to the next shiny thing) is often the culprit. This usually stemming from the pressure from leadership to ship stuff fast, as usual. Once you sacrifice quality of any kind, that bar gets lowered for good, and before you know it there are so many examples of poor quality that any issues get lost in the noise.
• hendry 4y ago
  People talk about tackling tech debt, though I want to see results in the form of reduced/stable lines of code! The author mentions 180k of code... but did the team actually chip away at that SLOC thanks to "Tech debt Friday"? https://github.com/kaihendry/graphsloc https://github.com/kaihendry/graphsloc is how I track projects.
• revskill 4y ago
  The only reason i raised issues about tech debt in previous companies is because i don't want to inherit from a messy codebase created from the formers. Even with a good test suite, you can go very far. Clean test (interface) is more important than clean code (implementation)
• trevorLane 4y ago
  teams should push for a 70/30 feature to tech-debt ratio. product is so good at feeding the dev machine, devs rarely get time to evaluate the landscape and push back.
• tanin 4y ago
  Sometimes I feel I'm the only person who thinks tech debt is not that difficult to solve. Everything needed to be solved is under your control. It may take time and be boring, but you have everything you need to solve it. Product-market-fit, customer acquisition, and etc. are often much more urgent/difficult to solve, and we should focus on those first. What I've seen in a lot of teams is tech debt (or speculative tech debt if we move quickly) is exaggerated to the point that they cannot launch quickly to acquire and iterate with customers, which is a huge mistake in product development.
  • intelVISA 4y ago
    I'm fond of: v1 - shitshow for product market fit, deliberately debt heavy for speed/iteration v2 - from the ashes, this is what v1 should've been, emphasis on the long-term You've already done the hardest bit (imo) by figuring out the kludge blueprint that is v1 it's reasonably easy to build v2 with the lessons learned from v1 fresh in mind. Of course, v1 is the default model in most shops and there is no v2.
    • tanin 4y ago
      That is a good way of development. In a lot of the times, there is no V2 due to 1 of the 2 states: 1. V1 is too successful. You cannot slow down the growth. It is bad, but not as bad as people make it out to be. Many startups are trying very hard to be in this state. 2. V1 fails, so there is no point for V2. In a large company, there is an additional state: long term ownership is hard. By the time, we should implement V2. The original team is already promoted for PMF and move on to a new shiny project. The new team would just complain tirelessly about the tech debt because obviously they aren't getting the reward and are stuck with a shit job. Now nobody would advocate for this way again. Not sure why my argument now turns the opposite direction.
      • intelVISA 4y ago
        There wouldn't be tech debt for v2 is the point: it'd be from scratch code-wise whilst applying the design/architecture lessons learned. A lot of tech decisions are only apparent after the project is built - revealing them with a rapid protoype and then scrapping it to build "what should have been". Ultimately, as you say, it is pointless as there is little business incentive to work like this as most customers would gladly accept the crappy prototype as final.
• kitanata 4y ago
  10% is under investing and this is a bad methodology. How do you know what to work on? Is tech debt just the stuff that annoys you or is it actually bad? What you need to tackle tech debt is a quality program with metrics. I’m talking code coverage, cyclomatic complexity checking, linters and scanners, DORA, SAST, DAST, etc. Quantify your quality. Then quantify the risks and costs of not improving it. Then you need to target the areas of code your tools tell you to and you need to make a conscious effort to solve those very specific things. “Module A’s complexity score is 26. Our standards say this needs to be 10. Therefore this is considered a quality item. Therefore it goes into the sprint as a strategic investment.” Software and business leaders, when developers talk about tech debt… they are talking about managing complexity. (Shit breaking all the time because you don’t have tests in a complex system. It’s failing because of the complexity.) High complexity is expensive. If you do not balance the need to manage complexity against features and you do not act intentionally about your quality your software will eventually fail or if you’re lucky it will just reach a stage where you can’t maintain it anymore and you’ll scrap and rebuild. Investing in quality makes you go zoom.
• romanhn 4y ago
  This was a failure of product management and, doubly so, engineering management. A PM working in software absolutely needs to understand the concept of technical debt. Not necessarily to know where the debt is on their own, but to work closely with engineers on understanding and managing it along with product functionality. That said, I reserve most of my scorn for the eng manager. This is the person whose duty it is to say that yes, in fact 20% is the right proportion for tech debt work after it's been neglected for so long. This is the person to stand up to leadership and explain why this is necessary. Autonomous teams with bottom-up decision making are much more likely to push towards the right thing even with clueless management, whereas there is little hope when incompetent decisions are fed to the team in a top-down manner (yes, autonomous teams indeed exist - though they are typically a sign of competent management).
• ochronus 4y ago
  On a related note, I've written about how to tackle tech debt - dedicated capacity/time is just one approach. https://leadership.garden/tips-on-prioritizing-tech-debt/ https://leadership.garden/tips-on-prioritizing-tech-debt/
• pictur 4y ago
  I think the concept of technical debt is a kind of corporate culture. it can be repaid over time, but it's likely to reappear unless you reduce the things that caused it. that is, of course, it cannot be completely eliminated. but for example, things like serially redundant feature development can be reduced.
• hbrn 4y ago
  Tech Debt Friday worked for this team, good for them. It will probably not work for another team. Tech debt is never ending discussion. But ask yourself: why is almost every team struggles figuring out how much time to dedicate to tech debt? Why is there a constant tension between engineering and management? The answer is simple: engineers and business people don't understand each other. We live in different worlds and speak different languages. You can't solve this problem with a methodology. If Tech Debt Friday or Google's 20% works for you, that's just luck (or wishful thinking). Once you understand where the problem is coming from, the solution is also simple: find someone who speaks both languages and trust them to decide. Typically that person is a product engineer: it's easier to explain business to an engineer than engineering to a business person. And when I say trust, I mean both engineering and management should do it. I.e. if that person says a couple of outages is not a big deal, it's not a big deal. If that person says we need to spend next week refactoring, then that's what you spend your next week on. Obviously you can still challenge those decisions, but you have to accept that this person is an expert in their domain. They don't know everything, but they know more than you. Even if you're the CEO. Even if you're coming from FAANG with 20 years of experience. That's why companies where founder is a product engineer typically don't have a tech debt problem (they still have tech debt, they just don't have a problem with it). I know it sucks to not have a mathematical or managerial solution to tech debt, but the tech debt is inherently complex and humane. Tech debt happens when humans are solving problems for other humans. The only solution to it is to have another human in between. The quality of that solution will depend on the quality of that human. It's not going to be perfect. But it's the best you can do. All other approaches are just shots in the dark.
  • sibeliuss 4y ago
    > And when I say trust, I mean both engineering and management should do it. I.e. if that person says a couple of outages is not a big deal, it's not a big deal. If that person says we need to spend next week refactoring, then that's what you spend your next week on. Obviously you can still challenge those decisions, but you have to accept that this person is an expert in their domain. This is certainly the answer to this problem!
  • Nomentatus 4y ago
• goto11 4y ago
  The only solution I have seen work: Paying back text debt should be an integrated part of regular development rather than a separate task. When you change or add a new feature in an area of the code, you clean up tech debt in the area, at least to the level where it is better than it was before. This also avoid "moving around deck-chairs"-refactoring since the refactoring are coupled to specific development tasks. You refactor to make implementing the task easier and cleaner - no more, no less. If some area of code is "ugly" but works and don't need any functional changes, leave it be. Scheduling tech-dept-payment separately runs the risk of getting de-prioritized. If a deadline is approaching or the company need to cut expenses, I'm sure the dedicated "tech-debt-payback" time is the first to get cut, "temporarily".
• andrewfong 4y ago
  The interesting aspect IMHO is not that they set aside 10% of their time to fix tech debt per se, but that they set aside 10% of their time to collaborate on it / engage in mob programming.
• adityaathalye 4y ago
  Simple debt is a poor metaphor. Jenga tower is a better one. An even better one is "Jenga tower of subprime debt" [1]. Xkcd nails it as usual: https://xkcd.com/2347/ https://xkcd.com/2347/ The author writes: > "To their credit, I came in when the code was like a crumbling Jenga tower" Structurally bad systems look a lot like that. Small misalignments and nonlinearities compound to make the structure vulnerable to a sneeze. Relatedly, I feel it is not so much what % was applied, but what targets were de-risked. Things that tend to matter include mean time to recovery, time to market from _planning_ to deployment (not just commit to deployment), defect rate (e.g. hotfixes and rollbacks). If the superstructure is good, one can start with cheap panel walls and gradually swap them out for nicer ones with better properties. [1] cf. a post I wrote to ruminate that angle: https://www.evalapply.org/posts/software-debt/index.html#main https://www.evalapply.org/posts/software-debt/index.html#mai... edit: add source
• glintik 4y ago
  Why not 15%? Or may be 5% will be enough. Actually, it’s a bad idea to plan time in such manner. If devs need, they should be able to refactor/fix things during normal development process without any “just 10%”.
• SideburnsOfDoom 4y ago
  > PM argued: So, you are telling me that we must spend 20% of our time just to keep the lights on? Yes. If the "keeping the lights on" work has been neglected for long enough then yes, in fact 20% is a lowball number. The context is not so much "keeping the software running" but "reducing the friction in further changes to, and deployments of that software"
• vitriol83 4y ago
  In places where there is not much time for code refactoring, the following is helpful: Imagine an idealised future state of the codebase, which everyone buys into, and make sure any new feature is going in that direction. Refactoring existing code can be death by a thousand cuts- having a parallel new codebase which is incrementally adopted can be more efficient and quicker to market.