43 ms·

How to approach and prioritize technical debt

94 points by ochronus 5y ago

wellpast 5y ago
> I've seen some managers (engineering and product alike) say phrases like, "well, if you created a better solution in the first place, we wouldn't be in this situation" – this is just stupid, inhumane, and ultimately pointless This is such a disappointing aspect of the 'technical debt' concept and surrounding discussion. The unspoken truth is that technical debt is most often a euphemism for inexperience and lack of skill. This may be an uncomfortable truth but it doesn't have to be inhumane. It's not inhumane to identify a lack of skill and then address it. The problem with 'technical debt' you'll won't find a concrete definition, and you won't see a consistent usage across the board. It is far more used as a way for engineers to posture as if "we meant to build it that way", we were making an explicit cost-benefit tradeoff... spend more time or get it out quickly. Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued. The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set. But this incorrect. How much upstream scope/features delivered is certainly a function of time. Feature/Capability X, Y, Z will take more time than X and Y alone. But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set. Many will balk at this, I realize. Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'. 'Technical debt' hurts everyone because it doesn't admit to any of this, and shifts the buck away from any personal skill issue. So, too frequently the engineer goes back to fix the technical debt and not their own skill set. That's why there's always a perpetuating game of technical debt over years until the great 7 year rewrite.
- ochronus 5y ago
  I wish this was true, life would be so much easier :)
- nerdponx 5y ago
  > It is far more used as a way for engineers to posture as if "we meant to build it that way", we were making an explicit cost-benefit tradeoff... spend more time or get it out quickly. > Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued. I don't see how the second statement follows from the first. Are you saying that good engineers always have enough time to implement every feature "the right way" on the first try? > But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set. Of course it is a function of immediate time available today. > Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'. This sounds like it's targeted at one specific developer you used to work with (I have worked with developers like this too, it sucks)... but that has nothing to do with time/quality tradeoffs in design and development. > The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set. This is like Calvinism for programmers. Yes, some programmers are better at software architecture and design than others. Yes, some programmers leave behind a mess which, if not fixed, does truly turn into "legacy code" and "tech debt". But it's ridiculous to assert that all technical debt follows this pattern. Design is often iterative; people are not omniscient. Bad engineers fail to evaluate the tradeoffs correctly, or aren't aware of all their options, and make bad decisions. Good engineers understand the tradeoffs and know their options, and make good decisions. Let me give an example from my actual job: I was tasked with implementing a sort of graph lookup system. I had two options: pre-compute a large portion of the graph, leading to a nice big flat table that could be easily queried at runtime with a single join, or leave the entire thing in a normalized format and do a big multi-join lookup at runtime. I chose the second option, because this allowed me develop a system and codebase that was simple and easy to understand, offloading the complicated work to the database. Moreover, it allowed me to avoid the various operational issues and technical details around setting up and maintaining the intermediate pre-computed table. The performance requirements were such that the increase in latency and server load wasn't important for the first version of the product. The business was and remains happy with the product. Now that it's humming along and users are starting to put more load on it, migrating to the first option is now on our engineering calendar for the year. With ~12 months of accumulated knowledge about this system and its use cases, with an increasingly stable API, and with an increasingly rigorous & thorough test suite, I feel confident that swapping out the internals (including the database layout) is a relatively low-risk operation at this point. The internals have already been subject to several incremental refactorings simplifications as the business stabilized the product requirements; these were implemented without any visible effect on users, and with minimal changes to other "internal consumers" who depended on the API. If I had tried to implement things that way from day 1, I would have had a system that was more complicated, more difficult to maintain, more difficult to explain to other programmers, and had more operational elements that could break. But I did indeed take a loan of "tech debt" in the form of worse performance and high server load, which we will have to pay off eventually. Perhaps if I was a better programmer, I would have had the skills to implement the first option right, on the first try. But I don't know many (any?) programmers who do have those kinds of skills, while also having the wherewithal to document the thing, write useful and nontrivial test assertions, develop tooling for OpenAPI conformance, et alia. Not to mention that the flexibility granted by simplicity actually has turned out to be very important for being able to respond to changes in product requests quickly. It seems (so far) that I did make the right decisions in this situation. I think that makes me at least a decent software engineer.
- sfteus 5y ago
  This is an insightful comment that holds true a good portion of the time. However, there is one takeaway that is overlooked in my experience: resolving tech debt that exists due to lack of experience tends to build said experience, _provided_ the less-experienced developers are involved in understanding the underlying issue and planning a solution.
- sirwhinesalot 5y ago
  If you've worked with an engineer that "intended" to accrue technical debt, I feel sorry for you but I hope you realize how uncommon that is. Technical debt is the answer to management requests that cannot be accomplished in the requested time with the available resources. No more, no less. The word debt refers to the fact that the shortcut needed to do so will cause problems repeatedly in the future (interest that will need to be paid). In my experience, engineers will make this clear to management, and will point out the additional time needed to implement a solution that avoids those problems, and management will make the decision to continue as is. For a concrete example, management requests that you implement real-time collaborative editing in an existing tool. You have enough time to replace all the backend stuff with a CRDT or OT based implementation, but the frontend architecture is not equipped to deal with collaborative editing and there is no time to do both with the people available. So, in order to meet the management deadline, quick hacks are added to bolt on the collaborative UI on top of the existing UI. Eventually new features have to be added that simply cannot be supported in the existing UI architecture without massive workarounds. Carving out some time to slowly refactor the UI architecture is rarely done, so the problem accumulates until development speed slows to a crawl. That is technical debt, and it's entirely a byproduct of management/development negotiations. Has nothing to do with engineers. Nobody could have predicted that the architecture would need to support an entirely different backend in the future. EDIT: Just to be clear, technical debt happens as a result of a conscious tradeoff between one or more options, where the more problematic option is chosen to get to an "acceptable" solution in a shorter period or with less resources. Some dev writing a shitty architecture is not technical debt. Depending on a library that later gets EOLed is not technical debt. Figuring out you have a shitty architecture or an EOL library and choosing to continue building on it despite clear evidence it is a bad idea, in service of shorter time to market, that is technical debt.
- TameAntelope 5y ago
  This is true sometimes, but a few things happen as you keep writing software: 1. By writing the needed code the "debt incurring" way, you learn how to do it the "right" way. So you may start with inexperience, but you gain the experience and are now better able to fix the issues you introduced. 2. Similar to this, even if you have all of the necessary experience and skill, you still may not be fully able to picture how the feature will need to evolve in the future to support additional use cases. Once you see that happening, you can design a better implementation, but not before then. Sometimes you have to do it wrong to get the feedback you need to do it right. 3. There are often real tradeoffs that get made that aren't about skill, but about time. I don't think you can discount this situation, in my experience it happens very, very often. A great way to avoid nearly all of this is to just set "EOL criteria" for your software. "This project will not work beyond 100 users." or, "This feature will need a rewrite once we have to account for the edge case we ignored in this implementation." Know the limits of what you've built, and make it clear that you need to rebuild once you've reached those limits, and it's at least marginally more likely you'll get the support you need from the people with the purse strings.
louwrentius 5y ago
The best way to deal with tech debt is to never use the term in the first place. Almost nobody really uses tech debt as a concept the way it was intended by Ward Cunningham. It seems mostly to be abused to absolve oneself or others of responsibilities. I or others didn’t make a mistake, it was just “tech debt”. Sounds much more benign like this. Like tech debt is inevitable, unavoidable and current issues could not have been prevented. Yeah, sure. When I hear people talk about tech debt, my bullshit meter is immediately on high sensitivity. I fully understand why people don’t want to expose their own mistakes or those of others. But I think it kills trust for those who see right through it. Having to talk about tech debt instead of the real, actual issues feels emotionally immature. I mean this in a kind way, but I mean it. It results in this whole cottage industry dealing with tech dept, in a similar vein as the Agile industry. I don’t think all this dressing up of rather clear cut issues as tech debt adds any value and only creates a layer of bullshit work. I think we need less bullshit and more environments where people can openly admit to and discuss mistakes without immediately being raked over the coals / attacked.
- michaelfeathers 5y ago
  A term I use to recenter on Ward Cunningham's concept is 'Design Lag' - How far is the code behind your current understanding of the problem?
  - louwrentius 5y ago
    That is something I can get behind. But the rest of the world has taken the concept of TD and warped it to mean ‘my fuckup with a more neutral label’. Check definitions when you talk about TD with someone.
- ess3 5y ago
  I agree with you. But I guess the problem stems from having to prioritize between building the next shiny thing or actually going back and fixing the problem. Even worse is that many of these problems have rather low short-term impact and a much larger long-term impact, which makes it even harder to justify/explain to the person making the prioritization. It's much easier to hide it behind the facade of "technical debt" to which everyone intuitively agrees it's a problem. But yes having people with the competence to actually understand the clear cut issues also making the prioritization would be nice.
lbriner 5y ago
It is true that the intention of tech debt is supposed to be when you make a conscious choice to take a shortcut now knowing that you will pay interest later and that other people use it to mean anything suboptimal whether due to lack of skill or changing requirements but in many ways the OP article stands for all of it anyway. Defects, tech debt, refactors, whatever you want to call them need to be measured against all other work in the backlog and given as objective cost and risk as possible. Engineers tend to like neat and tidy but there is terrible code in our system that has simply never been changed and still works so we leave it alone.
- gnat 5y ago
  From what I've seen, there are three big schools with technical debt (past choices that now make changes difficult): 1) Ignore it and gradually ossify as the code congeals and it becomes harder and harder to make any change at all. 2) Assign a fixed budget of time, e.g. 20%, to refactoring and bugfixing and updating models and "moving to the new version of the language/library which lets us ...". 3) Prioritise the engineering tidy-ups alongside features. This involves (a) being able to credibly estimate the effort and value in the same dimensions as feature work is estimated and prioritised; (b) the people who do the prioritisation believing that engineering-desired work is as necessary as sales-desired or PM-desired work. That last route requires people's beliefs and abilities to align. The middle route bypasses the entire prioritisation conflict at the cost of sometimes working too much or too little on technical debt. I think that's why you see a lot of fixed budgets for internal work.
suncore 5y ago
Here's a few observations (after long time experience and involvement in research around technical debt): 1) It is impossible to avoid gathering technical debt. The code will deteriorate in one way or another. You need to prepare to fix it since you can't avoid it. 2) It is so extremely difficult to make a correct "risk assessment" on technical debt so you should avoid doing so at all. You will just end up arguing all day on the merits of clean code/architecture vs feature growth. Instead, keep two backlogs and reserve a set amount of resources on each, e.g. 20% on reducing technical debt and 80% for product features and other development. 3) The "cost" of reducing technical debt is actually negative. That's the whole point of working with it, to increase development velocity.
- crdrost 5y ago
  Code doesn't deteriorate... Like we're talking about a banana growing spots or whatever... Code gets plastered over with features and abstraction layers, but that's an active process that we are complicit in doing. More germane for this discussion, technical debt was originally defined as a positive thing that you want to go get... It is the mismatch between our domain model and how our users think about the domain. It is positive because “enough with all of the planning and interviewing and requirements gathering and careful architecting, can I just build something and have my users criticize it and do four or five drafts until I get it right?” ... The debt is the drafts before you get it right, the interest is the constant translation between the language of your users and the language that the system is expressed in. You have a “contracts” table that contains things that are not contracts because every purchase foreign keys to a contract, but your users have since wanted to know what they do with the purchases that are not associated with any contract, where do they go. And now every query that aggregates over contracts needs to exclude the non-contract contracts, this is part of the interest you are paying. But at some point it started to mean that we had to upgrade our dependencies, that's tech debt, or this kludge that I threw in, that's tech debt, or the fact that we never worked out a shared library between the front end and the back end and so any Python code in utils.py needs to be manually kept up to date with a file utils.js in a separate Git repo so that we are sure that we can do these things both on the front end and the back end. More broadly, anything that we no longer care for is tech debt. And that's where you really get this idea that it is deteriorating, that's more a measure of our own patience deteriorating, especially as we never seem to have time for the refactors we want. Perversely the cause of not being able to refactor has nothing to do with tech debt and cannot be solved in this way. It's multifaceted so at different places it emerges in different ways, but usually it's an incentive problem. At some places it is that the only incentives are for feature development. At other places it is that every developer is working on a different thing rather than prioritizing one thing that the business needs and delivering it, and allowing developers to use their slack time in this equation to improve the product however they see fit... In yet another it is because the team lead resists any suggestion that the framework being used is too heavyweight for the problem being solved and so trying to keep these very clean abstraction layers is causing people to have to rewrite the same basic thing in five different unrelated places, because it has to bubble up from the data layer into the service layer into the business layer into the controller layer into the API into the consumer layer into the app state layer into the frontend model layer...
WallyFunk 5y ago
Would be interesting to apply game theory to tech debt, because it is, after all, The Great Game we all play, whether we admit it or not.
- ochronus 5y ago
  How would you imagine it? Sounds interesting indeed!
0xbadcafebee 5y ago
I think you can visualize tech debt by recording a chart where the indicator "debt" is $cost = ($num_tech_debt * $instance). $num_tech_debt is the number of different pieces of technical debt, and instance is a number incremented each time technical debt affects you. For every "instance" of technical debt affecting something, the cost goes up. If you aren't addressing tech debt, the chart will just go up and up and up. The only way to make it go down is to eliminate the number of technical debt things, which will reduce the cost equation (unless some other technical debt's instances go up). I suck at math, so maybe somebody else has a better way to represent that?
ochronus 5y ago
I love the discussions folks :) keep it up, I'm learning a lot <3
lowbloodsugar 5y ago
Debt would be if someone said "I need to build a bridge across here that can handle an Army, and I've got $1m", we engineers reply "It will take $2m", and they respond "Ok, I will borrow $1m so you can build the bridge I need". Instead what happens is they say "Well, build what you can for $1m", and you say "Ok, we can make 'a bridge' for that", and then either (a) your infantry can cross, but the tanks have to get diverted 20 miles out of the way, or (b) the tanks end up in the river along with the bridge. Since (b) is bad, you then have to spend a lot of time planning the routes for the tanks, and making sure the tanks have the right air cover, etc etc, i.e. doing more work. More likely, however, is that the manager, who is not a structural engineer, sees a perfectly good bridge and orders the tanks across anyway, causing the loss of the bridge, the tanks, and the war. It's not debt. It's just (at best) an incomplete solution or (at worst) a bad solution that fails at the worst possible moment - e.g. database collapses during registration for the largest event of the year. Ah, but surely, if you build the lightweight solution for $1m, and acknowledge the increased costs of managing the problems that it doesn't solve, then thats fine? Sure, but that's not technical debt either! That is scoping: we (engineers + business) identify a workable solution that provides some business value. And then we do that well. When Cunningham, for example, talks about what to do about debt, e.g. YAGNI, these are all good ideas: scoping. But the term "debt" is incorrect in this case. The term "debt" will only ever get a team in trouble as it indicates that the team is unwilling to confront the actual problem as engineering, but only as a broken metaphor.
- joshfee 5y ago
  What you describe is not what I've experienced with debt. Usually it is a conflict where short term goals and long term goals are at odds with each other. Using your bridge analogy, its more like "We only need an infantry bridge right now, but we know that in a year we need to support tanks". Well we can either spend $2m to build the tank bridge now, or we can spend $1m today for the infantry, but in a year you'll have to spend another $1.5m to make modifications to support tanks. The debt there is knowing that "we will spend $500k more doing it this way (interest) but we will benefit in the meantime". There are other factors too that make the true cost of the debt higher. For instance less flexibility as requirements change. I often find myself saying "this would be an easy change if we had initially built it in the more extensible way, but at the time you didn't need/want for extensibility so now your easy feature is actually a large refactor"
sam0x17 5y ago
One thing I've had a lot of success with as I help early to mid stage (usually YC) startups try to tune things is empowering engineering by giving bizdev and stakeholders the understanding and expectation that ~25% of eng's time should be spent on things like writing tests, refactoring, and otherwise reducing tech debt. Putting a focus on reducing tech debt up front via CI, automated testing (including both unit and integration tests), coding style guidelines, CI checks for all of the above, etc., and giving engineering the time to attend to these issues and practices has turned around many flailing or failing startups that I've consulted with and/or provided advice to. The ones that do fail are usually the ones that resist addressing tech debt in favor of shipping more features. In truth, it is usually quite possible to do both at the same time, and it is in the very least possible to tackle new features in a way that doesn't increase tech debt at the previous bad rate through updated automated testing and CI requirements that ensure new features are fully tested before they are merged. The hardest thing, especially with non-technical founders, is getting them to understand that yes, their precious features need to be pushed back a bit, but that this will prevent major issues, downtime, and slowdowns next quarter or next year, and that issues they are experiencing now are often due to neglecting tech debt in the past. Once they get this, the company usually has a much better outlook. This is also an investing criteria (what is your ongoing plan to reduce technical debt) for some of the better tech investors, I've heard.
- anamax 5y ago
  What's missing in the above is some notion that the "feature machines" should actually do some work to show what the feature will accomplish. Better yet, is that feature more profitable than some other feature? "you can have X in three months or Y in two weeks but if you pick either one, that delays Z by the same amount of time" focuses the attention.
  - sam0x17 5y ago
    Very, very true. Part of my process usually involves an ICE-based prioritization of the feature backlog and being very particular about what will be tackled in the short term.

94 points by ochronus 5y ago

> I've seen some managers (engineering and product alike) say phrases like, "well, if you created a better solution in the first place, we wouldn't be in this situation" – this is just stupid, inhumane, and ultimately pointless This is such a disappointing aspect of the 'technical debt' concept and surrounding discussion. The unspoken truth is that technical debt is most often a euphemism for inexperience and lack of skill. This may be an uncomfortable truth but it doesn't have to be inhumane. It's not inhumane to identify a lack of skill and then address it. The problem with 'technical debt' you'll won't find a concrete definition, and you won't see a consistent usage across the board. It is far more used as a way for engineers to posture as if "we meant to build it that way", we were making an explicit cost-benefit tradeoff... spend more time or get it out quickly. Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued. The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set. But this incorrect. How much upstream scope/features delivered is certainly a function of time. Feature/Capability X, Y, Z will take more time than X and Y alone. But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set. Many will balk at this, I realize. Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'. 'Technical debt' hurts everyone because it doesn't admit to any of this, and shifts the buck away from any personal skill issue. So, too frequently the engineer goes back to fix the technical debt and not their own skill set. That's why there's always a perpetuating game of technical debt over years until the great 7 year rewrite.

ochronus 5y ago
I wish this was true, life would be so much easier :)
nerdponx 5y ago
> It is far more used as a way for engineers to posture as if "we meant to build it that way", we were making an explicit cost-benefit tradeoff... spend more time or get it out quickly. > Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued. I don't see how the second statement follows from the first. Are you saying that good engineers always have enough time to implement every feature "the right way" on the first try? > But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set. Of course it is a function of immediate time available today. > Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'. This sounds like it's targeted at one specific developer you used to work with (I have worked with developers like this too, it sucks)... but that has nothing to do with time/quality tradeoffs in design and development. > The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set. This is like Calvinism for programmers. Yes, some programmers are better at software architecture and design than others. Yes, some programmers leave behind a mess which, if not fixed, does truly turn into "legacy code" and "tech debt". But it's ridiculous to assert that all technical debt follows this pattern. Design is often iterative; people are not omniscient. Bad engineers fail to evaluate the tradeoffs correctly, or aren't aware of all their options, and make bad decisions. Good engineers understand the tradeoffs and know their options, and make good decisions. Let me give an example from my actual job: I was tasked with implementing a sort of graph lookup system. I had two options: pre-compute a large portion of the graph, leading to a nice big flat table that could be easily queried at runtime with a single join, or leave the entire thing in a normalized format and do a big multi-join lookup at runtime. I chose the second option, because this allowed me develop a system and codebase that was simple and easy to understand, offloading the complicated work to the database. Moreover, it allowed me to avoid the various operational issues and technical details around setting up and maintaining the intermediate pre-computed table. The performance requirements were such that the increase in latency and server load wasn't important for the first version of the product. The business was and remains happy with the product. Now that it's humming along and users are starting to put more load on it, migrating to the first option is now on our engineering calendar for the year. With ~12 months of accumulated knowledge about this system and its use cases, with an increasingly stable API, and with an increasingly rigorous & thorough test suite, I feel confident that swapping out the internals (including the database layout) is a relatively low-risk operation at this point. The internals have already been subject to several incremental refactorings simplifications as the business stabilized the product requirements; these were implemented without any visible effect on users, and with minimal changes to other "internal consumers" who depended on the API. If I had tried to implement things that way from day 1, I would have had a system that was more complicated, more difficult to maintain, more difficult to explain to other programmers, and had more operational elements that could break. But I did indeed take a loan of "tech debt" in the form of worse performance and high server load, which we will have to pay off eventually. Perhaps if I was a better programmer, I would have had the skills to implement the first option right, on the first try. But I don't know many (any?) programmers who do have those kinds of skills, while also having the wherewithal to document the thing, write useful and nontrivial test assertions, develop tooling for OpenAPI conformance, et alia. Not to mention that the flexibility granted by simplicity actually has turned out to be very important for being able to respond to changes in product requests quickly. It seems (so far) that I did make the right decisions in this situation. I think that makes me at least a decent software engineer.
sfteus 5y ago
This is an insightful comment that holds true a good portion of the time. However, there is one takeaway that is overlooked in my experience: resolving tech debt that exists due to lack of experience tends to build said experience, _provided_ the less-experienced developers are involved in understanding the underlying issue and planning a solution.
sirwhinesalot 5y ago
If you've worked with an engineer that "intended" to accrue technical debt, I feel sorry for you but I hope you realize how uncommon that is. Technical debt is the answer to management requests that cannot be accomplished in the requested time with the available resources. No more, no less. The word debt refers to the fact that the shortcut needed to do so will cause problems repeatedly in the future (interest that will need to be paid). In my experience, engineers will make this clear to management, and will point out the additional time needed to implement a solution that avoids those problems, and management will make the decision to continue as is. For a concrete example, management requests that you implement real-time collaborative editing in an existing tool. You have enough time to replace all the backend stuff with a CRDT or OT based implementation, but the frontend architecture is not equipped to deal with collaborative editing and there is no time to do both with the people available. So, in order to meet the management deadline, quick hacks are added to bolt on the collaborative UI on top of the existing UI. Eventually new features have to be added that simply cannot be supported in the existing UI architecture without massive workarounds. Carving out some time to slowly refactor the UI architecture is rarely done, so the problem accumulates until development speed slows to a crawl. That is technical debt, and it's entirely a byproduct of management/development negotiations. Has nothing to do with engineers. Nobody could have predicted that the architecture would need to support an entirely different backend in the future. EDIT: Just to be clear, technical debt happens as a result of a conscious tradeoff between one or more options, where the more problematic option is chosen to get to an "acceptable" solution in a shorter period or with less resources. Some dev writing a shitty architecture is not technical debt. Depending on a library that later gets EOLed is not technical debt. Figuring out you have a shitty architecture or an EOL library and choosing to continue building on it despite clear evidence it is a bad idea, in service of shorter time to market, that is technical debt.
TameAntelope 5y ago
This is true sometimes, but a few things happen as you keep writing software: 1. By writing the needed code the "debt incurring" way, you learn how to do it the "right" way. So you may start with inexperience, but you gain the experience and are now better able to fix the issues you introduced. 2. Similar to this, even if you have all of the necessary experience and skill, you still may not be fully able to picture how the feature will need to evolve in the future to support additional use cases. Once you see that happening, you can design a better implementation, but not before then. Sometimes you have to do it wrong to get the feedback you need to do it right. 3. There are often real tradeoffs that get made that aren't about skill, but about time. I don't think you can discount this situation, in my experience it happens very, very often. A great way to avoid nearly all of this is to just set "EOL criteria" for your software. "This project will not work beyond 100 users." or, "This feature will need a rewrite once we have to account for the edge case we ignored in this implementation." Know the limits of what you've built, and make it clear that you need to rebuild once you've reached those limits, and it's at least marginally more likely you'll get the support you need from the people with the purse strings.

louwrentius 5y ago

The best way to deal with tech debt is to never use the term in the first place. Almost nobody really uses tech debt as a concept the way it was intended by Ward Cunningham. It seems mostly to be abused to absolve oneself or others of responsibilities. I or others didn’t make a mistake, it was just “tech debt”. Sounds much more benign like this. Like tech debt is inevitable, unavoidable and current issues could not have been prevented. Yeah, sure. When I hear people talk about tech debt, my bullshit meter is immediately on high sensitivity. I fully understand why people don’t want to expose their own mistakes or those of others. But I think it kills trust for those who see right through it. Having to talk about tech debt instead of the real, actual issues feels emotionally immature. I mean this in a kind way, but I mean it. It results in this whole cottage industry dealing with tech dept, in a similar vein as the Agile industry. I don’t think all this dressing up of rather clear cut issues as tech debt adds any value and only creates a layer of bullshit work. I think we need less bullshit and more environments where people can openly admit to and discuss mistakes without immediately being raked over the coals / attacked.

michaelfeathers 5y ago
A term I use to recenter on Ward Cunningham's concept is 'Design Lag' - How far is the code behind your current understanding of the problem?
- louwrentius 5y ago
  That is something I can get behind. But the rest of the world has taken the concept of TD and warped it to mean ‘my fuckup with a more neutral label’. Check definitions when you talk about TD with someone.
ess3 5y ago
I agree with you. But I guess the problem stems from having to prioritize between building the next shiny thing or actually going back and fixing the problem. Even worse is that many of these problems have rather low short-term impact and a much larger long-term impact, which makes it even harder to justify/explain to the person making the prioritization. It's much easier to hide it behind the facade of "technical debt" to which everyone intuitively agrees it's a problem. But yes having people with the competence to actually understand the clear cut issues also making the prioritization would be nice.

lbriner 5y ago

It is true that the intention of tech debt is supposed to be when you make a conscious choice to take a shortcut now knowing that you will pay interest later and that other people use it to mean anything suboptimal whether due to lack of skill or changing requirements but in many ways the OP article stands for all of it anyway. Defects, tech debt, refactors, whatever you want to call them need to be measured against all other work in the backlog and given as objective cost and risk as possible. Engineers tend to like neat and tidy but there is terrible code in our system that has simply never been changed and still works so we leave it alone.

gnat 5y ago
From what I've seen, there are three big schools with technical debt (past choices that now make changes difficult): 1) Ignore it and gradually ossify as the code congeals and it becomes harder and harder to make any change at all. 2) Assign a fixed budget of time, e.g. 20%, to refactoring and bugfixing and updating models and "moving to the new version of the language/library which lets us ...". 3) Prioritise the engineering tidy-ups alongside features. This involves (a) being able to credibly estimate the effort and value in the same dimensions as feature work is estimated and prioritised; (b) the people who do the prioritisation believing that engineering-desired work is as necessary as sales-desired or PM-desired work. That last route requires people's beliefs and abilities to align. The middle route bypasses the entire prioritisation conflict at the cost of sometimes working too much or too little on technical debt. I think that's why you see a lot of fixed budgets for internal work.

suncore 5y ago

Here's a few observations (after long time experience and involvement in research around technical debt): 1) It is impossible to avoid gathering technical debt. The code will deteriorate in one way or another. You need to prepare to fix it since you can't avoid it. 2) It is so extremely difficult to make a correct "risk assessment" on technical debt so you should avoid doing so at all. You will just end up arguing all day on the merits of clean code/architecture vs feature growth. Instead, keep two backlogs and reserve a set amount of resources on each, e.g. 20% on reducing technical debt and 80% for product features and other development. 3) The "cost" of reducing technical debt is actually negative. That's the whole point of working with it, to increase development velocity.

crdrost 5y ago
Code doesn't deteriorate... Like we're talking about a banana growing spots or whatever... Code gets plastered over with features and abstraction layers, but that's an active process that we are complicit in doing. More germane for this discussion, technical debt was originally defined as a positive thing that you want to go get... It is the mismatch between our domain model and how our users think about the domain. It is positive because “enough with all of the planning and interviewing and requirements gathering and careful architecting, can I just build something and have my users criticize it and do four or five drafts until I get it right?” ... The debt is the drafts before you get it right, the interest is the constant translation between the language of your users and the language that the system is expressed in. You have a “contracts” table that contains things that are not contracts because every purchase foreign keys to a contract, but your users have since wanted to know what they do with the purchases that are not associated with any contract, where do they go. And now every query that aggregates over contracts needs to exclude the non-contract contracts, this is part of the interest you are paying. But at some point it started to mean that we had to upgrade our dependencies, that's tech debt, or this kludge that I threw in, that's tech debt, or the fact that we never worked out a shared library between the front end and the back end and so any Python code in utils.py needs to be manually kept up to date with a file utils.js in a separate Git repo so that we are sure that we can do these things both on the front end and the back end. More broadly, anything that we no longer care for is tech debt. And that's where you really get this idea that it is deteriorating, that's more a measure of our own patience deteriorating, especially as we never seem to have time for the refactors we want. Perversely the cause of not being able to refactor has nothing to do with tech debt and cannot be solved in this way. It's multifaceted so at different places it emerges in different ways, but usually it's an incentive problem. At some places it is that the only incentives are for feature development. At other places it is that every developer is working on a different thing rather than prioritizing one thing that the business needs and delivering it, and allowing developers to use their slack time in this equation to improve the product however they see fit... In yet another it is because the team lead resists any suggestion that the framework being used is too heavyweight for the problem being solved and so trying to keep these very clean abstraction layers is causing people to have to rewrite the same basic thing in five different unrelated places, because it has to bubble up from the data layer into the service layer into the business layer into the controller layer into the API into the consumer layer into the app state layer into the frontend model layer...

WallyFunk 5y ago

Would be interesting to apply game theory to tech debt, because it is, after all, The Great Game we all play, whether we admit it or not.

ochronus 5y ago
How would you imagine it? Sounds interesting indeed!

0xbadcafebee 5y ago

I think you can visualize tech debt by recording a chart where the indicator "debt" is $cost = ($num_tech_debt * $instance). $num_tech_debt is the number of different pieces of technical debt, and instance is a number incremented each time technical debt affects you. For every "instance" of technical debt affecting something, the cost goes up. If you aren't addressing tech debt, the chart will just go up and up and up. The only way to make it go down is to eliminate the number of technical debt things, which will reduce the cost equation (unless some other technical debt's instances go up). I suck at math, so maybe somebody else has a better way to represent that?

ochronus 5y ago

I love the discussions folks :) keep it up, I'm learning a lot <3

lowbloodsugar 5y ago

Debt would be if someone said "I need to build a bridge across here that can handle an Army, and I've got $1m", we engineers reply "It will take $2m", and they respond "Ok, I will borrow $1m so you can build the bridge I need". Instead what happens is they say "Well, build what you can for $1m", and you say "Ok, we can make 'a bridge' for that", and then either (a) your infantry can cross, but the tanks have to get diverted 20 miles out of the way, or (b) the tanks end up in the river along with the bridge. Since (b) is bad, you then have to spend a lot of time planning the routes for the tanks, and making sure the tanks have the right air cover, etc etc, i.e. doing more work. More likely, however, is that the manager, who is not a structural engineer, sees a perfectly good bridge and orders the tanks across anyway, causing the loss of the bridge, the tanks, and the war. It's not debt. It's just (at best) an incomplete solution or (at worst) a bad solution that fails at the worst possible moment - e.g. database collapses during registration for the largest event of the year. Ah, but surely, if you build the lightweight solution for $1m, and acknowledge the increased costs of managing the problems that it doesn't solve, then thats fine? Sure, but that's not technical debt either! That is scoping: we (engineers + business) identify a workable solution that provides some business value. And then we do that well. When Cunningham, for example, talks about what to do about debt, e.g. YAGNI, these are all good ideas: scoping. But the term "debt" is incorrect in this case. The term "debt" will only ever get a team in trouble as it indicates that the team is unwilling to confront the actual problem as engineering, but only as a broken metaphor.

joshfee 5y ago
What you describe is not what I've experienced with debt. Usually it is a conflict where short term goals and long term goals are at odds with each other. Using your bridge analogy, its more like "We only need an infantry bridge right now, but we know that in a year we need to support tanks". Well we can either spend $2m to build the tank bridge now, or we can spend $1m today for the infantry, but in a year you'll have to spend another $1.5m to make modifications to support tanks. The debt there is knowing that "we will spend $500k more doing it this way (interest) but we will benefit in the meantime". There are other factors too that make the true cost of the debt higher. For instance less flexibility as requirements change. I often find myself saying "this would be an easy change if we had initially built it in the more extensible way, but at the time you didn't need/want for extensibility so now your easy feature is actually a large refactor"

sam0x17 5y ago

One thing I've had a lot of success with as I help early to mid stage (usually YC) startups try to tune things is empowering engineering by giving bizdev and stakeholders the understanding and expectation that ~25% of eng's time should be spent on things like writing tests, refactoring, and otherwise reducing tech debt. Putting a focus on reducing tech debt up front via CI, automated testing (including both unit and integration tests), coding style guidelines, CI checks for all of the above, etc., and giving engineering the time to attend to these issues and practices has turned around many flailing or failing startups that I've consulted with and/or provided advice to. The ones that do fail are usually the ones that resist addressing tech debt in favor of shipping more features. In truth, it is usually quite possible to do both at the same time, and it is in the very least possible to tackle new features in a way that doesn't increase tech debt at the previous bad rate through updated automated testing and CI requirements that ensure new features are fully tested before they are merged. The hardest thing, especially with non-technical founders, is getting them to understand that yes, their precious features need to be pushed back a bit, but that this will prevent major issues, downtime, and slowdowns next quarter or next year, and that issues they are experiencing now are often due to neglecting tech debt in the past. Once they get this, the company usually has a much better outlook. This is also an investing criteria (what is your ongoing plan to reduce technical debt) for some of the better tech investors, I've heard.

anamax 5y ago
What's missing in the above is some notion that the "feature machines" should actually do some work to show what the feature will accomplish. Better yet, is that feature more profitable than some other feature? "you can have X in three months or Y in two weeks but if you pick either one, that delays Z by the same amount of time" focuses the attention.
- sam0x17 5y ago
  Very, very true. Part of my process usually involves an ICE-based prioritization of the feature backlog and being very particular about what will be tackled in the short term.