6 ms·

Technical Debt

90 points by baha_man 18y ago

DannoHung 18y ago
Can I securitize technical debt?
- gruseom 18y ago
  The people who pay the debt back probably have to be the same ones who accumulated it in the first place, otherwise the learning process (for which the whole thing is a metaphor) is short-circuited.
- run4yourlives 18y ago
  If you are a contractor, it can actually be profit! :-)
  - deleted 18y ago
    [deleted]
joe_the_user 18y ago
A great, crucial article. Even after reading the article, I think that folks often don't understand that Technical Debt is not a "bad thing" but a thing that must be understood and managed. When you understand this, it puts software engineering into a very clear perspective.
- jsdalton 18y ago
  > a thing that must be understood and managed. That's true, but how do you manage it? Especially when it's so difficult to measure in the first place. One idea that popped into my head would be to keep a running tab of technical debt that's being accumulated in a project. It's tough though, because developers who are in "accumulating debt" mode tend to be the ones with the least amount of time to spend on upkeeping a document that tracks their debt.
  - gojomo 18y ago
    I leave a lot of TODO and FIXME comments in code (which are then tracked and highlighted by my IDE). These range from code that may actually be broken on certain rare or unexpected corner cases (though not dangerously so), to known inefficiencies, to areas where the rough idea for a much better approach exists but can't yet be fully developed. Very roughly, they are a measure of known technical debt.
    - rs 18y ago
      Good practice. Do you end up cleaning them up ?
      - gojomo 18y ago
        I think I wind up adding new ones at about the same rate as old ones disappear, but I'd have to run a report over the history to know for sure. It's always nice when I come across a note that's been made irrelevant by other parallel improvements along the same theme -- even though the note wasn't the impetus for the other changes. It's like finding a $20 on the street!
    - etal 18y ago
      Same here (in gvim). I also like pylint's default behavior of issuing warnings where it finds these and penalizing the code's final score in the report -- it's a proxy for code problems that have been silenced or worked around, but still need some attention later. Integrating this info with a bug tracker seems like a good idea too for sufficiently complex projects.
  - gruseom 18y ago
    It's managed by the thought process of the programmers. A good programmer knows when and where the technical debt is and wants to pay it off. If the programmers don't understand this or don't care, there's not much that can be done. Teaching works, up to a point (but no further). You can't manage this with documents for the same reason you can't manage thinking with documents. If I were going to try anyway, and were working with a team of more than 2 or 3, I might put a big piece of paper on the wall and ask everyone to write on it whenever they notice technical debt. The bigger the issue, the bigger the note. Maybe use different colors for different parts of the system. After a while, patterns would emerge. But the real purpose of this would not be to track accumulated debt; it would be a device to encourage people to talk about it and think about it.
- joe_the_user 18y ago
  Hmm, I had seen a Martin Fowler article on technical debt before but the link to Ward Cunningham is new at the least. As a part-time economist, it occurs to me that the debt metaphor actually doesn't work on some levels: Companies constantly increase their monetary debt, since THAT can be paid through increased profits. You actually wouldn't want a company that grew without increasing debt - that would be wasted resources. Technical debt is much worse to accumulate since it can only be paid by increased development time. Also, one accumulates technical debt whether one wishes to or not. This is because often the only way to understand how to do a system right is to first do it wrong.
jwb119 18y ago
the article is good, but i can't help the feeling that it is just applying a clever name to a common sense principle that is relevant to just about every field (including finance and technology)
- StrawberryFrog 18y ago
  If giving it a clever name is the only way to explain it to management (which may be the case here) then it's worth it.
  - Clark76 18y ago
    "Technical Debt" is a surprisingly self-explanatory term. I have used it to great success with my management team when arguing for time for refactoring, code reviews, and other non-user facing work.
    - StrawberryFrog 18y ago
      Exactly. technical debt is a great tool for explaining to non-technical management why you'd want to invent time in refactoring, reviews etc that have no obvious tangible result in the finished product (short version: it's a drag factor makes it harder to add features and keep the bug count down later). It's also a good tool to allow the management to do what management does best: take the technical information and make management/financial decisions based on it, on how much investment can be made now. It turns a binary either/or choice: devs saying "Our code is bad, we need to rewrite" into a quantitative one: "How much technical debt do we have? Is it increasing? Can we reduce it?".
gruseom 18y ago
It's good to see Ward Cunningham recognized for one of his many great contributions. I once heard a whole NPR piece about wikis and their history which didn't mention Ward as their inventor. It only occurred to me later that I should have written in. The video of him talking about the debt metaphor is worth watching, especially the last minute where it gets subtle.
mattjaynes 18y ago
Great article. It reminds me of another dilemma I came across earlier this week. I had a 280MB xml file containing about 120,000 records. I needed a way to parse out a subset of the records (about 15,000) and then put their data into a db. I was developing on a vps with only 256MB of RAM, so I wanted to avoid memory intensive operations. I'm using ruby, so I started looking into how to 'stream' the xml with ruby in such a way that I could read if it was the type of record I wanted to keep, somehow manipulate and store the data and move to the next record. The more I looked into that strategy, the more complex and ominous it seemed. I just really wanted to find a simpler way that wouldn't require all the apparent tediousness of streaming the xml file (it may seem simple enough, but there are dependencies you have to install, api's you have to read and learn, etc). There were just too many moving pieces and it made me feel eery about the outcome. Another aspect of it was just pure laziness. I just don't care how I get that data into the db - I just want to get it in there. Fortunately, the glorious wonderfulness of linux utilities saved me from a long tedious solution. Behold: csplit -q catalog.xml '/<title_index_item>/' '{*}' This splits the large xml file into sub-xml files that start with 'xx' by default, followed by an incremental number, xx10004 for example. It splits the file based on the <title_index_item> tag - which is the tag for the items I want. See `man csplit` for more info... find xx* | xargs grep -L 'label="instant"' | xargs rm -f The 'find' lists all the sub-xml files, then we grep for the filenames that do not contain what I'm looking for, then delete those files. I'm left with a directory of 15,000+ xml files with just the type of data I'm interested in. find . | wc -l This command is just so I can track the progress of the operations. Obviously `ls` would return too much data, so we pipe the file list to `wc -l` which gives us the number of lines - which in this case is the number of files in the dir. So, two lines of code on the command-line instead of a far-more complex ruby/streaming-xml solution. Now I can have a ruby script process each individual file and add the data to the db - a much simpler problem to deal with. My point is that you can accumulate code debt by doing the seemingly 'right' solution sometimes. There are probably coders that will cringe that I just used a couple of shell commands to do this rather than write up a long, well-documented, properly OOP, TDD, etc "right" solution. However, the "right" solution in that case would have accumulated code debt - more code to maintain, more moving pieces, more things that can go wrong. I'd trade maintaining two lines of shell code over 10's or 100's of lines of ruby code and it's dependencies any day of the week.
10ren 18y ago
Technical debt is appropriate in prototype code, where you aim at getting something you can play with as a user; and also get a feel for its technical feasibility. This works fine. Unfortunately, I ended up wanting to go back to the code to reuse/build on my approaches to the key technical problems... but the hacked mess was tortuous to read or understand (interest payments). So the dilemma is: how to write readable code, when it's only for a prototype and so not worth the effort? My current tradeoff is to make the prototype code too simple, by cutting out tricky special cases, and by deleting approaches I coded and all my comments discussing them, assessing issues etc. That is, make the code "workmanlike". It might be stupid, incomplete, inefficient and incorrect - but in a crystal clear way. I actually find this discipline incredibly difficult, and I often lose days at a time exploring alternative coding. This often pays off; but the biggest payoff comes from having a clear and concrete basis to build on. I've just finished the current prototype, and many solutions to technical issues have now clicked into place as obvious to me - because I can now see them. So that works well. And I also have something - incomplete, buggy, inefficient etc - that I can play with as a user. Ugh. In other words: a crystal clear sense of what needs to be improved. Maybe..."auditable"?
akeefer 18y ago
I've thought a lot about why technical debt seems to be such a hard concept for non-engineers to grasp; certainly given how few business people or managers seem to understand it, it clearly must be difficult. I think it's because A) it only affects a truly long-lived, stateful process, like a software (or many other types of engineering process) where the output of one phase is the input to the next phase and B) there just aren't that many sorts of processes that occur in ordinary life. Sure, in any field you can screw up in such a way that it hurts you in the future by damaging your reputation or annoying your customers, or make bad technical investments that turn out to cost you. But in engineering disciplines, doing your core activity (engineering) poorly can hurt your ability to keep doing your core activity (engineering) in the future. If you're in sales and you cut corners with one client, it doesn't make your phone calls take longer. If you're in a restaurant and you cook things too fast or cut corners on ingredients, it doesn't make your future cooking take longer. If you're a doctor and you're too hasty and mis-diagnose a patient, it doesn't make your future surgeries and checkups take longer. But in software, if you're too hasty adding feature A, it does make your future features take longer to develop. And that's because in software, your future features are built on the previous ones, whereas with sales, cooking, or medicine the duration of any given prospect/customer/patient is relatively short-lived and finite. That's my best theory as to why it's so hard for non-engineers to understand.
skmurphy 18y ago
Gerald Weinberg refers to this same concept as "maintenance debt" in his four volume "Quality Software Engineering Series" from the early-mid-90's. I think it's an independent formulation of the same concept. The volume most concerned with it is Vol4 "Anticipating Change" http://www.amazon.com/Quality-Software-Management-Anticipating-Change/dp/0932633323 http://www.amazon.com/Quality-Software-Management-Anticipati... The whole series is definitely worth reading.
kailashbadu 18y ago
In a nutshell, could it be argued that kludges are fine as long as the programmer? • knows the exact location of the hacks. • Is aware of the repercussions and is assured that the system can get away with the kludges for the time being without any hindrance in the system’s effectiveness to get things done. • Knows how to fix them if need be. • Communicate the situation with others, including the fellow programmers and the managers.
biohacker42 18y ago
I'm not sure debt is a good metaphor for code complexity. Debt grows logarithmically, if the complexity of your code base is growing logarithmically you're doing great. The problem is software complexity grows exponentially. And that's why you can't manage it like you can manage high interest debt. You can try to manage it, but I have yet to see that management succeed. Every single time a company has tried to manage technical debt, the debt simply goes unpaid. How many people here who've worked on a sizable and mature code base can name temp files and variables that have turned absolutely permanent over the years? That's why software complexity, in large organizations, should be approached like an infestation. Erase and eradicate on sight, don't try to save some for later.
alrex021 18y ago
First time I came across this metaphor is not to long ago reading the "Getting Real" book by 37signals. http://gettingreal.37signals.com/ http://gettingreal.37signals.com/
djahng 18y ago
That's just like Herbert Kroemer's Conservation of Misery. PhD students in ECE at UCSB know all about that.
alecco 18y ago
Technical Debt is a very good new metaphor. This article looks like a quick blogspam/linkbait/younameit. This guy writes books on UML.

Technical Debt

90 points by baha_man 18y ago

DannoHung 18y ago

Can I securitize technical debt?

gruseom 18y ago
The people who pay the debt back probably have to be the same ones who accumulated it in the first place, otherwise the learning process (for which the whole thing is a metaphor) is short-circuited.
run4yourlives 18y ago
If you are a contractor, it can actually be profit! :-)
- deleted 18y ago
  [deleted]

joe_the_user 18y ago

A great, crucial article. Even after reading the article, I think that folks often don't understand that Technical Debt is not a "bad thing" but a thing that must be understood and managed. When you understand this, it puts software engineering into a very clear perspective.

jsdalton 18y ago
> a thing that must be understood and managed. That's true, but how do you manage it? Especially when it's so difficult to measure in the first place. One idea that popped into my head would be to keep a running tab of technical debt that's being accumulated in a project. It's tough though, because developers who are in "accumulating debt" mode tend to be the ones with the least amount of time to spend on upkeeping a document that tracks their debt.
- gojomo 18y ago
  I leave a lot of TODO and FIXME comments in code (which are then tracked and highlighted by my IDE). These range from code that may actually be broken on certain rare or unexpected corner cases (though not dangerously so), to known inefficiencies, to areas where the rough idea for a much better approach exists but can't yet be fully developed. Very roughly, they are a measure of known technical debt.
  - rs 18y ago
    Good practice. Do you end up cleaning them up ?
    - gojomo 18y ago
      I think I wind up adding new ones at about the same rate as old ones disappear, but I'd have to run a report over the history to know for sure. It's always nice when I come across a note that's been made irrelevant by other parallel improvements along the same theme -- even though the note wasn't the impetus for the other changes. It's like finding a $20 on the street!
  - etal 18y ago
    Same here (in gvim). I also like pylint's default behavior of issuing warnings where it finds these and penalizing the code's final score in the report -- it's a proxy for code problems that have been silenced or worked around, but still need some attention later. Integrating this info with a bug tracker seems like a good idea too for sufficiently complex projects.
- gruseom 18y ago
  It's managed by the thought process of the programmers. A good programmer knows when and where the technical debt is and wants to pay it off. If the programmers don't understand this or don't care, there's not much that can be done. Teaching works, up to a point (but no further). You can't manage this with documents for the same reason you can't manage thinking with documents. If I were going to try anyway, and were working with a team of more than 2 or 3, I might put a big piece of paper on the wall and ask everyone to write on it whenever they notice technical debt. The bigger the issue, the bigger the note. Maybe use different colors for different parts of the system. After a while, patterns would emerge. But the real purpose of this would not be to track accumulated debt; it would be a device to encourage people to talk about it and think about it.
joe_the_user 18y ago
Hmm, I had seen a Martin Fowler article on technical debt before but the link to Ward Cunningham is new at the least. As a part-time economist, it occurs to me that the debt metaphor actually doesn't work on some levels: Companies constantly increase their monetary debt, since THAT can be paid through increased profits. You actually wouldn't want a company that grew without increasing debt - that would be wasted resources. Technical debt is much worse to accumulate since it can only be paid by increased development time. Also, one accumulates technical debt whether one wishes to or not. This is because often the only way to understand how to do a system right is to first do it wrong.

jwb119 18y ago

the article is good, but i can't help the feeling that it is just applying a clever name to a common sense principle that is relevant to just about every field (including finance and technology)

StrawberryFrog 18y ago
If giving it a clever name is the only way to explain it to management (which may be the case here) then it's worth it.
- Clark76 18y ago
  "Technical Debt" is a surprisingly self-explanatory term. I have used it to great success with my management team when arguing for time for refactoring, code reviews, and other non-user facing work.
  - StrawberryFrog 18y ago
    Exactly. technical debt is a great tool for explaining to non-technical management why you'd want to invent time in refactoring, reviews etc that have no obvious tangible result in the finished product (short version: it's a drag factor makes it harder to add features and keep the bug count down later). It's also a good tool to allow the management to do what management does best: take the technical information and make management/financial decisions based on it, on how much investment can be made now. It turns a binary either/or choice: devs saying "Our code is bad, we need to rewrite" into a quantitative one: "How much technical debt do we have? Is it increasing? Can we reduce it?".

gruseom 18y ago

It's good to see Ward Cunningham recognized for one of his many great contributions. I once heard a whole NPR piece about wikis and their history which didn't mention Ward as their inventor. It only occurred to me later that I should have written in. The video of him talking about the debt metaphor is worth watching, especially the last minute where it gets subtle.

mattjaynes 18y ago

Great article. It reminds me of another dilemma I came across earlier this week. I had a 280MB xml file containing about 120,000 records. I needed a way to parse out a subset of the records (about 15,000) and then put their data into a db. I was developing on a vps with only 256MB of RAM, so I wanted to avoid memory intensive operations. I'm using ruby, so I started looking into how to 'stream' the xml with ruby in such a way that I could read if it was the type of record I wanted to keep, somehow manipulate and store the data and move to the next record. The more I looked into that strategy, the more complex and ominous it seemed. I just really wanted to find a simpler way that wouldn't require all the apparent tediousness of streaming the xml file (it may seem simple enough, but there are dependencies you have to install, api's you have to read and learn, etc). There were just too many moving pieces and it made me feel eery about the outcome. Another aspect of it was just pure laziness. I just don't care how I get that data into the db - I just want to get it in there. Fortunately, the glorious wonderfulness of linux utilities saved me from a long tedious solution. Behold: csplit -q catalog.xml '/<title_index_item>/' '{*}' This splits the large xml file into sub-xml files that start with 'xx' by default, followed by an incremental number, xx10004 for example. It splits the file based on the <title_index_item> tag - which is the tag for the items I want. See `man csplit` for more info... find xx* | xargs grep -L 'label="instant"' | xargs rm -f The 'find' lists all the sub-xml files, then we grep for the filenames that do not contain what I'm looking for, then delete those files. I'm left with a directory of 15,000+ xml files with just the type of data I'm interested in. find . | wc -l This command is just so I can track the progress of the operations. Obviously `ls` would return too much data, so we pipe the file list to `wc -l` which gives us the number of lines - which in this case is the number of files in the dir. So, two lines of code on the command-line instead of a far-more complex ruby/streaming-xml solution. Now I can have a ruby script process each individual file and add the data to the db - a much simpler problem to deal with. My point is that you can accumulate code debt by doing the seemingly 'right' solution sometimes. There are probably coders that will cringe that I just used a couple of shell commands to do this rather than write up a long, well-documented, properly OOP, TDD, etc "right" solution. However, the "right" solution in that case would have accumulated code debt - more code to maintain, more moving pieces, more things that can go wrong. I'd trade maintaining two lines of shell code over 10's or 100's of lines of ruby code and it's dependencies any day of the week.

10ren 18y ago

Technical debt is appropriate in prototype code, where you aim at getting something you can play with as a user; and also get a feel for its technical feasibility. This works fine. Unfortunately, I ended up wanting to go back to the code to reuse/build on my approaches to the key technical problems... but the hacked mess was tortuous to read or understand (interest payments). So the dilemma is: how to write readable code, when it's only for a prototype and so not worth the effort? My current tradeoff is to make the prototype code too simple, by cutting out tricky special cases, and by deleting approaches I coded and all my comments discussing them, assessing issues etc. That is, make the code "workmanlike". It might be stupid, incomplete, inefficient and incorrect - but in a crystal clear way. I actually find this discipline incredibly difficult, and I often lose days at a time exploring alternative coding. This often pays off; but the biggest payoff comes from having a clear and concrete basis to build on. I've just finished the current prototype, and many solutions to technical issues have now clicked into place as obvious to me - because I can now see them. So that works well. And I also have something - incomplete, buggy, inefficient etc - that I can play with as a user. Ugh. In other words: a crystal clear sense of what needs to be improved. Maybe..."auditable"?

akeefer 18y ago

I've thought a lot about why technical debt seems to be such a hard concept for non-engineers to grasp; certainly given how few business people or managers seem to understand it, it clearly must be difficult. I think it's because A) it only affects a truly long-lived, stateful process, like a software (or many other types of engineering process) where the output of one phase is the input to the next phase and B) there just aren't that many sorts of processes that occur in ordinary life. Sure, in any field you can screw up in such a way that it hurts you in the future by damaging your reputation or annoying your customers, or make bad technical investments that turn out to cost you. But in engineering disciplines, doing your core activity (engineering) poorly can hurt your ability to keep doing your core activity (engineering) in the future. If you're in sales and you cut corners with one client, it doesn't make your phone calls take longer. If you're in a restaurant and you cook things too fast or cut corners on ingredients, it doesn't make your future cooking take longer. If you're a doctor and you're too hasty and mis-diagnose a patient, it doesn't make your future surgeries and checkups take longer. But in software, if you're too hasty adding feature A, it does make your future features take longer to develop. And that's because in software, your future features are built on the previous ones, whereas with sales, cooking, or medicine the duration of any given prospect/customer/patient is relatively short-lived and finite. That's my best theory as to why it's so hard for non-engineers to understand.

skmurphy 18y ago

Gerald Weinberg refers to this same concept as "maintenance debt" in his four volume "Quality Software Engineering Series" from the early-mid-90's. I think it's an independent formulation of the same concept. The volume most concerned with it is Vol4 "Anticipating Change" http://www.amazon.com/Quality-Software-Management-Anticipating-Change/dp/0932633323 http://www.amazon.com/Quality-Software-Management-Anticipati... The whole series is definitely worth reading.

kailashbadu 18y ago

In a nutshell, could it be argued that kludges are fine as long as the programmer? • knows the exact location of the hacks. • Is aware of the repercussions and is assured that the system can get away with the kludges for the time being without any hindrance in the system’s effectiveness to get things done. • Knows how to fix them if need be. • Communicate the situation with others, including the fellow programmers and the managers.

biohacker42 18y ago

I'm not sure debt is a good metaphor for code complexity. Debt grows logarithmically, if the complexity of your code base is growing logarithmically you're doing great. The problem is software complexity grows exponentially. And that's why you can't manage it like you can manage high interest debt. You can try to manage it, but I have yet to see that management succeed. Every single time a company has tried to manage technical debt, the debt simply goes unpaid. How many people here who've worked on a sizable and mature code base can name temp files and variables that have turned absolutely permanent over the years? That's why software complexity, in large organizations, should be approached like an infestation. Erase and eradicate on sight, don't try to save some for later.

alrex021 18y ago

First time I came across this metaphor is not to long ago reading the "Getting Real" book by 37signals. http://gettingreal.37signals.com/ http://gettingreal.37signals.com/

djahng 18y ago

That's just like Herbert Kroemer's Conservation of Misery. PhD students in ECE at UCSB know all about that.

alecco 18y ago

Technical Debt is a very good new metaphor. This article looks like a quick blogspam/linkbait/younameit. This guy writes books on UML.