13 ms·

How a startup can survive technical debt

136 points by fidrelity 6y ago

quantified 6y ago
I work in a fear-based culture right now. The CTO never saw a kludge he didn’t want into production stat, and in his dual role as CEO won every disagreement. There are many pieces on tech debt that are true and valid, this one IMHO covers more salient issues than most. If you’re reading this comment before you read the article: it’s about how to avoid getting caught too badly in the first place, not about magic bullets to get you out when you’re basically bankrupt.
- rendall 6y ago
  > I work in a fear-based culture right now That sucks. I have been there. Awful. For me, the silver lining is that I no longer ignore stinky red flags
  - minimuffins 6y ago
    Yeah, I think it can be painfully helpful to go through that miserable experience at least once. But my very strong advice to others: Do NOT put up with that. You don't have to. There are so many companies, even now, where you can get decent work with much more sensible norms. There's always kind of a gravitational pull toward excess debt because "get the [fix/feature] out there in front of [users/client/exec]" is such a tempting drug for management, even if they're generally good. But the amount and toxicity of the debt varies widely. Find a company that's sane about this if you value your own sanity imo.
    - quantified 6y ago
      Thank you for the support! Doing my best, and leaving when the right opportunity appears.
  - hndudette2 6y ago
    Yep, same, and it hurts the most loyal and hardworking the most. Do what's best for yourself and find a better job.
- ianamartin 6y ago
  I will never again work in a company with a dual CEO/CTO. No matter how well-intentioned and talented that person is, it always leads to a toxic environment. It's an instant red flag that the person is incapable of delegating responsibilities. Separation of duties at the C-level is important. There needs to be some amount of healthy friction in the leadership team to make appropriate compromises. Hope you are able to get out of this situation soon.
KingOfCoders 6y ago
From my experience with lots of VC money.
s17n 6y ago
"Revenue solves all known problems"
Haydos585x2 6y ago
I enjoyed this article and I think it's helpful for technical leaders in early stage startups. I feel like a lot of these companies will continue having problems mostly because the people that are attracted to smaller/early-early companies are the same kinds of people that don't want to work in structured (more corporate/deliberate) environments. My experience is a bit more on the agency side where technical debt is caused by developers/project managers not really caring about the longevity of an application. The idea of going back and fixing anything that isn't specifically called out and invoiced for is so far away from the priorities of these businesses.
- austincheney 6y ago
  I suspect the tech debt that many start ups encounter is a matter of execution. They likely require original solutions to tough problems in very short time. The way to reduce tech debt in this case is to make deliberate time at great expense to frequently refactor complexity out of the system. In the corporate world, on the other hand, the problem is an astonishing fear of originality. Most places I have worked have invented here mentality baked into every decision. Every 5 to 8 years they start over completely with a different tool, framework, or language hoping it magically solves for the group’s prior poor implementation.
- clumsysmurf 6y ago
  > My experience is a bit more on the agency side where technical debt is caused by developers/project managers not really caring about the longevity of an application. The idea of going back and fixing anything that isn't specifically called out and invoiced for is so far away from the priorities of these businesses. This resonates with me. I consider myself a better fit for product focus vs project focus, preferring to find something meaningful and nurture it in terms of value to customer, and value to developers (code quality). Recently I accepted a job at an Agency, which a coworker described as a "chop shop". I became depressed. The app I would work on, got frenzy activity from Oct - March. It had to be released in March no matter what. Between that time there was no activity, no refactoring, no addressing tech debt. In fact, one week in December the client was unable to pay for some technical budgeting reason and we all just sat around for that week despite looming deadline. Well, over the years this thing accumulated tech debt. Even though it was a cash cow for the agency: reliable income at reliable times from a good customer. One would think the health of the product would somehow invite extra effort. Every year new devs would start working on it (new devs, because turnover at these places is high). They would look at the mess and begin resume driven development to use the shiniest new things that were left out, hoping to fix and compensate for the problems from just sheer neglect, and then move on for the cycle to repeat. That said, there are people that just prefer hopping from project to project to experience new things.
k__ 6y ago
How many startups are actually killed by tech debt? I saw many successfull companies with shitty software. I had the impression the business part of things killed much more companies if they didn't get it right.
- 29athrowaway 6y ago
  Tech debt rarely kills a company, but it does kill teams, in descending hierarchical order. It usually goes like this: Day 1 Stakeholder: "The app is down, what is happening?" Engineering lead: "We are working on it. Do not worry". Day 2 Stakeholder: "The app is down, what is happening?" Engineering lead: "Do not worry, we got this". Fast forward to Day 15: Stakeholder: "Can you explain why the app is down again?" Engineering lead: "We got this" Stakeholder: "That is what you said last 2 weeks ago. We will send someone to take a look at it" External auditor: "wtf wtf wtf. These guys do not know what they're doing." Day 16: Stakeholder: "Say hello to Bob, he is your replacement. Please work with him making sure he has all the context he needs during this transition". Engineering lead: "But, but... it's just technical debt" Stakeholder: "You may leave now"
  - xwdv 6y ago
    Why would the app be down that long though? Just take on more technical debt to get it back up and running. If Bob gets the app up and running then that’s all that matters to the stakeholder.
    - 29athrowaway 6y ago
      It doesn't have to be continuously down. It only has to be down often.
      - yardstick 6y ago
        To the end users and management the two are essentially the same outcome: customers leave your non-functional service.
qwantim1 6y ago
There is no right kind of technical debt. By that I mean write code intentionally.
- stefanmichael 6y ago
  but the sprint ends tomorrow and I'll get a bad performance review if my tickets fall over to the next sprint!!! /s obviously but this is more real than it should be
mtlynch 6y ago
One of the big ah-ha moments I had while reading Start Small, Stay Small by Rob Walling was that most developers have an aversion to carrying technical debt because they've experienced managers that never allow them to pay it back later. But when it's your startup (or small software company), you can choose when to pay back technical debt. It sounds obvious, but I hadn't thought of it that way. Once I realized I had full control of when to pay back tech debt, it made me more comfortable accruing it strategically.
- x0x0 6y ago
  Hey, I thought I recognized your name! I enjoy your book reviews.
  - mtlynch 6y ago
    Oh, cool. Glad that they're useful!
- RangerScience 6y ago
  Job prior to this one was at a raw stage startup (two cofounders, both new college grad). Most of the back-end code base was written by a now-departed first employee (also a new grad), and it was a trash fire. The "tech" founder had a decent front-end code base, where they were very much doing what you describe: I could tell which parts of the code were written when, based on their skill growth, etc etc. As an employee, it wasn't my call when to pay down the back-end's debt. I was very happy to get out of that. Anyway. Yes, it's different when it's your debt, and what makes it your debt is then whether you're the one who decides when to pay it.
  - Kiro 6y ago
    Wasn't the point that as a founder you're more likely to accrue technical debt? Your anecdote sounds like the opposite.
- onion2k 6y ago
  But when it's your startup (or small software company), you can choose when to pay back technical debt. This is true but you will always feel that paying off the debt isn't growing the business or giving any real benefit to the customer. If you can fight that then it works. If you can't then you'll get more and more technical debt.
mobjack 6y ago
You also need to get comfortable taking out tech debt as a startup. A lot of times you can get away with it at little or no cost. I wasted lots of time prematurely paying back tech debt only to have that code later deleted or rewritten. It is important to properly prioritize tech debt and pay back things that the business is currently paying high interest on. Once the product matures it makes sense to be more proactive in doing things right, but in the early stages, things are changing too fast that you just need to get out a rough prototype.
theptip 6y ago
One tactic I have used with moderate success is to explicitly carve out time for fixing tech debt. A few things we’ve tried: 1. Pledge to schedule 20% (pick your percentage) of the sprint as tech debt payback rather than stories. The problem here is that this stuff tends to drift to the bottom of the sprint and then slip if anything else is delayed. This is where most teams start as you can start at an arbitrarily small percentage and scale up gradually. 2. Fixit Friday; the middle Friday of a sprint is for fixing tech debt, not working on sprint tasks. Engineers are encouraged to advocate for tasks they think would speed us up, and we also have done themed pushes where e.g. the whole team chips in to migrate the codebase to a new linter. One issue here is that it’s hard to make progress on larger initiatives, which leads to: 3. Fixit Sprint. At the beginning of the quarter there is often a bit of downtime as the leadership digests the last quarter and plans the new one. This can be a good time to take a sprint off product work, and really dig into some substantial re-tooling or refactoring. Depending on how much resource you’re allocating to tech debt, you can mix and match these, I’d say 1 & 2 are probably mutually exclusive, but you could combine either with 3, or just do 3 instead. I think it’s important to be clear about when you are taking on tech debt, and when you have cleared the hurdle and are paying it back, and clear rules like these help to communicate that message.
- valenterry 6y ago
  I feel when you have to do this, it's already a lost cause. The company either thinks strategically or it doesn't. If it doesn't then it will only hurt you if you do. Just deal with it, work in the way they except (i.e. create even more tech debt for some fast results) and once it all crashes, tell them the whole thing needs a full rewrite (which you know will fail anyways). Not really much you can do here except for leaving. If the company thinks strategically, then no need for something like a 20% schedule. You should be empowered to make these decisions yourself and you should sometimes create a lot of strategical tech debt and sometimes take a month to fix something if you know it will pay off. You are the expert - you need to understand the business for these decisions and then you communicate back (as you already mentioned), but that's about it.
  - JamesBarney 6y ago
ianamartin 6y ago
One technique I've used as a team lead to limit tech debt that makes it into production is to have devs write prototypes in a different language than what we actually support in prod. This has a few really nice advantages. 1. Devs enjoy getting to use new languages in the real world, and helps keep us learning. 2. The better you know a language, the more tempting it is to take shortcuts. When you don't know a language well enough to write really gross things that work "for now", you have to think carefully about the simplest possible solution to a problem that you can express clearly in an unfamiliar language. 3. You learn tons the first time you solve a particular problem. At the end of the solution, when all of your hacks and tradeoffs are fresh in your mind, you are the best possible person to tackle all the shortcomings and immediately do a rewrite in a language you are expert in. 3. Management really can't twist your arm to just go ahead and transition the prototype to MVP. All you can really do is throw it out there as a public beta with no guarantees while you build the MVP properly using everything you just learned from doing it the first. 4. You can start collecting feedback on what users want included in v1 and get an idea of where the user base is heading with their desires and plan some of that into your design, again reducing long-term tech debt. If your prototype sticks the landing well enough for management to decide to move forward to MVP, then it's good enough to mark your territory in the problem space while you do the rewrite. You won't lose competitive advantage while it's sitting out there in a separate VPC collecting users and activity. In a healthy company, this isn't a difficult sell to management. They'll understand the short and long-term value to the company. In more toxic environments, it will look more like you're throwing a poison pill into your dev process—which you kind of are. So, you know, tread carefully. But when people buy in to this process, it works really nicely and has far better long-term outcomes.
- kyawzazaw 6y ago
  How is the runaway to allow such practices?
  - ianamartin 6y ago
    The time consuming aspects of new product dev are normally not around the time it takes to write the code. It's translating product requirements into solvable problems, logic flows that solve those problems, bumping into unseen edge cases in the product concept, and coming to consensus about what compromises are acceptable in the prototype, bumping into sharp edges while you flesh solutions out because you didn't anticipate a thing when you started your base design or didn't understand the relationships between objects and functionality when you kicked the project off. These problems that take the most time in any project are mostly human, conceptual, product, and communication problems. Not code problems. Once you have sorted out all of these problems and solved them, a rewrite in a different language with a better design moves very quickly. You don't have to double the runway or the time to MVP. Time estimations are always wrong anyway. But in my experience when I've gotten buy-in for this approach I would guess it adds 25-30% of actual time overhead to a roadmap. Prototypes always take longer than expected, and an immediate redesign/rewrite takes less time than expected. Selling this approach to leadership really boils down to clearly identifying the value of developer time. It's not code; it's solving business problems. Once those are conceptually identified and solved, the design and code tend to fall into place without a ton of trouble. The problem is that no one really knows what the business problems are until you try to solve them and really dig into the details. Prototypes should be understood as the process of defining the problem space and uncovering all the hidden issues that people haven't really thought through just yet. MVPs should be an actual product based on that exploration and problem definition and solving. What I'm suggesting here is really just a process boundary that reflects the difference between the two things. "Can runway handle that?" is really a lot like asking if you can afford to build a product at all.
galaxyLogic 6y ago
Code is like a performance by musicians. They play together typically based on a plan known as "score". But when they play a piece for the first time the performance is typically much lacking. They have to rehearse the symphony or rock-opera or whatever. Removing technical debt is like playing the song again, this time better. You learn what works what not and can do it better. The challenge is how to know the code-performance is good enough.
kissgyorgy 6y ago
It's very simple; no juniors at the beginning! They can make things work, but they don't have the experience to avoid tech debt even in the short term. I have seen this as a CTO, unfortunately I came too late and the company broke. The terrible management played a bigger part, but the code base was so terrible and the tech debt so big they couldn't ship even very simple features in a timely manner. Testing and release was insanely slow.
- deleted 6y ago
  [deleted]
- Viliam1234 6y ago
  Is the problem actually the presence of juniors, or rather the fact that in "agile" all developers are treated as replaceable, so the juniors randomly get assigned strategically important tasks where they can do long-term damage? I imagine that after a month, when the senior developer(s) laid down the foundations, it should be possible to find work for the juniors (not all at the same time, but adding them gradually). They just cannot be assigned tickets at random.
- x0x0 6y ago
  I can't emphasize this strongly enough: no juniors. Probably for the first 3+ years. They're a disaster, and it's taken my company a year -- with another year or more to go -- to recover. It's not even necessarily their fault; they just need piles of time and mentorship and direction that you can't afford to give yet. Worse, you'll quite possibly end up firing them when you eventually start reigning them in and they don't like being told how to build things. Trust me, it's been an extremely painful experience.
  - cs-szazz 6y ago
    "Worse, you'll quite possibly end up firing them when you eventually start reigning them in and they don't like being told how to build things." Had this exact thing happen at my company. It's tough when you have to start doing this, but even tougher if you don't. There are times I certainly think we'd have the same output with half the people if we'd kept the bar higher.
boffinism 6y ago
The title is good, and contains an often overlooked point: startups need to survive tech debt precisely because they do not need or want to avoid tech debt. Startups generally don't have the resources to do things properly. They don't have the people, the cash, or the time. So they trade on their future value: get cash from investors on the promise of future returns. Pay below-market salaries but promise quick promotions and options packages. Take on tech debt to get to market now, and hope to pay it off later. And, when it works, this is a great strategy. Tech debt is good if you're a startup. But only if you survive it. Personally I've never seen a startup that was actually killed by tech debt, although it does sometimes cause good employees to quit, and of course it causes product delays. It's like any form of debt: a dangerous but oftentimes necessary tool.
Animats 6y ago
Exit before the technical debt comes due. Also, if you're growing fast enough that it matters, you'll need to redesign the whole thing anyway. Google did that on their search engine at least five times. In the early days, it took weeks to update the index.
ulisesrmzroche 6y ago
Every bug fix is paying down technical debt. We don't squash every bug, we fix the most important ones. Otherwise, we'd be swatting bugs all day. If there is a bug in the woods, and the bug takes a shit, but no one is around to hear it... Now, is it going to kill a startup? Hardly. At that point, an MVP, it's probably still small enough that the better decision is to rewrite it altogether. If you split the server and the UI to begin with, you can re-design the front-end and make a party out of it, try to make it pretty this time. If you're at the point where you need more than 1 person to handle the whole app, then you're not a startup anymore, imo. You have a proven small business.
bizzleDawg 6y ago
I've found it helpful consider the type of technical debt too. Get the data models right and most other technical debt is a lot easier to pay down. If you keep adding work on top of bad data models, things get more and more out of hand. I'll always fix data modelling as soon as the abstraction is found to be incorrect, even though that can cause a delay. But, if you have repeated functions, or two similar react components which could be merged, that's a far lower risk to current and future development velocity.
passerby1 6y ago
I wonder how many startups were killed solely due to tech debt repayment. I had this sin too. Not that there were too much of debt, but it's unbalanced wish to repay it and stuck with that was the source of never finished refactoring that took enough time to kill everything.
AntiImperialis4 6y ago
If you're a startup, there is no way you will move fast if you don't incur tech debt. Why is it seen as a bad thing? It should be a given. Once you know that your company is there to stay, you hire experienced people and create a strategy to pay it back.
kgc 6y ago
There's another way: Move fast enough to justify rewrites. This even happens at larger companies like Facebook when they rewrote React. https://engineering.fb.com/2017/09/26/web/react-16-a-look-inside-an-api-compatible-rewrite-of-our-frontend-ui-library/ https://engineering.fb.com/2017/09/26/web/react-16-a-look-in...
varjag 6y ago
Be like America, thrive by incurring debt.
- oblio 6y ago
  https://www.lynalden.com/fraying-petrodollar-system/ https://www.lynalden.com/fraying-petrodollar-system/

136 points by fidrelity 6y ago

I work in a fear-based culture right now. The CTO never saw a kludge he didn’t want into production stat, and in his dual role as CEO won every disagreement. There are many pieces on tech debt that are true and valid, this one IMHO covers more salient issues than most. If you’re reading this comment before you read the article: it’s about how to avoid getting caught too badly in the first place, not about magic bullets to get you out when you’re basically bankrupt.

rendall 6y ago
> I work in a fear-based culture right now That sucks. I have been there. Awful. For me, the silver lining is that I no longer ignore stinky red flags
- minimuffins 6y ago
  Yeah, I think it can be painfully helpful to go through that miserable experience at least once. But my very strong advice to others: Do NOT put up with that. You don't have to. There are so many companies, even now, where you can get decent work with much more sensible norms. There's always kind of a gravitational pull toward excess debt because "get the [fix/feature] out there in front of [users/client/exec]" is such a tempting drug for management, even if they're generally good. But the amount and toxicity of the debt varies widely. Find a company that's sane about this if you value your own sanity imo.
  - quantified 6y ago
    Thank you for the support! Doing my best, and leaving when the right opportunity appears.
- hndudette2 6y ago
  Yep, same, and it hurts the most loyal and hardworking the most. Do what's best for yourself and find a better job.
ianamartin 6y ago
I will never again work in a company with a dual CEO/CTO. No matter how well-intentioned and talented that person is, it always leads to a toxic environment. It's an instant red flag that the person is incapable of delegating responsibilities. Separation of duties at the C-level is important. There needs to be some amount of healthy friction in the leadership team to make appropriate compromises. Hope you are able to get out of this situation soon.

KingOfCoders 6y ago

From my experience with lots of VC money.

s17n 6y ago

"Revenue solves all known problems"

Haydos585x2 6y ago

I enjoyed this article and I think it's helpful for technical leaders in early stage startups. I feel like a lot of these companies will continue having problems mostly because the people that are attracted to smaller/early-early companies are the same kinds of people that don't want to work in structured (more corporate/deliberate) environments. My experience is a bit more on the agency side where technical debt is caused by developers/project managers not really caring about the longevity of an application. The idea of going back and fixing anything that isn't specifically called out and invoiced for is so far away from the priorities of these businesses.

austincheney 6y ago
I suspect the tech debt that many start ups encounter is a matter of execution. They likely require original solutions to tough problems in very short time. The way to reduce tech debt in this case is to make deliberate time at great expense to frequently refactor complexity out of the system. In the corporate world, on the other hand, the problem is an astonishing fear of originality. Most places I have worked have invented here mentality baked into every decision. Every 5 to 8 years they start over completely with a different tool, framework, or language hoping it magically solves for the group’s prior poor implementation.
clumsysmurf 6y ago
> My experience is a bit more on the agency side where technical debt is caused by developers/project managers not really caring about the longevity of an application. The idea of going back and fixing anything that isn't specifically called out and invoiced for is so far away from the priorities of these businesses. This resonates with me. I consider myself a better fit for product focus vs project focus, preferring to find something meaningful and nurture it in terms of value to customer, and value to developers (code quality). Recently I accepted a job at an Agency, which a coworker described as a "chop shop". I became depressed. The app I would work on, got frenzy activity from Oct - March. It had to be released in March no matter what. Between that time there was no activity, no refactoring, no addressing tech debt. In fact, one week in December the client was unable to pay for some technical budgeting reason and we all just sat around for that week despite looming deadline. Well, over the years this thing accumulated tech debt. Even though it was a cash cow for the agency: reliable income at reliable times from a good customer. One would think the health of the product would somehow invite extra effort. Every year new devs would start working on it (new devs, because turnover at these places is high). They would look at the mess and begin resume driven development to use the shiniest new things that were left out, hoping to fix and compensate for the problems from just sheer neglect, and then move on for the cycle to repeat. That said, there are people that just prefer hopping from project to project to experience new things.

k__ 6y ago

How many startups are actually killed by tech debt? I saw many successfull companies with shitty software. I had the impression the business part of things killed much more companies if they didn't get it right.

29athrowaway 6y ago
Tech debt rarely kills a company, but it does kill teams, in descending hierarchical order. It usually goes like this: Day 1 Stakeholder: "The app is down, what is happening?" Engineering lead: "We are working on it. Do not worry". Day 2 Stakeholder: "The app is down, what is happening?" Engineering lead: "Do not worry, we got this". Fast forward to Day 15: Stakeholder: "Can you explain why the app is down again?" Engineering lead: "We got this" Stakeholder: "That is what you said last 2 weeks ago. We will send someone to take a look at it" External auditor: "wtf wtf wtf. These guys do not know what they're doing." Day 16: Stakeholder: "Say hello to Bob, he is your replacement. Please work with him making sure he has all the context he needs during this transition". Engineering lead: "But, but... it's just technical debt" Stakeholder: "You may leave now"
- xwdv 6y ago
  Why would the app be down that long though? Just take on more technical debt to get it back up and running. If Bob gets the app up and running then that’s all that matters to the stakeholder.
  - 29athrowaway 6y ago
    It doesn't have to be continuously down. It only has to be down often.
    - yardstick 6y ago
      To the end users and management the two are essentially the same outcome: customers leave your non-functional service.

qwantim1 6y ago

There is no right kind of technical debt. By that I mean write code intentionally.

stefanmichael 6y ago
but the sprint ends tomorrow and I'll get a bad performance review if my tickets fall over to the next sprint!!! /s obviously but this is more real than it should be

mtlynch 6y ago

One of the big ah-ha moments I had while reading Start Small, Stay Small by Rob Walling was that most developers have an aversion to carrying technical debt because they've experienced managers that never allow them to pay it back later. But when it's your startup (or small software company), you can choose when to pay back technical debt. It sounds obvious, but I hadn't thought of it that way. Once I realized I had full control of when to pay back tech debt, it made me more comfortable accruing it strategically.

x0x0 6y ago
Hey, I thought I recognized your name! I enjoy your book reviews.
- mtlynch 6y ago
  Oh, cool. Glad that they're useful!
RangerScience 6y ago
Job prior to this one was at a raw stage startup (two cofounders, both new college grad). Most of the back-end code base was written by a now-departed first employee (also a new grad), and it was a trash fire. The "tech" founder had a decent front-end code base, where they were very much doing what you describe: I could tell which parts of the code were written when, based on their skill growth, etc etc. As an employee, it wasn't my call when to pay down the back-end's debt. I was very happy to get out of that. Anyway. Yes, it's different when it's your debt, and what makes it your debt is then whether you're the one who decides when to pay it.
- Kiro 6y ago
  Wasn't the point that as a founder you're more likely to accrue technical debt? Your anecdote sounds like the opposite.
onion2k 6y ago
But when it's your startup (or small software company), you can choose when to pay back technical debt. This is true but you will always feel that paying off the debt isn't growing the business or giving any real benefit to the customer. If you can fight that then it works. If you can't then you'll get more and more technical debt.

mobjack 6y ago

You also need to get comfortable taking out tech debt as a startup. A lot of times you can get away with it at little or no cost. I wasted lots of time prematurely paying back tech debt only to have that code later deleted or rewritten. It is important to properly prioritize tech debt and pay back things that the business is currently paying high interest on. Once the product matures it makes sense to be more proactive in doing things right, but in the early stages, things are changing too fast that you just need to get out a rough prototype.

theptip 6y ago

One tactic I have used with moderate success is to explicitly carve out time for fixing tech debt. A few things we’ve tried: 1. Pledge to schedule 20% (pick your percentage) of the sprint as tech debt payback rather than stories. The problem here is that this stuff tends to drift to the bottom of the sprint and then slip if anything else is delayed. This is where most teams start as you can start at an arbitrarily small percentage and scale up gradually. 2. Fixit Friday; the middle Friday of a sprint is for fixing tech debt, not working on sprint tasks. Engineers are encouraged to advocate for tasks they think would speed us up, and we also have done themed pushes where e.g. the whole team chips in to migrate the codebase to a new linter. One issue here is that it’s hard to make progress on larger initiatives, which leads to: 3. Fixit Sprint. At the beginning of the quarter there is often a bit of downtime as the leadership digests the last quarter and plans the new one. This can be a good time to take a sprint off product work, and really dig into some substantial re-tooling or refactoring. Depending on how much resource you’re allocating to tech debt, you can mix and match these, I’d say 1 & 2 are probably mutually exclusive, but you could combine either with 3, or just do 3 instead. I think it’s important to be clear about when you are taking on tech debt, and when you have cleared the hurdle and are paying it back, and clear rules like these help to communicate that message.

valenterry 6y ago
I feel when you have to do this, it's already a lost cause. The company either thinks strategically or it doesn't. If it doesn't then it will only hurt you if you do. Just deal with it, work in the way they except (i.e. create even more tech debt for some fast results) and once it all crashes, tell them the whole thing needs a full rewrite (which you know will fail anyways). Not really much you can do here except for leaving. If the company thinks strategically, then no need for something like a 20% schedule. You should be empowered to make these decisions yourself and you should sometimes create a lot of strategical tech debt and sometimes take a month to fix something if you know it will pay off. You are the expert - you need to understand the business for these decisions and then you communicate back (as you already mentioned), but that's about it.
- JamesBarney 6y ago

ianamartin 6y ago

One technique I've used as a team lead to limit tech debt that makes it into production is to have devs write prototypes in a different language than what we actually support in prod. This has a few really nice advantages. 1. Devs enjoy getting to use new languages in the real world, and helps keep us learning. 2. The better you know a language, the more tempting it is to take shortcuts. When you don't know a language well enough to write really gross things that work "for now", you have to think carefully about the simplest possible solution to a problem that you can express clearly in an unfamiliar language. 3. You learn tons the first time you solve a particular problem. At the end of the solution, when all of your hacks and tradeoffs are fresh in your mind, you are the best possible person to tackle all the shortcomings and immediately do a rewrite in a language you are expert in. 3. Management really can't twist your arm to just go ahead and transition the prototype to MVP. All you can really do is throw it out there as a public beta with no guarantees while you build the MVP properly using everything you just learned from doing it the first. 4. You can start collecting feedback on what users want included in v1 and get an idea of where the user base is heading with their desires and plan some of that into your design, again reducing long-term tech debt. If your prototype sticks the landing well enough for management to decide to move forward to MVP, then it's good enough to mark your territory in the problem space while you do the rewrite. You won't lose competitive advantage while it's sitting out there in a separate VPC collecting users and activity. In a healthy company, this isn't a difficult sell to management. They'll understand the short and long-term value to the company. In more toxic environments, it will look more like you're throwing a poison pill into your dev process—which you kind of are. So, you know, tread carefully. But when people buy in to this process, it works really nicely and has far better long-term outcomes.

kyawzazaw 6y ago
How is the runaway to allow such practices?
- ianamartin 6y ago
  The time consuming aspects of new product dev are normally not around the time it takes to write the code. It's translating product requirements into solvable problems, logic flows that solve those problems, bumping into unseen edge cases in the product concept, and coming to consensus about what compromises are acceptable in the prototype, bumping into sharp edges while you flesh solutions out because you didn't anticipate a thing when you started your base design or didn't understand the relationships between objects and functionality when you kicked the project off. These problems that take the most time in any project are mostly human, conceptual, product, and communication problems. Not code problems. Once you have sorted out all of these problems and solved them, a rewrite in a different language with a better design moves very quickly. You don't have to double the runway or the time to MVP. Time estimations are always wrong anyway. But in my experience when I've gotten buy-in for this approach I would guess it adds 25-30% of actual time overhead to a roadmap. Prototypes always take longer than expected, and an immediate redesign/rewrite takes less time than expected. Selling this approach to leadership really boils down to clearly identifying the value of developer time. It's not code; it's solving business problems. Once those are conceptually identified and solved, the design and code tend to fall into place without a ton of trouble. The problem is that no one really knows what the business problems are until you try to solve them and really dig into the details. Prototypes should be understood as the process of defining the problem space and uncovering all the hidden issues that people haven't really thought through just yet. MVPs should be an actual product based on that exploration and problem definition and solving. What I'm suggesting here is really just a process boundary that reflects the difference between the two things. "Can runway handle that?" is really a lot like asking if you can afford to build a product at all.

galaxyLogic 6y ago

Code is like a performance by musicians. They play together typically based on a plan known as "score". But when they play a piece for the first time the performance is typically much lacking. They have to rehearse the symphony or rock-opera or whatever. Removing technical debt is like playing the song again, this time better. You learn what works what not and can do it better. The challenge is how to know the code-performance is good enough.

kissgyorgy 6y ago

It's very simple; no juniors at the beginning! They can make things work, but they don't have the experience to avoid tech debt even in the short term. I have seen this as a CTO, unfortunately I came too late and the company broke. The terrible management played a bigger part, but the code base was so terrible and the tech debt so big they couldn't ship even very simple features in a timely manner. Testing and release was insanely slow.

deleted 6y ago
[deleted]
Viliam1234 6y ago
Is the problem actually the presence of juniors, or rather the fact that in "agile" all developers are treated as replaceable, so the juniors randomly get assigned strategically important tasks where they can do long-term damage? I imagine that after a month, when the senior developer(s) laid down the foundations, it should be possible to find work for the juniors (not all at the same time, but adding them gradually). They just cannot be assigned tickets at random.
x0x0 6y ago
I can't emphasize this strongly enough: no juniors. Probably for the first 3+ years. They're a disaster, and it's taken my company a year -- with another year or more to go -- to recover. It's not even necessarily their fault; they just need piles of time and mentorship and direction that you can't afford to give yet. Worse, you'll quite possibly end up firing them when you eventually start reigning them in and they don't like being told how to build things. Trust me, it's been an extremely painful experience.
- cs-szazz 6y ago
  "Worse, you'll quite possibly end up firing them when you eventually start reigning them in and they don't like being told how to build things." Had this exact thing happen at my company. It's tough when you have to start doing this, but even tougher if you don't. There are times I certainly think we'd have the same output with half the people if we'd kept the bar higher.

boffinism 6y ago

The title is good, and contains an often overlooked point: startups need to survive tech debt precisely because they do not need or want to avoid tech debt. Startups generally don't have the resources to do things properly. They don't have the people, the cash, or the time. So they trade on their future value: get cash from investors on the promise of future returns. Pay below-market salaries but promise quick promotions and options packages. Take on tech debt to get to market now, and hope to pay it off later. And, when it works, this is a great strategy. Tech debt is good if you're a startup. But only if you survive it. Personally I've never seen a startup that was actually killed by tech debt, although it does sometimes cause good employees to quit, and of course it causes product delays. It's like any form of debt: a dangerous but oftentimes necessary tool.

Animats 6y ago

Exit before the technical debt comes due. Also, if you're growing fast enough that it matters, you'll need to redesign the whole thing anyway. Google did that on their search engine at least five times. In the early days, it took weeks to update the index.

ulisesrmzroche 6y ago

Every bug fix is paying down technical debt. We don't squash every bug, we fix the most important ones. Otherwise, we'd be swatting bugs all day. If there is a bug in the woods, and the bug takes a shit, but no one is around to hear it... Now, is it going to kill a startup? Hardly. At that point, an MVP, it's probably still small enough that the better decision is to rewrite it altogether. If you split the server and the UI to begin with, you can re-design the front-end and make a party out of it, try to make it pretty this time. If you're at the point where you need more than 1 person to handle the whole app, then you're not a startup anymore, imo. You have a proven small business.

bizzleDawg 6y ago

I've found it helpful consider the type of technical debt too. Get the data models right and most other technical debt is a lot easier to pay down. If you keep adding work on top of bad data models, things get more and more out of hand. I'll always fix data modelling as soon as the abstraction is found to be incorrect, even though that can cause a delay. But, if you have repeated functions, or two similar react components which could be merged, that's a far lower risk to current and future development velocity.

passerby1 6y ago

I wonder how many startups were killed solely due to tech debt repayment. I had this sin too. Not that there were too much of debt, but it's unbalanced wish to repay it and stuck with that was the source of never finished refactoring that took enough time to kill everything.

AntiImperialis4 6y ago

If you're a startup, there is no way you will move fast if you don't incur tech debt. Why is it seen as a bad thing? It should be a given. Once you know that your company is there to stay, you hire experienced people and create a strategy to pay it back.

kgc 6y ago

There's another way: Move fast enough to justify rewrites. This even happens at larger companies like Facebook when they rewrote React. https://engineering.fb.com/2017/09/26/web/react-16-a-look-inside-an-api-compatible-rewrite-of-our-frontend-ui-library/ https://engineering.fb.com/2017/09/26/web/react-16-a-look-in...

varjag 6y ago

Be like America, thrive by incurring debt.

oblio 6y ago
https://www.lynalden.com/fraying-petrodollar-system/ https://www.lynalden.com/fraying-petrodollar-system/