5 ms·
> Engineering wants to work off tech debt, refine, and innovate. I know I'm not the only one who is backing away slowly from the concept of tech debt, and I'm
by hinkley 5y ago
> Engineering wants to work off tech debt, refine, and innovate.
I know I'm not the only one who is backing away slowly from the concept of tech debt, and I'm not the only one trying to drag other people with me.
Tech debt has turned into hippie-dippie bullshit in the minds of many people. It was an attempt at enlightened self interest by appealing to the bean counters, but if you're talking to bean counters you've already lost.
The developers want easy things to be easy so they don't have to spend half their time slogging through code that does everything the wrong way, half their time explaining why everything takes so long (heading toward infinity), and half their time implementing new functionality. And yes that's 3/2 which is why we have people doing 60 hour work weeks.
A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear. I don't know if I agree entirely with that characterization, but I definitely think that it's less wrong.
- pgwhalen 5y ago> A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear. It sounds like you agree with the idea in general, but just don't like the current metaphor. Can you expand on what about this alternate metaphor fits the phenomenon better?
- hallway_monitor 5y agoI can. Even if you design a good system up front, the requirements graph changes over time until your great ideas have turned into hindrances. This can be scaling issues, but mostly it's where reality turned out to be different than your abstractions were built for. Maybe you made one module very generic but now it's cumbersome to add features to. Maybe you didn't add enough flexibility in another place which resulted in a buildup of suboptimal code. Maybe extreme concurrency was planned for but never needed and now the person has left and no one else understands how that part of the system works. So now, over time, this emergent phenomenon that some call code rot causes your beautiful, elegant greenfield project to turn into a monster that no one wants to touch. It's hard to modify, easy to break, and takes forever to work on. It's legacy code.
- dvtrn 5y agoI've found myself holding a bit of chagrin at the term 'technical debt' lately as well, mostly because it feels like saying it, pointing to it, quantifying it, and shoving 'it' under people's noses and proving its there (not to mention the usual ceremonies of what I'd like to do about it) isn't doing me any favors anymore. And at that point.... ....I don't know that simply calling it something else is going to help. Yes I'm a bit jaded, grab a drink, next one's on me
- laichzeit0 5y agoBegin by defining technical debt. Write down an actual definition for the phrase. Once you have a working definition that everyone agrees on you will find all your problems with the term will disappear. (Hint: good luck finding an agreed on definition).
- user3939382 5y agoMy analysis is that the problem is "tech debt is bad". Just like real debt, there are times when you sign your name to it on purpose because it's a good decision. For example, early on in a project when the requirements are fuzzy and undiscovered. Like financial debt, you pay interest on tech debt, so you do want to strategically pay it down when the stars align, maybe you've recently put out some new features, and you can buy the political capital with customers (and by extension, management). Debt isn't a bad analogy but an appreciation for the nuance would be nice, I don't commonly see that.
- shkkmo 5y agoTechnical debt is unavoidable, but letting technical debt grow without managing it is a great way to kill a project. Explaining the benefits of paying down particlar pieces of technical debt is important, but another important piece of the puzzle is talking in advance with stakeholders about when, where and how much technical debt you want to take on to launch a product. When the existence of technical debt isn't talked about until it needs to be repaid and thus comes as a surprise, the conversation becomes a lot harder. Additionally many of the decisions about how much technical debt to take on to launch something can be improved by bringing stakeholders directly into the conversation, at least on a general level.
- hinkley 4y agoThere’s a whiff of victim blaming here that bothers me. When taking on tech debt you often get a lot of pillow talk. We have great reasons to do the wrong thing for the right reasons, but that is all forgotten after the crisis is over. Either immediately for those really bad at it, or when the next crisis hits. I do agree though that when neither transparency nor awareness are good, there will be several crises in the middle that are triggered by the tech debt. I’ve seen that play out a couple of times, typically when I’ve joined a project in progress, and still building trust before I can start pointing out brutal facts. Some developers are blind to different forms of tech debt. And as long as there is dissenting opinion the business will doubt any information that suggests they can’t have their cake and eat it too. I’ve known a handful of jaded engineers at least who, instead of being afraid or awed by titles, end up treating managers like willful children. It… does not yield consistently good results, but then again I don’t know anyone or any process that does. I’ve seen worse, and I’ve seen better.
- goopthink 5y agoOP here. We had a lot of discussion about where Tech Debt falls on the spectrum of projects to work on. That merits its own post. But the gist of it is that resolving tech debt has value. The challenge is that most engineering organizations are awful at describing and communicating that value. For example: "If we refactor this code and migrate to a new server, we'll be able to deploy in 15 minutes instead of 6 hours, and we'll be able to ship features faster to customers with lower risk. This means that we'll get back ~90h of engineering time per week (6 hours x 5 people x 3 teams), and our defect rate will go down by 50% (We see 10% of customers churn due to preventable defects).. Therefore, there is $702,000 savings in engineering efficiency (more time available) and would decrease churn by 10%, resulting in [X] more value retained." Once our engineering team started thinking about the business case for why tech debt should be prioritized, they started to make really compelling arguments for why something needed to be worked on, and we could evaluate it against explicit business/feature requests. This value rolled up into the form of decreasing costs or increasing revenue. Again: resolving tech debt has value, and the engineering leadership needs to work on figuring out what that value is and how it stacks up against all of the other valuable work that the company wants to do to grow. Engineering leaders who don't understand this often struggle at the executive level and ultimately fail the teams they represent.
- yen223 5y agoThe problem I think is engineering haven't figured out good ways to measure software productivity, and so it's very hard to make the kinds of estimates that we need to properly justify work that otherwise delivers zero immediate business value. (I also suspect that not all refactoring work is equal, and that most refactoring work don't result in any positive engineering or business improvements.)
- opportune 5y agoTech debt is too general a term to be useful. A lot of what people call tech debt is just working code that they don’t understand (maybe because all of the people that did understand it are now gone). Or it’s complex, which may just be because it’s dealing with a complex issue and accumulated lots of bug fixes over time. Or tech debt is just nitpicky design choices which may very well be not-great but aren’t actually causing any problems. None of that is worth investing in beyond a fix-as-you-go, in my opinion, due to there usually being problems to solve with much greater business impact. On the other hand, you have tech debt like: a complex set of dependencies/services with poor interfaces that makes what should be simple changes very complex and risky. Something implemented so inefficiently that it’s slow enough to impact UX or wastes a lot of computing resources ($$$). Or code that is just so, so bad that making any change is like walking in a minefield - and while a lot of engineers are way too quick to put code in this bucket, I do think it’s possible (eg a complete ball of mud application with tons of global state, very poorly applied concurrency patterns, extreme lack of parameterization in favor of copy-paste). That said, I have often seen “tech debt” used as a rationalization to work on some nitpick with no business impact and without a clear improvement. In my opinion, this is partially because working on such tech debt can be very easy: it doesn’t require understanding more complex business requirements, it might be very easy to test with existing integration tests, it has little risk to production, and it doesn’t have task/planning dependencies with other people.
- 015a 5y agoHere's the stance I take: there are many reasons why something might be labeled tech debt. However; assuming the remediation of that debt is implemented in good faith, by reasonable skilled engineers; there is near-always business value in remediating it. It may not always be the same $/hr return on investment, but there is usually positive business value. Secondary point: there is "near-always" positive business value; near-always; and its actually impossible to predict if a given tech debt mitigation project will result in negative business value. You can only know in retrospect. An example you give and label as bad: some crusty old system, outdated language, legacy, everyone who knows it is gone. But, it works. Mitigation: rewrite it in something more modern that everyone on the team knows. Do that, and out of thin air your organization just minted a handful of new Experts on the system. They're forced to dive into the old code. Document the new stuff. Projects like this can be pretty fun! Counter-point: The old system works. Why change it? We just need engineers to dive in, learn it, keep the old crusty libraries and frameworks up to date, lets hire and plan for that. And people fuckin wonder why engineering retention is so bad! "No one who works here knows how it works" Huh I wonder why they left! Let's pass all our problems onto the new generation; they don't get to own anything, their job is to fix shit and bolt on new functionality at a tenth the rate they feel should be possible, enjoy being an indefinite renter of the system until you get as fed up as the generation prior, move companies, and do it again. Business value? You're providing cool projects to your employees. That's it. We'll make up reasons it benefits customers. "We'll be in a better position to scale and reduce API latencies" bs. "Feature velocity will be higher" maybe. Throw out some more tactile reasons like it'll ease hiring because when people hear PHP they run for the hills. Yeah that's more correct, but the biggest reason is expert retention. Have you got experts? If yes: don't rewrite. If no: maybe rewrite. Let engineers rewrite, they feel ownership and responsibility over the system, they probably stay longer. That's it. "Complexity" is similar. Its complex because the business problem is complex. Sure! Engineers just need to dive in and learn it. When? Is the organization willing to literally block off time for the engineers to do this? Lol. Has anyone ever worked at an org that did this? It could work! But what they may block off time for is: rewriting it to make it less complex. The end result probably won't be less complex. But you'll have gained an SME! And the organization can feel good perceiving the problem area as "now less complex" (whereas in reality all you did was train up an SME via way of a refactor). When does it go wrong? Obviously; if a legacy system works, or solves a complex problem, and the rewrite doesn't quite capture what it does. The old code is right there; that's the spec. If the new doesn't match the old spec, it probably comes down to someone not fully understanding the old code. Guess what: that was gonna cause a problem in the future. You just expedited and uncovered a potential problem; but in a context where it can usually be remediated significantly quicker because you're building off a new system with tons of experts, rather than an old system no one likes working in. That's business value! And sure; there are much rarer cases where the entire project goes way over time, the new system sucks, gets caught in development hell, people leave mid-project... that sucks. It can happen. But I'll reiterate: its literally impossible to know with any reasonable certainty that this will happen before taking it on. This means its an extremely poor reason to not to do it. To be clear: I am not asserting that addressing tech debt should always be a priority. I'm asserting that there is near-always positive business value, and that People are extremely bad at estimating what that positive business value will be because it doesn't have a dollar-sign attached to it like CS and Sales orgs have.
- gopher_space 5y agoI've begun to see tech debt as business processes that are too expensive to replace. You can either encapsulate the process with modern thinking or sell your company to someone who has investors willing to pay to create new processes. Just swapping tooling shouldn't be considered tech debt if you can actually do it without breaking income.
- nouveaux 5y agoI am using a low code/no code tool for some back office services and we're investing into the enterprise version. The enterprise version is on-prem only. To migrate from cloud to on-prem, we have to export the cloud data to the on-prem service. Each "section" has a download to json button but there is no download all button. So we have to go to all 50+ sections, click download, upload to our on-prem server, and click in a few places to "reset" things. The fact that these guys cannot build a migration button smells like major technical debt. These guys aren't stupid since they have built an incredible tool. Somewhere along the line, their product became fragmented enough, where it would be too costly to build a feature that would lower the barrier to entry for their enterprise service. We're talking about high 5 figure to a low 6 figure yearly contract. They're telling us we need to go click for an hour to do the migration. So if you don't think this is technical debt, I'm curious as to your interpretation of the situation.
- cebert 5y agoExcellent point. If you have no other option than to do this manually, it sounds like a great use case to automate this problem with playwright.dev.
- hinkley 4y agoThat seems like self sabotage to me. Unless their on-prem solution is less expensive, wouldn’t they make more money if the process were simpler? Some software tasks are tedious and developers try to avoid doing them. But externalizing them to your customers is not a great strategy.
- jacobsenscott 5y agoTech debt and refactoring are the two most overloaded terms in software development. Most often "tech debit" isn't - it is code that met the need when it was written, but the need changed later. Wear and tear seems much more appropriate. And re-writing from scratch isn't refactoring, but that seems to be the most frequent usage of term these days.
- XorNot 5y agoThe counter-point here is that while you can argue about the name, there is a very real issue that poor architecture will slow progress not because it's "hard" to implement features, but because organizationally it becomes impossible to get them merged. We have an endless parade of commentary on HN about what and how much should be in commits and pull requests and what have you, but the thing I've always found credulous about these ideas is how they survive contact with a code base where a "small" feature is going to have a very wide-ranging pattern of changes necessary to support it. It doesn't take much before their are features you functionally can't implement, because there's no possible way to build a neat set of commits for them and get that through reviewers - particularly if there's a lot of activity, because the immediate effect is usually that you'll break a lot of other minor patches with the changes (or they keep breaking you). I'm not sure though that I've seen anyone actually prioritize fixing technical debt in these terms though.
- danjac 5y agoThe problem with "tech debt" is that in the heads of non-developers it is equated with financial debt. When you take on financial debt for a business it's a long-term investment (a buy-to-rent mortgage on a property, or a business loan to buy a factory plant for example). So OK, we're a startup that needs to find the right market, so we'll take on X amount of tech debt to get there, and then when we are in a better place we'll pay down the tech debt in installments of X amount per month. Mostly what I've seen of "tech debt" has not been this kind of deliberate decision making of tradeoffs. It's more like: 1. Non-tech founders want to start a tech company. 2. Seeing the high salaries of developers they opt to hire their college roommate or relative. 3. Friend/relative is horribly inexperienced and builds a pile of crap. 4. Said pile of crap nevertheless gets market traction. 5. Customers complain of endless bugs & sales want long list of features. 6. Friend/relative is promoted to CTO. Time to hire experienced devs! 7. Experienced devs say "this is a pile of crap, throw it away and start over". 8. Founders tell devs no, you have to fix bugs and add features. Hop to it. 9. Developers have no leeway or political capital to fix the underlying problems, do what they can, and eventually leave. So it's more akin to an 18 year old with a gambling habit who has just discovered payday loans.
- hinkley 4y agoAt some point you have to decide if integrity means knowing the right thing to do and doing it, or knowing the right thing to do and asking permission to do it. If you can’t get your coworkers on board, then they can make you look bad faster than the project will reach an intervention. If you can, you can stop having abstract conversations with managers about being Good and just start telling them that this is how long things take now. Relentless refactoring is a set of skills that are interesting but difficult enough that people lose the plot. It’s really about amortizing all difficult work across all of the work, not just the difficult bits. This is actually a strategy from real-time systems, where maximum reaction time matters, and something that XP knew and Scrum does not. A thing I’ve noticed is that there’s often a lot of gaming by management once the engineers start to say “no” but haven’t mastered yet. The bargaining is expensive in time, energy, and good will (once people realize they’re trying to trick us into doing something we said was hard by going at it sideways). When you add a day, two days, or week to all estimates, then the ratio between the smallest and largest stories decreases. The incentive to waste time trying to trick people into saying yes to something that violates the project triangle is drastically reduced. They won’t like it, but they weren’t liking anything else we tried to tell them so that’s not really a new situation. What’s new is that they see we’re serious and they don’t like it.