6 ms·
We're dealing with accumulated tech debt burying us right now. For years the team was told to deliver features, that the upgrades to node dependencies weren't
by bloopernova 3y ago
We're dealing with accumulated tech debt burying us right now.
For years the team was told to deliver features, that the upgrades to node dependencies weren't of a high enough priority. Now some of the dependencies are so old they're complicating upgrading past node 12.
Yet the cybersecurity people are demanding that vulnerabilities be fixed.
Thankfully we have a stronger product owner now, and spent most of the last 6 months getting things updated. Yet there are still those in senior leadership who openly question why our team hasn't delivered the features they want.
We're working on educating them. And something pithy and catchy is useful, so I'm definitely going to mention this in future: "fixing tech debt has a price, not fixing it has a cost."
It's the kind of thing that if you can get a "cool" or influential senior leader to repeat, the rest will fall in line.
- preommr 3y ago> It's the kind of thing that if you can get a "cool" or influential senior leader to repeat, the rest will fall in line. sympathetic-lol at the desperate copium.
- bloopernova 3y agoIf I am honest with myself, it's about 40% copium and 60% experience with the directors in question. There's 2 "cool" directors who were tech people but are also influential in that peer group. You can't trust them to represent you fairly, but you can trust that they'll listen to your argument. The 40% copium stems from knowing that some of the "only features have business value" group are stubborn and think everyone should just work more.
- earnesti 3y agoI was working for a company, where the engineering team was doing this kind of refactoring, test development and what not mainteinance coding with large team for about 2 years. Meanwhile not many new business features were being developed and the competition took over, and eventually everyone got fired. I think the leadership was just too lazy to actually push the development of the product, but let the engineers do this pussy work of endless refactoring was big part of the problem. Not only one of course.
- jacquesm 3y agoThat's exactly it: you need balance. Not so much focus on tech for tech's sake that the competition gets to bury you but also not so little that you end up burying yourself.
- aero142 3y agoWhen senior leadership is non-technical, they can't tell when non-customer visible changes are useful or they just have mediocre engineering. They also can't tell when their software stack is such a mess it's beyond fixing.
- blackoil 3y agoThat's the most important job of a CTO. Communicate the cost and implications of any development done by tech team. This could be feature required by product/sales/CEO or internally by tech team.
- bloopernova 3y agoThat's a great point! We don't have commercial competitors because we're an inhouse project, but there's still a balance to be made, you're right.
- IshKebab 3y agoIMO the best way is just to spend part of your time refactoring and part of your time adding features. It basically solves all the problems. You still deliver features so your company hopefully doesn't go under. And you can reduce technical debt without having to awkwardly justify not delivering features. In my experience in the medium term (~2-5 years) it doesn't even slow down feature delivery since high technical debt can have an absolutely catastrophic effect on development speed. At the previous company I worked in I'd say we delivered work at about 50% of the potential rate because the infrastructure they'd developed was so janky. A big Python/YAML/Makefile mess with zero static types. CI time was over 2 hours and used hundreds of CPU-hours, even for fixing a typo.
- rr808 3y agoNode 12? Holy s** that is only 4 years old. Meanwhile in the real world most applications are running on 10+ year old stacks - Java 8, Python 2, Angular.js etc etc
- thfuran 3y agoWe just last month upgraded away from a dependency that seems to have been abandoned since about 2008.
- wiseowise 3y ago> Meanwhile in the real world most applications are running on 10+ year old stacks - Java 8, Python 2, Angular.js etc etc Source: trust me, bro
- corethree 3y agoThe sky is blue. Source: trust me, bro The earth is round. Source: trust me, bro
- Scarblac 3y agoWe retired our last Angular.js app literally last week, and have a project ongoing to replace the final Python 2 backend. And someone seriously suggested in a meeting last week to keep part of it around...
- AllegedAlec 3y agoWe're running .net 4.5.2 Source: trust me, bro
- ikekkdcjkfke 3y agoMany babies were thrown out with .net core though
- AllegedAlec 3y agoWe develop for systems that at of this time are still stuck on XP/Vista. We'd love to at least be able to progress to 4.8.1 but as of now we can't.
- tomrod 3y ago> Yet there are still those in senior leadership who openly question why our team hasn't delivered the features they want. Is this _really_ so hard for stakeholders to understand? I've found some limited success by dual-boating -- upgrading as features are created. "We can't build this feature without an upgrade" carries a lot of weight -- but sometimes you just have to bite the bullet and demand an upgrade. Are stakeholders completely oblivious to the existential threat to their bonus if their company is hacked and driven to bankruptcy in lawsuits? Sure some bigs can avoid this, but not the mids and smalls.
- tessierashpool 3y agoIs this _really_ so hard for stakeholders to understand? for decades now, every generation of programmers has been larger than the previous one. every generation of engineering managers has probably grown in a proportionate way. so, for decades now, every generation of engineering managers has been learning the same basic "how software differs from manufacturing" lessons over and over again. (partly because business schools don't seem to have learned this lesson yet even once.)
- TeMPOraL 3y agoWait, is this the "amount of programmers in the industry doubles every ${not many} years, therefore software development is dominated by fresh juniors who know fuck all about anything" argument, but applied to engineering management? Can't find the sources for the OG argument about programmers, but it does sound plausible.
- ozim 3y agoSorry but do you have an example of a company that got hacked and driven to bankruptcy by lawsuits? Most of what people see in open are companies that get hacked, share price dips for a month or two if it is software company, if it is non software company share price does not even dip. Okta went down quite a lot since 2022 but still far away from bankrupt and latest "hack" did not move it that much really. Where in my opinion (as a tech person) they should be out of business already even after slightest "hack". Microsoft Azure latest hack did not really affect their stock...
- n0us 3y agoIt’s okay we can’t get past Node 8
- wdfx 3y agoIk going to switch off our last node 6 instances in a few weeks time.
- godelski 3y ago> why our team hasn't delivered the features they want. Good work takes time. But I think it is an optimization problem. Rather, that different people are optimizing different things. Your team is optimizing long term profits. The senior leadership who's frustrated is optimizing short term profits. I think one way to handle these situations is to clarify which is the more important optimization problem. That makes points like "opportunity costs vs marginal cost" discussions. The same way we turn down immediate profits to go to school, because in the long run we make more money (and for other reasons). But our systems often push us to focus on short term goals because these are far easier to measure.
- arcbyte 3y agoI agree clarity is needed but i would caution that it is not ethical to let executives that get to make that determination. I don't think we as engineers can ethically ever let management optimize our work for the short term. They will ALWAYS want the short term improvement and by even giving them that decision we abdicate our own judgement. That abdication is totally unethical.
- kstrauser 3y ago> Yet the cybersecurity people are demanding that vulnerabilities be fixed. Talk to them. They may have your back. It was always a happy day when the security audit we were going through at that moment wanted us to attest that all our dependencies were sufficiently updated. That meant we had an external mandate to do the work we'd been wanting to do for ages anyway. When I was a CISO, I would've been glad to "order" one of my colleagues to do that if they asked me to.
- gardenhedge 3y agoA bottom up approach to tech this way obviously does not work long term. Startups will eat your lunch.
- Tainnor 3y agoNode 12? That's cute. We have an application that runs on Java 6.
- MrStonedOne 3y agoThe issue isn't tech debt, its tech interest. Looking at the historical maintenance cost of the stacks when choosing them and prioritizing ones with a better history of backwards and forwards compatibility in their ecosystem does far more to prevent tech debt pile up than trying to reach a strict habit of bending over backwards to update dependencies.
- kibibu 3y agoThe "debt" metaphor is a good one already though, and one that senior leaders should well understand.