4 ms·

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscrib

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ controllers, 400+ libraries, LOC around 60k), that sits at the heart of everything we do. The company is struggling to grow and believes the cause is that engineering is slow. We have asked to refactor this code base multiple times, and point to the technical debt as the cause features that should take a day to implement taking between 3-4 weeks, typically. It is only recently that the penny has finally dropped and they've realised if they don't invest in replacing this thing (there is too much technical debt to fix, we're calling bankruptcy and moving to a brand new architecture piecemeal), the business is likely to fail within 1-2 years. That means my current employer is likely to go bust because of technical debt within 2 years max unless we become really good at fixing this. We are optimistic. We have to be, right?

luhn 10y ago
Can you expand on how technical debt caused a two-week outage?
- TheCoreh 10y ago
  If I had to guess, I'd say it's probably an issue with the build/deploy system. Perhaps someone deployed a broken build, then tried to revert/rollback, and realized that the previous version didn't build "cleanly" anymore. This could happen if you have a lot of dependencies, switched compiler versions but left the binaries "in place" and deployed changes incrementally.
  - RangerScience 10y ago
    Yet another reason I swear by Heroku - you can rollback to the actual prior release, not rollback the code and try to rebuild.
    - ardivekar 10y ago
      Doesn't AWS do this too?
      - luhn 10y ago
        Depends on which of their 150 services you're talking about. You can't do it on CodeDeploy.
      - RangerScience 10y ago
        Which service? EC2 doesn't - at least not inherently, there might be a way to snapshot the machine's state; I'm not sure.
- PaulRobinson 10y ago
  There are two problems with giving a full answer: firstly I'm still under NDA for some of that, and too much detail would breach that; secondly, my exact memory of it is limited. In short, my predecessor had attempted a move to SOA without understanding dependencies, circuit breaking and failure modes. This would then cause scenarios where the entire front-end would fail to render on a single down-stream service taking a little longer than necessary. When identifying how to stop that happening, I discovered a large number of comments tagged "TODO" with statements like "Refactor this when we have time" or "We need to find a way to do this better". Further down on the downstream services there were rather esoteric SQL queries doing large joins that nobody had done a query plan on. It was hard to identify these because the ORM had been trusted to do magic, and it was happy to do so, but there was a point where it was not apparent _why_ these joins were happening, but when you found the code, there were more comments "This needs improving", "We should refactor this", etc. We were able to get something back quite quickly with liberal application of indexes, and it took us a day or two to refactor the queries enough to mean response times came down, but the error rate was still > 20%, and it was random, so 1-in-5 page loads of the front end service would fail. We refactored the code to circuit break and handle degraded services better, but that took a few days, and then we started working down to the back end service and figuring out the final steps. It was a small team looking after legacy code that everybody knew was a bit messy. A few weeks before this code was shuttered, I heard from a friend that some of our content did not render at all on certain Android devices. I identified the cause as a half-finished refactor (again, my predecessor), that had never been finished because he had been pushed to work on something else. This caused a dramatic decline within a key market segment that resulted in declining ad revenue, subscriptions and overall viability of the business. Basically, when you start something, finish it. If you find yourself putting in comments like "We should refactor this" anywhere in your code base, and you're doing so because the business is pushing you to work on new features, you have a massive problem culturally that is going to cause a rise in technical debt that raises risk to revenue. All technical debt ultimately will lead to problems that the business will see on balance sheets, but they will rarely successfully identify the cause as being technical debt because they can't see, understand or rationalise it. They think it's engineers being grumpy idealists. People play too fast and loose with the concept of "MVP" for my tastes, and it's a problem I see over and over again. The risk of that is, long-term, it will cause business failure.
72deluxe 10y ago
I would say that 8 years to write 60k LOC is slow. I worked as the sole GUI software engineer for a hardware firm and wrote > 100K LOC in 3 years, not including the test projects proceeding the actual real project. This was in C++, and included client/server stuff, entirely custom resizeable GUI, OpenGL 3D graphics and modelling of 3D assets and textures etc too. And getting it running under OSX + Win32, fixing issues on both. And that wasn't a stressful place to work with insane deadlines - it was fairly relaxed for the most part.
- deleted 10y ago
  [deleted]
- gizmo 10y ago
  You can't compare C++ with Ruby code. Rails code especially can very easily become a hairball where everything happens "somewhere else". You don't have static type checking or other compile time hints to figure out what is going on. You can't see which functions are called from which call sites. Refactoring tools? Forget it. Ruby is a very compact and flexible language, but if you're not disciplined you'll pay the price for it. A 60kloc C++ project is small and easily manageable, a 60kloc Ruby hairball can drive a person insane.
- RangerScience 10y ago
  Rails code. You get a lot of bang for your buck. That, and the "greater" metric for the project complexity is the stupidly high number of controllers and models.
- golergka 10y ago
  LOC differ not only by the language or technical difficulty of the task, but by business requirement difficulty as well. If, for example, they're in banking and finance, and those LOC deal with fine details of tax code... Oh boy.
- wheaties 10y ago
  As a former C++ dev, you can't compare Ruby LOC to C++ LOC. Just 4x the Ruby and then it becomes more fair. C++ is just verbose. #import <algorithm.h> doesn't fix everything blocks fix.
sydd 10y ago
IMO this is the price to pay for a dynamically typed language. 60K LOC is not much in a static language, you can use tools to refactor it easily or to visualize the control flow. But with a dynamically typed language? Its a nightmare. You change one thing and cannot possibly know what else could have gone wrong.
- pizza234 10y ago
  It depends. I work on such type of codebase, but we have a fully covering testing suite, so applying changes is not a problem (interestingly, I've just realized that the line count of the testing code is 50%+ more than the base application code itself). So ultimately I think company culture (that is, emphasis on automated testing, for dynamically typed languages) is the crucial factor.
- PaulRobinson 10y ago
  The legacy app has > 80% test coverage. Refactoring is still slow because there are all sorts of business assumptions put into place that add functionality without ever questioning the need for it. Basically, for a long time, the company never really re-evaluated what it had learned and spent time trimming things down, so as a result there is this ungodly mess. At the heart of what the business does, there is no real need for more than a dozen models. So why do we have so many more? Nobody ever refactored away stuff we didn't need any more, and so weird things happen. There is also a coupling issue that is endemic to all monoliths. We're moving to a micro-service architecture with clean domain separation, and we'll probably go to 1/10th of the code base in LOC terms within 12 months, even if we move some of that functionality into Go, Java or Python services (all options).
  - sydd 10y ago
    Unit tests just make the mess a bit easier to solve, but are far from a perfect solution. Wanna move and rename a function? Do it, then spend hours fixing your 10 broken tests, writing new ones, and test you app for hours because unit test dont cover integration. A static language + an IDE does it automatically within seconds.

luhn 10y ago

Can you expand on how technical debt caused a two-week outage?

TheCoreh 10y ago
If I had to guess, I'd say it's probably an issue with the build/deploy system. Perhaps someone deployed a broken build, then tried to revert/rollback, and realized that the previous version didn't build "cleanly" anymore. This could happen if you have a lot of dependencies, switched compiler versions but left the binaries "in place" and deployed changes incrementally.
- RangerScience 10y ago
  Yet another reason I swear by Heroku - you can rollback to the actual prior release, not rollback the code and try to rebuild.
  - ardivekar 10y ago
    Doesn't AWS do this too?
    - luhn 10y ago
      Depends on which of their 150 services you're talking about. You can't do it on CodeDeploy.
    - RangerScience 10y ago
      Which service? EC2 doesn't - at least not inherently, there might be a way to snapshot the machine's state; I'm not sure.
PaulRobinson 10y ago
There are two problems with giving a full answer: firstly I'm still under NDA for some of that, and too much detail would breach that; secondly, my exact memory of it is limited. In short, my predecessor had attempted a move to SOA without understanding dependencies, circuit breaking and failure modes. This would then cause scenarios where the entire front-end would fail to render on a single down-stream service taking a little longer than necessary. When identifying how to stop that happening, I discovered a large number of comments tagged "TODO" with statements like "Refactor this when we have time" or "We need to find a way to do this better". Further down on the downstream services there were rather esoteric SQL queries doing large joins that nobody had done a query plan on. It was hard to identify these because the ORM had been trusted to do magic, and it was happy to do so, but there was a point where it was not apparent _why_ these joins were happening, but when you found the code, there were more comments "This needs improving", "We should refactor this", etc. We were able to get something back quite quickly with liberal application of indexes, and it took us a day or two to refactor the queries enough to mean response times came down, but the error rate was still > 20%, and it was random, so 1-in-5 page loads of the front end service would fail. We refactored the code to circuit break and handle degraded services better, but that took a few days, and then we started working down to the back end service and figuring out the final steps. It was a small team looking after legacy code that everybody knew was a bit messy. A few weeks before this code was shuttered, I heard from a friend that some of our content did not render at all on certain Android devices. I identified the cause as a half-finished refactor (again, my predecessor), that had never been finished because he had been pushed to work on something else. This caused a dramatic decline within a key market segment that resulted in declining ad revenue, subscriptions and overall viability of the business. Basically, when you start something, finish it. If you find yourself putting in comments like "We should refactor this" anywhere in your code base, and you're doing so because the business is pushing you to work on new features, you have a massive problem culturally that is going to cause a rise in technical debt that raises risk to revenue. All technical debt ultimately will lead to problems that the business will see on balance sheets, but they will rarely successfully identify the cause as being technical debt because they can't see, understand or rationalise it. They think it's engineers being grumpy idealists. People play too fast and loose with the concept of "MVP" for my tastes, and it's a problem I see over and over again. The risk of that is, long-term, it will cause business failure.

72deluxe 10y ago

I would say that 8 years to write 60k LOC is slow. I worked as the sole GUI software engineer for a hardware firm and wrote > 100K LOC in 3 years, not including the test projects proceeding the actual real project. This was in C++, and included client/server stuff, entirely custom resizeable GUI, OpenGL 3D graphics and modelling of 3D assets and textures etc too. And getting it running under OSX + Win32, fixing issues on both. And that wasn't a stressful place to work with insane deadlines - it was fairly relaxed for the most part.

deleted 10y ago
[deleted]
gizmo 10y ago
You can't compare C++ with Ruby code. Rails code especially can very easily become a hairball where everything happens "somewhere else". You don't have static type checking or other compile time hints to figure out what is going on. You can't see which functions are called from which call sites. Refactoring tools? Forget it. Ruby is a very compact and flexible language, but if you're not disciplined you'll pay the price for it. A 60kloc C++ project is small and easily manageable, a 60kloc Ruby hairball can drive a person insane.
RangerScience 10y ago
Rails code. You get a lot of bang for your buck. That, and the "greater" metric for the project complexity is the stupidly high number of controllers and models.
golergka 10y ago
LOC differ not only by the language or technical difficulty of the task, but by business requirement difficulty as well. If, for example, they're in banking and finance, and those LOC deal with fine details of tax code... Oh boy.
wheaties 10y ago
As a former C++ dev, you can't compare Ruby LOC to C++ LOC. Just 4x the Ruby and then it becomes more fair. C++ is just verbose. #import <algorithm.h> doesn't fix everything blocks fix.

sydd 10y ago

IMO this is the price to pay for a dynamically typed language. 60K LOC is not much in a static language, you can use tools to refactor it easily or to visualize the control flow. But with a dynamically typed language? Its a nightmare. You change one thing and cannot possibly know what else could have gone wrong.

pizza234 10y ago
It depends. I work on such type of codebase, but we have a fully covering testing suite, so applying changes is not a problem (interestingly, I've just realized that the line count of the testing code is 50%+ more than the base application code itself). So ultimately I think company culture (that is, emphasis on automated testing, for dynamically typed languages) is the crucial factor.
PaulRobinson 10y ago
The legacy app has > 80% test coverage. Refactoring is still slow because there are all sorts of business assumptions put into place that add functionality without ever questioning the need for it. Basically, for a long time, the company never really re-evaluated what it had learned and spent time trimming things down, so as a result there is this ungodly mess. At the heart of what the business does, there is no real need for more than a dozen models. So why do we have so many more? Nobody ever refactored away stuff we didn't need any more, and so weird things happen. There is also a coupling issue that is endemic to all monoliths. We're moving to a micro-service architecture with clean domain separation, and we'll probably go to 1/10th of the code base in LOC terms within 12 months, even if we move some of that functionality into Go, Java or Python services (all options).
- sydd 10y ago
  Unit tests just make the mess a bit easier to solve, but are far from a perfect solution. Wanna move and rename a function? Do it, then spend hours fixing your 10 broken tests, writing new ones, and test you app for hours because unit test dont cover integration. A static language + an IDE does it automatically within seconds.