5 ms·

> In my career, I have taken a particular approach based on building new systems from scratch (before they succumb to the three laws), but this is a lot harder

> In my career, I have taken a particular approach based on building new systems from scratch (before they succumb to the three laws), but this is a lot harder than it sounds This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them. Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. We have an existing system. This needs to be maintained and extended because it's our core business. We cannot just say "oh it's become too complex. Let's go and spend the next few years creating a new product, so we'll have a multi-year feature freeze". The only feasible way I see to do anything even close to this is to split up the software into domain-like silos slowly and incrementally. At that point you can start considering rewriting silos as they become too complex/cumbersome/big. However, at that point you're already doing complex, lengthy and generally risky refactoring, and speaking of it being a write from scratch is very obviously not true.

pif 2y ago
I agree with you. Too many of us developers tend to forget that the customer only looks at the features. Clean code, unit tests, rigorous processes... they are only as valuable as the rate of features they make possible. Complexity and technical debt are our problem, and the customer could not care less about how we solve it, or even whether we solve it. As long as we are struggling with an old piece of sh*t, that's good for us: it means the product is selling and the effort to maintain it still makes sense! Selling without complexity and technical debt would be much better, I think we all agree on this, but it's a receipt whose success I have never personally witnessed.
- kunley 2y ago
  "customer could not care less". This perspective is too narrow, or even a fallacy, often repeated to justify abandoning some good practices. Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them. Most of customers do such choices.
  - jagged-chisel 2y ago
    In software, all that is hidden from the end customer. How do you look for the smart choices in code made by the team who implemented any of the software you download and install?
    - FroshKiller 2y ago
      Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.
      - pif 2y ago
        > Poor choices during development reveal themselves in many ways: ... All the examples you provided miss the point. Revenues are the point; all the rest is ancillary at best.
        FroshKiller 2y ago
        I'm not sure what you mean. jagged-chisel asked how we look for smart choices in the software we download and install. I gave examples of how poor choices manifest for end users, the idea being that a lack of those issues suggests smart choices. I don't see how revenue is relevant to the question. Revenue for whom? And how did my examples miss the point exactly?
      - lelanthran 2y ago
        > Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on. Not necessarily. Some otherwise good systems might have one bug in each of those categories. And, most other poorly designed and poorly executed systems just chug along fine without the customer realising anything.
        stouset 2y ago
- resonious 2y ago
  Customers care a lot about defects, and I've seen defects increase dramatically due to complexity and technical debt.
- pydry 2y ago
  What gets forgotten is the thing you can't see. Persistent refactoring is one of those things where the cost is clear and well known (expensive developer time) while the payoff is big, but very diffuse, very hard to explain and often gets attributed to something else - sometimes even by the developers who did it. It's not the only thing like that. Plenty of companies skimp on cost centers and fail spectacularly as a result - sometimes without even ever realizing that skimping on the relevant cost centres was the reason why they failed.
  - AndyMcConachie 2y ago
    Refactoring only matters if decision makers care about long term consequences and value. One thing that makes it difficult to make these kinds of decisions wisely is that no one knows how long a piece of software will live in production. If I could know for sure that a piece of software would exist for 50 years in production I could make a better case for refactoring. Likewise, if I knew for sure that a piece of software would get completely replaced in 1 year I would not bother with refactoring. Never in my career have I been able to accurately predict how long software would live in production. I have had throwaway code live for years and things I thought would last forever get replaced almost instantly.
    - pydry 2y ago
      Refactoring doesn't just benefit you in 50 years it can benefit within weeks. I agree that it's a waste on throwaway code, throwaway code is common and what ends up being throwaway code is hard to predict, but I don't think it's impossible to get a sense of the likelihood of the code being tossed, and that sense can be used to tweak your refactoring budget.
      - Supermancho 2y ago
        > Refactoring doesn't just benefit you in 50 years it can benefit within weeks. Proving the value is the hard problem. It's almost impossible to prove the value of small changes today. Product wants things to do X and they decided the value tradeoff was good, not the engineers who are tasked with making it so.
- makingstuffs 2y ago
  While it is hard to generalise to an absolute I do largely agree with this sentiment. The amount of times I have advised clients to not use something like Wordpress to just end up having to use a page builder like Divi hurts my soul. At the end of the day the majority don’t care about things like carrying bloat/legacy code, rigorous testing patterns, using modern tech stacks or typesafe codebases. They simply want you to “make thing go brr”. I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities. I guess the main thing from a business perspective is that I’m getting paid but, at the same time, ouch, my feels.
  - abirch 2y ago
    I agree with you though I'd state the cheapest upfront option. They tend to cost more in the long run. See Terry Pratchett's boot theory: https://en.m.wikipedia.org/wiki/Boots_theory https://en.m.wikipedia.org/wiki/Boots_theory
    - datadrivenangel 2y ago
      The problem with this theory is that the $50 boot manufacturers sell out to private equity when the founder wants to retire and then the PE firm jacks up the price to $75 and drops the quality below the $10 cardboard boots... so you're out even more money to have wet feet.
  - bluGill 2y ago
    > I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities. Until more client gets burned with a cheap option that becomes expensive to maintain they won't care. Many clients (across the whole industry, I don't know your clients which could be very different from average) will not get burned as most problems will never get that complex. However most clients have a lot of small problems that are quick to solve meaning that most developers spend their time on the small minority of clients that have complex problems where this matters.
- sinaid 2y ago
  The customer would also be elated if they paid you nothing and they were your only customer.
vesinisa 2y ago
Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen. A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand. That last part is also the most common reason why large system replacement projects fail, go over buddget, or are not even attempted. Often the people controlling the money do not have the technical knowledge how much complexity the old system hides. The safe bet is to keep it going.
- AllegedAlec 2y ago
  > Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen. Yes. It can happen. If you can't do it with a 99.9% succes rate you cannot make it a general business practice. > A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand. Beyond my general disagreement is shouldn't need full parity (man our clients would be pissed off if we cut functionality "because it was easier to rebuild a system that was functioning quite well on your end", and they have guns!), I don't think this takes into full account how much can be hidden in years and years of commits and code. We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". This, in spite of all handwringing about how this would never have happened if people maintained proper documentation, will happen in any legacy application. We're talking about a hundred people working side by side over decades. These things will slip through, or 'best practices' change so much that things that were a good idea then are entirely unknown now. Even something as non-intrusive as attempting to stranglevine this will take up a lot of time and effort, if it can be done correctly.
  - vesinisa 2y ago
    > We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". Interesting - when thinking of systems that I've personally worked on which I could never imagine being replaced, the first one that came to mind was one with exactly the same problem: a massive cache of SQL stored procedures at the core that nobody seemed to fully understand. This particular company had even implemented stuff like HTTP auth token validation and all access control with SQL scripts without any centralized access control architecture. It was outright terrifying to touch the SQL. When I started at the company the manager advertised they "really have no upper limit" on how much they compensate developers who stick with them for years. I guess a big problem with implementing large parts of the application logic with SQL is that it's simply not a tool fit for the purpose. Incrementally replacing, or even reasoning about, logic splintered accross assorted SQL scripts is very difficult. So nobody has the full picture of what are all the things the monolith does.
apantel 2y ago
The issues always enter when a codebase has to be extended in some way that was not part of the original design. You actually have to do ‘the big refactor’ right at that point of extension such that it is as if the new requirements were known up front when the system was originally designed. Otherwise you end up with a complicated mess very quickly.
kazinator 2y ago
What seems like an infeasible solution? The article doesn't describe the author's "particular approach". You snipped: > [...] a lot harder than it sounds – more on that in a different post. I don't see a newer post in that blog as of now (May 29, 2024).
- AllegedAlec 2y ago
  > What seems like an infeasible solution? The article doesn't describe the author's "particular approach". The entire idea of "well let's rebuild this before it ever gets bad". That only works in the one tiny niche corner of greenfield development that nearly no one works in. To say "oh yeah just don't ever let software get complex" is like saying "well have you tried excising any oncogenic mutations before they cause cancer"?
waynesonfire 2y ago
Entropy is a good analogy for software complexity. In that it takes more energy to decrease entropy than to increase it—is fundamental in understanding why in nature, without external intervention, systems tend to evolve towards greater disorder. As a result, a single engineer can wreck such havoc on a system that it's essentially impossible to repair. This seems to be the a key to the issue and it's a people problem.
astrobe_ 2y ago
> This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them. > Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. I think that what often happens, is that the manifestation of an old system crumbling under the weight of its complexity is that a "new kid on the block" (startup, etc.) eventually takes over. As long as the new thing only forgoes the old cruft very few care about and add stuff many care about, there's a possible takeover.
thefaux 2y ago
A lot of what you're describing sounds like the innovator's dilemma. One of the things that makes legacy systems so difficult is that everything is there for a reason, but often the reason is to satisfy a small minority of customers. It's easy to look at a component and say, gee this feature is only used by 2% of customers, we should consider dropping it. But that 2% is someone at the company's customer and they will raise hell if you get rid of it. So naturally the system gets filled with all these special cases to handle niche customer use cases none of which can ever be removed. This is when a startup comes along and targets the high value basic use cases and drops most advanced functionality. Then they start acquiring bigger and bigger customers, complexity accretes to meet their needs and then eventually the startup is the legacy company and the cycle repeats. The question as a developer is do I want to work on such a system? For me, the answer is a resounding no. If I agree to work on such a system, I will essentially be enslaved to the bad design decisions made before I joined the company. It will never be truly satisfying to work on that system. I am speaking from experience. But that is why companies pay a salary. The established company mindset is not the startup mindset is not the hobby project mindset. It's up to you to figure out what is best for you at any time and that choice may change as life circumstances change.
ughitsaaron 2y ago
I actually laughed out loud when I read that conclusion. Not only does it sound infeasible and unsustainable, it also sounds not a little arrogant and likely annoying for everyone else working with them.
- AllegedAlec 2y ago
  Can you imagine saying to your product owner or whatever "oh yeah we're not gonna do anything new. We'll spend the next year or so rebuilding this service because the code looks ugly."

pif 2y ago

I agree with you. Too many of us developers tend to forget that the customer only looks at the features. Clean code, unit tests, rigorous processes... they are only as valuable as the rate of features they make possible. Complexity and technical debt are our problem, and the customer could not care less about how we solve it, or even whether we solve it. As long as we are struggling with an old piece of sh*t, that's good for us: it means the product is selling and the effort to maintain it still makes sense! Selling without complexity and technical debt would be much better, I think we all agree on this, but it's a receipt whose success I have never personally witnessed.

kunley 2y ago
"customer could not care less". This perspective is too narrow, or even a fallacy, often repeated to justify abandoning some good practices. Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them. Most of customers do such choices.
- jagged-chisel 2y ago
  In software, all that is hidden from the end customer. How do you look for the smart choices in code made by the team who implemented any of the software you download and install?
  - FroshKiller 2y ago
    Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.
    - pif 2y ago
      > Poor choices during development reveal themselves in many ways: ... All the examples you provided miss the point. Revenues are the point; all the rest is ancillary at best.
      - FroshKiller 2y ago
        I'm not sure what you mean. jagged-chisel asked how we look for smart choices in the software we download and install. I gave examples of how poor choices manifest for end users, the idea being that a lack of those issues suggests smart choices. I don't see how revenue is relevant to the question. Revenue for whom? And how did my examples miss the point exactly?
    - lelanthran 2y ago
      > Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on. Not necessarily. Some otherwise good systems might have one bug in each of those categories. And, most other poorly designed and poorly executed systems just chug along fine without the customer realising anything.
      - stouset 2y ago
resonious 2y ago
Customers care a lot about defects, and I've seen defects increase dramatically due to complexity and technical debt.
pydry 2y ago
What gets forgotten is the thing you can't see. Persistent refactoring is one of those things where the cost is clear and well known (expensive developer time) while the payoff is big, but very diffuse, very hard to explain and often gets attributed to something else - sometimes even by the developers who did it. It's not the only thing like that. Plenty of companies skimp on cost centers and fail spectacularly as a result - sometimes without even ever realizing that skimping on the relevant cost centres was the reason why they failed.
- AndyMcConachie 2y ago
  Refactoring only matters if decision makers care about long term consequences and value. One thing that makes it difficult to make these kinds of decisions wisely is that no one knows how long a piece of software will live in production. If I could know for sure that a piece of software would exist for 50 years in production I could make a better case for refactoring. Likewise, if I knew for sure that a piece of software would get completely replaced in 1 year I would not bother with refactoring. Never in my career have I been able to accurately predict how long software would live in production. I have had throwaway code live for years and things I thought would last forever get replaced almost instantly.
  - pydry 2y ago
    Refactoring doesn't just benefit you in 50 years it can benefit within weeks. I agree that it's a waste on throwaway code, throwaway code is common and what ends up being throwaway code is hard to predict, but I don't think it's impossible to get a sense of the likelihood of the code being tossed, and that sense can be used to tweak your refactoring budget.
    - Supermancho 2y ago
      > Refactoring doesn't just benefit you in 50 years it can benefit within weeks. Proving the value is the hard problem. It's almost impossible to prove the value of small changes today. Product wants things to do X and they decided the value tradeoff was good, not the engineers who are tasked with making it so.
makingstuffs 2y ago
While it is hard to generalise to an absolute I do largely agree with this sentiment. The amount of times I have advised clients to not use something like Wordpress to just end up having to use a page builder like Divi hurts my soul. At the end of the day the majority don’t care about things like carrying bloat/legacy code, rigorous testing patterns, using modern tech stacks or typesafe codebases. They simply want you to “make thing go brr”. I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities. I guess the main thing from a business perspective is that I’m getting paid but, at the same time, ouch, my feels.
- abirch 2y ago
  I agree with you though I'd state the cheapest upfront option. They tend to cost more in the long run. See Terry Pratchett's boot theory: https://en.m.wikipedia.org/wiki/Boots_theory https://en.m.wikipedia.org/wiki/Boots_theory
  - datadrivenangel 2y ago
    The problem with this theory is that the $50 boot manufacturers sell out to private equity when the founder wants to retire and then the PE firm jacks up the price to $75 and drops the quality below the $10 cardboard boots... so you're out even more money to have wet feet.
- bluGill 2y ago
  > I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities. Until more client gets burned with a cheap option that becomes expensive to maintain they won't care. Many clients (across the whole industry, I don't know your clients which could be very different from average) will not get burned as most problems will never get that complex. However most clients have a lot of small problems that are quick to solve meaning that most developers spend their time on the small minority of clients that have complex problems where this matters.
sinaid 2y ago
The customer would also be elated if they paid you nothing and they were your only customer.

vesinisa 2y ago

Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen. A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand. That last part is also the most common reason why large system replacement projects fail, go over buddget, or are not even attempted. Often the people controlling the money do not have the technical knowledge how much complexity the old system hides. The safe bet is to keep it going.

AllegedAlec 2y ago
> Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen. Yes. It can happen. If you can't do it with a 99.9% succes rate you cannot make it a general business practice. > A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand. Beyond my general disagreement is shouldn't need full parity (man our clients would be pissed off if we cut functionality "because it was easier to rebuild a system that was functioning quite well on your end", and they have guns!), I don't think this takes into full account how much can be hidden in years and years of commits and code. We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". This, in spite of all handwringing about how this would never have happened if people maintained proper documentation, will happen in any legacy application. We're talking about a hundred people working side by side over decades. These things will slip through, or 'best practices' change so much that things that were a good idea then are entirely unknown now. Even something as non-intrusive as attempting to stranglevine this will take up a lot of time and effort, if it can be done correctly.
- vesinisa 2y ago
  > We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". Interesting - when thinking of systems that I've personally worked on which I could never imagine being replaced, the first one that came to mind was one with exactly the same problem: a massive cache of SQL stored procedures at the core that nobody seemed to fully understand. This particular company had even implemented stuff like HTTP auth token validation and all access control with SQL scripts without any centralized access control architecture. It was outright terrifying to touch the SQL. When I started at the company the manager advertised they "really have no upper limit" on how much they compensate developers who stick with them for years. I guess a big problem with implementing large parts of the application logic with SQL is that it's simply not a tool fit for the purpose. Incrementally replacing, or even reasoning about, logic splintered accross assorted SQL scripts is very difficult. So nobody has the full picture of what are all the things the monolith does.

apantel 2y ago

The issues always enter when a codebase has to be extended in some way that was not part of the original design. You actually have to do ‘the big refactor’ right at that point of extension such that it is as if the new requirements were known up front when the system was originally designed. Otherwise you end up with a complicated mess very quickly.

kazinator 2y ago

What seems like an infeasible solution? The article doesn't describe the author's "particular approach". You snipped: > [...] a lot harder than it sounds – more on that in a different post. I don't see a newer post in that blog as of now (May 29, 2024).

AllegedAlec 2y ago
> What seems like an infeasible solution? The article doesn't describe the author's "particular approach". The entire idea of "well let's rebuild this before it ever gets bad". That only works in the one tiny niche corner of greenfield development that nearly no one works in. To say "oh yeah just don't ever let software get complex" is like saying "well have you tried excising any oncogenic mutations before they cause cancer"?

waynesonfire 2y ago

Entropy is a good analogy for software complexity. In that it takes more energy to decrease entropy than to increase it—is fundamental in understanding why in nature, without external intervention, systems tend to evolve towards greater disorder. As a result, a single engineer can wreck such havoc on a system that it's essentially impossible to repair. This seems to be the a key to the issue and it's a people problem.

astrobe_ 2y ago

> This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them. > Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. I think that what often happens, is that the manifestation of an old system crumbling under the weight of its complexity is that a "new kid on the block" (startup, etc.) eventually takes over. As long as the new thing only forgoes the old cruft very few care about and add stuff many care about, there's a possible takeover.

thefaux 2y ago

A lot of what you're describing sounds like the innovator's dilemma. One of the things that makes legacy systems so difficult is that everything is there for a reason, but often the reason is to satisfy a small minority of customers. It's easy to look at a component and say, gee this feature is only used by 2% of customers, we should consider dropping it. But that 2% is someone at the company's customer and they will raise hell if you get rid of it. So naturally the system gets filled with all these special cases to handle niche customer use cases none of which can ever be removed. This is when a startup comes along and targets the high value basic use cases and drops most advanced functionality. Then they start acquiring bigger and bigger customers, complexity accretes to meet their needs and then eventually the startup is the legacy company and the cycle repeats. The question as a developer is do I want to work on such a system? For me, the answer is a resounding no. If I agree to work on such a system, I will essentially be enslaved to the bad design decisions made before I joined the company. It will never be truly satisfying to work on that system. I am speaking from experience. But that is why companies pay a salary. The established company mindset is not the startup mindset is not the hobby project mindset. It's up to you to figure out what is best for you at any time and that choice may change as life circumstances change.

ughitsaaron 2y ago

I actually laughed out loud when I read that conclusion. Not only does it sound infeasible and unsustainable, it also sounds not a little arrogant and likely annoying for everyone else working with them.

AllegedAlec 2y ago
Can you imagine saying to your product owner or whatever "oh yeah we're not gonna do anything new. We'll spend the next year or so rebuilding this service because the code looks ugly."