16 ms·

The Lava Layer Anti-Pattern (2014)

77 points by cheeseface 3y ago

userbinator 3y ago
Each wanted to re-write the application rather than maintain it, but the business owners would not allow them the resources to do it. One trick I've successfully used to get around this is to basically work on doing that in what would otherwise be slack time, at "background priority". Only when you're done do you give the business owners the proposal, and when they object, say that you already did it and show them the results. Of course this works only if you actually do have slack time, which is normally a given in the sort of enterprise environment these abominations usually get created in; and you understand the existing application to a sufficiently large extent and manage to find a way to simplify it to a fraction of its current size while maintaining backwards compatibility, something that seems to be the exact opposite of what the leads in this story wanted to do. "It's easier to ask for forgiveness than permission."
- mike_hock 3y ago
  There seems to be an innate human desire to fix problems right in front of your eyes that you can see clearly. And that's not limited to software development. People seem to be constantly developing workarounds for the deliberate sabotage by their own superiors preventing them from doing the work they're being paid to do, and even sacrificing their own time for it. Unfortunately, this keeps the parasites in power as it masks the damage their approaches are doing, or would be doing if left to run their natural course. Don't be a cuck and pick up others' shit. Especially not if they're higher-paid and get to give you orders.
  - imbnwa 3y ago
    >Unfortunately, this keeps the parasites in power as it masks the damage their approaches are doing, or would be doing if left to run their natural course. I hear this, but I also find there's practically no retrospective at the managerial level outside of an acquisition as that's the only way a culture shift is incurred at the upper levels, and that of course all depends on the acquirer. They're all too busy playing the game of minimizing their exposure, creating leverage and advantage for their ambitions, etc. I feel like a managerial culture is something that kicks off on day one and gets perpetuated, or it just turns into every other Enterprise eventually.
- chromanoid 3y ago
  Quote from https://medium.com/geekculture/the-dead-sea-effect-d71df13724f8 https://medium.com/geekculture/the-dead-sea-effect-d71df1372... Dead Sea Effect: The “bad” employees left behind are not always bad — hence the quotes. I disagree with that idea. I think it lacks empathy and perspective. There are a lot of instances where good employers stay in situations like this for very valid reasons. Here are some examples: [...] They believe they can singlehandedly change bad practices as an individual contributor. --- I can only say: take the whole company with you to remove technical _and_ organizational debt otherwise any rewrite will end up as the next lava layer.
shireboy 3y ago
Struggling with this on a current project, except I’d say it’s not currently consistent. It already has “lava layers”, lots of bugs and unintended behavior devs struggle to reproduce in dev/test. We’re debating rewrite vs refactor now. I find myself going back and forth between dogmatic and pragmatic approaches. Which maybe is a good balance to try to strike.
- physicles 3y ago
  You already know the answer: rewrite is almost never the right option. If you’re not familiar with everything the code does, how will you rebuild it? Step 1 is to get some tests up to allow yourself to refactor quickly without breaking too much stuff.
physicles 3y ago
I’m surprised that tests are only mentioned as a way to squash bugs and not as a way to make refactoring faster. Just today I converted a large library into a service and brought in some endpoints from another service. The diff is several thousand lines. But I’m not too worried about bugs because we have decent test coverage. I’m not sure why, but I really like working with legacy code (as long as I’m given some autonomy to improve it over time). Part of the enjoyment is the restraint, the humility when facing code that, despite not being squeaky clean, has racked up millions or billions of hours getting shit done. The code knows itself better than you do.
dmarinus 3y ago
In other words: any incomplete refactoring is technical debt. In case of considering a refactor always make sure you can finish it completely.
jwnin 3y ago
I've coined this "architecture archaeology" at work. The further you dig, you find a new technology in use, corresponding with new technical leadership.
- john-tells-all 3y ago
  technical leadership determines technology... with some (or a lot) of "legacy" tech remaining in place. At a company, I was brought in to help Version 2 get off the ground (serverless everything, Typescript). Version 1 worked just fine (PHP + VM), they kept none of it. When I left two years later, the new tech leadership started Version 3 (Ruby). Three rewrites in three years!
logophobia 3y ago
I once, for a short period, maintained web-application that had three different frontend frameworks (angular 1, angular 2, and one other I don't quite remember), and four different javascript builds that had to be run to build the application. Apparently management prioritized the giving demos to investors, and changes to different frameworks were aborted halfway through. It was completely impossible to work with. Each week the build failed for new random reasons. I hardly dared touch the thing. This "pattern" is really a failure from multiple parties: * Managing software engineers is an art, and you really need to understand what is happening to succeed. Only prioritizing short-term goals just ensures you're going to fail in the long-term. Make sure you understand technical debt. The speed of work in bad code-bases versus good code-bases can be orders of magnitude in difference. * Software engineers really need to use branches properly. Work that is halfway done should not be in the main branch. Consistency and simplicity is king here. Maintaining an old and new version of software for a while can be a pain, but it's much better than maintaining a halfway converted application. Pressure from management is no reason to release stuff halfway done. And if you need to demo, release a specific branch. Nowadays I don't even ask to do necessary maintenance. It's just part of the job. Always stick to the boyscout rule (leave things in a better state then you found it). Make your code-bases cleaner incrementally, and eventually you'll be in a much better state.
hardware2win 3y ago
Too many software charlatans with too big responsibilities (architecture/strategy), with too many books read that were written by evangelists and way too little experience in whole software development lifecycle. Write software, 1.5 year later update CV with fancy buzzwords you used, change job for better comp and repeat. Who cares how did the design mature? They always have some method/approach which is unparallelled in every metric, except it being reflected in reality. Years of brainwash caused new developers to make strange looks if you use "if" statement or write comments in your code. Also acting as if design pattern was some holy code instead of just a name for an approach to some problem (just normal code, but with name).
- mkl95 3y ago
  > Write software, 1.5 year later update CV with fancy buzzwords you used, change job for better comp and repeat. Who cares how did the design mature? That's on their employer for offering subpar comp. They could have offered that employee a raise instead, and asked him to rewrite his bad code.
  - greenyoda 3y ago
    > They could have offered that employee a raise instead, and asked him to rewrite his bad code. Why would an employee deserve a raise if they write bad code that has to be rewritten?
    - mkl95 3y ago
      Usually you don't get a raise because you deserve it, you get it because the company believes keeping you will result in a more positive financial outcome in the next few quarters than replacing you. Plenty of companies give their engineers a chance to pay their tech debt, whether it is bad code, bad infra decisions, etc. If you don't believe some engineer is capable of paying their tech debt, you may as well let them go. Or not give them a raise and hope they leave, which at most places is much cheaper.
- diarrhea 3y ago
Splizard 3y ago
Code unambiguously defines 'what' it's doing, so I'm not sure what value technical consistency provides. If anything, consistency hurts the capacity for developers to generalise their understanding of a system. Differences in representation ie. multiple perspectives, help people to develop their understanding of the underlying concepts of the system. This is important because code doesn't effectively record concepts, intents and oughts. These are things recorded within the context of the system. So the more context, usage and interactions available, the more opportunity for learning. I think consistency is one of those things that sounds appealing and offers the illusion of making development 'easier' but boils down to someone deciding by fiat that their poorly communicated representation of the system is correct and nobody else should add their own representations of the system to build up a useful level of context and understanding.
- edoloughlin 3y ago
  > Code unambiguously defines 'what' it's doing, so I'm not sure what value technical consistency provides Unless it’s really convoluted, most of the time I’m reading code, I’m concerned with why it’s doing something. Technical consistency helps understand the intent by removing distractions and the need to switch paradigms.
- tester756 3y ago
  Consistent code base makes it easier to read to onboard new people to review if you have e.g 3 different ways to handle error handling then it is a mess. one time you expect exceptions, other time monads, other time int values.
  - physicles 3y ago
    It also makes it faster to write code. If the code base has a strong culture around doing X, then whenever you do X, you don’t have to spend time thinking about the best way to do it.
- Timon3 3y ago
  Hard disagree. Inconsistency increases the amount of time needed for any feature and increases the likelihood of bugs. There is no upside in having to understand each individual piece of functionality from the ground up, because a lot of functionality is simply repeated in slightly different ways. What you call "hurting the capacity for developers to generalise their understanding of a system" is what I would call "building the capacity for developers to generalise their understanding of a system" - consistency is what allows for generalization.
jjk166 3y ago
There is a concept known as Chesterton's Fence - that you shouldn't take down a fence someone else put up until you know why they put it up, or more generally if you don't understand why something was previously considered a good solution to a problem, then it's possible you don't understand the problem it was meant to solve.
- mfashby 3y ago
  In my experience, so many things were _not_ a good solution to the problem but merely the first thing that looked like it worked. This leaves a trail of garbage behind which it's unclear if it's safe to remove or not, which is a huge cognitive overhead for someone trying to make changes later on. If something is there that's not going to be obvious in 6 months time please please document/comment it. Stop building mysterious fences!
  - mike_hock 3y ago
    That's right, but the advice is targeted at you when somebody's already left you a mystery fence. They should have done better and documented it, but they didn't and now you have to deal with it.
  - lfowles 3y ago
    Is this Chesterton's Garbage?
    - crabmusket 3y ago
      Chesterton's Roadside Picnic
      - ooterness 3y ago
        I love this analogy. Sometimes it's a perfectly good fence, sometimes it's an invisible spatial anomaly that turns people inside out. ¯\_(ツ)_/¯
  - Rexxar 3y ago
tanseydavid 3y ago
In my experience, it is very difficult to get a commitment for the effort to remove deadcode. The problem is essentially invisible to everyone except maybe the dev who is dealing with a high noise-to-signal ratio as a result. If it was a woodworking shop instead of a code repository, management would constantly see what a mess the shop is due to never being cleaned up properly. They would never tolerate this because of the way it looks. The woodshop looks sloppy, half-assed and almost sure to be an inefficient place to do work, if it does not get cleaned up on a regular basis. Because they cannot see deadcode in the repository in a similar manner, it is especially challenging to get them to care.
- JonChesterfield 3y ago
  You just delete it on the fly as you come across it, since it lives on in the version control system anyway. If your workplace doesn't have version control, leave.
- moritzwarhier 3y ago
  This might not add much to the discussion, since the post is not really about dead code removal, but your woodworking shop analogy reminds me of the bakery analogy at the beginninf of this old post by Joel Spolsky: https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

77 points by cheeseface 3y ago

Each wanted to re-write the application rather than maintain it, but the business owners would not allow them the resources to do it. One trick I've successfully used to get around this is to basically work on doing that in what would otherwise be slack time, at "background priority". Only when you're done do you give the business owners the proposal, and when they object, say that you already did it and show them the results. Of course this works only if you actually do have slack time, which is normally a given in the sort of enterprise environment these abominations usually get created in; and you understand the existing application to a sufficiently large extent and manage to find a way to simplify it to a fraction of its current size while maintaining backwards compatibility, something that seems to be the exact opposite of what the leads in this story wanted to do. "It's easier to ask for forgiveness than permission."

mike_hock 3y ago
There seems to be an innate human desire to fix problems right in front of your eyes that you can see clearly. And that's not limited to software development. People seem to be constantly developing workarounds for the deliberate sabotage by their own superiors preventing them from doing the work they're being paid to do, and even sacrificing their own time for it. Unfortunately, this keeps the parasites in power as it masks the damage their approaches are doing, or would be doing if left to run their natural course. Don't be a cuck and pick up others' shit. Especially not if they're higher-paid and get to give you orders.
- imbnwa 3y ago
  >Unfortunately, this keeps the parasites in power as it masks the damage their approaches are doing, or would be doing if left to run their natural course. I hear this, but I also find there's practically no retrospective at the managerial level outside of an acquisition as that's the only way a culture shift is incurred at the upper levels, and that of course all depends on the acquirer. They're all too busy playing the game of minimizing their exposure, creating leverage and advantage for their ambitions, etc. I feel like a managerial culture is something that kicks off on day one and gets perpetuated, or it just turns into every other Enterprise eventually.
chromanoid 3y ago
Quote from https://medium.com/geekculture/the-dead-sea-effect-d71df13724f8 https://medium.com/geekculture/the-dead-sea-effect-d71df1372... Dead Sea Effect: The “bad” employees left behind are not always bad — hence the quotes. I disagree with that idea. I think it lacks empathy and perspective. There are a lot of instances where good employers stay in situations like this for very valid reasons. Here are some examples: [...] They believe they can singlehandedly change bad practices as an individual contributor. --- I can only say: take the whole company with you to remove technical _and_ organizational debt otherwise any rewrite will end up as the next lava layer.

shireboy 3y ago

Struggling with this on a current project, except I’d say it’s not currently consistent. It already has “lava layers”, lots of bugs and unintended behavior devs struggle to reproduce in dev/test. We’re debating rewrite vs refactor now. I find myself going back and forth between dogmatic and pragmatic approaches. Which maybe is a good balance to try to strike.

physicles 3y ago
You already know the answer: rewrite is almost never the right option. If you’re not familiar with everything the code does, how will you rebuild it? Step 1 is to get some tests up to allow yourself to refactor quickly without breaking too much stuff.

physicles 3y ago

I’m surprised that tests are only mentioned as a way to squash bugs and not as a way to make refactoring faster. Just today I converted a large library into a service and brought in some endpoints from another service. The diff is several thousand lines. But I’m not too worried about bugs because we have decent test coverage. I’m not sure why, but I really like working with legacy code (as long as I’m given some autonomy to improve it over time). Part of the enjoyment is the restraint, the humility when facing code that, despite not being squeaky clean, has racked up millions or billions of hours getting shit done. The code knows itself better than you do.

dmarinus 3y ago

In other words: any incomplete refactoring is technical debt. In case of considering a refactor always make sure you can finish it completely.

jwnin 3y ago

I've coined this "architecture archaeology" at work. The further you dig, you find a new technology in use, corresponding with new technical leadership.

john-tells-all 3y ago
technical leadership determines technology... with some (or a lot) of "legacy" tech remaining in place. At a company, I was brought in to help Version 2 get off the ground (serverless everything, Typescript). Version 1 worked just fine (PHP + VM), they kept none of it. When I left two years later, the new tech leadership started Version 3 (Ruby). Three rewrites in three years!

logophobia 3y ago

I once, for a short period, maintained web-application that had three different frontend frameworks (angular 1, angular 2, and one other I don't quite remember), and four different javascript builds that had to be run to build the application. Apparently management prioritized the giving demos to investors, and changes to different frameworks were aborted halfway through. It was completely impossible to work with. Each week the build failed for new random reasons. I hardly dared touch the thing. This "pattern" is really a failure from multiple parties: * Managing software engineers is an art, and you really need to understand what is happening to succeed. Only prioritizing short-term goals just ensures you're going to fail in the long-term. Make sure you understand technical debt. The speed of work in bad code-bases versus good code-bases can be orders of magnitude in difference. * Software engineers really need to use branches properly. Work that is halfway done should not be in the main branch. Consistency and simplicity is king here. Maintaining an old and new version of software for a while can be a pain, but it's much better than maintaining a halfway converted application. Pressure from management is no reason to release stuff halfway done. And if you need to demo, release a specific branch. Nowadays I don't even ask to do necessary maintenance. It's just part of the job. Always stick to the boyscout rule (leave things in a better state then you found it). Make your code-bases cleaner incrementally, and eventually you'll be in a much better state.

hardware2win 3y ago

Too many software charlatans with too big responsibilities (architecture/strategy), with too many books read that were written by evangelists and way too little experience in whole software development lifecycle. Write software, 1.5 year later update CV with fancy buzzwords you used, change job for better comp and repeat. Who cares how did the design mature? They always have some method/approach which is unparallelled in every metric, except it being reflected in reality. Years of brainwash caused new developers to make strange looks if you use "if" statement or write comments in your code. Also acting as if design pattern was some holy code instead of just a name for an approach to some problem (just normal code, but with name).

mkl95 3y ago
> Write software, 1.5 year later update CV with fancy buzzwords you used, change job for better comp and repeat. Who cares how did the design mature? That's on their employer for offering subpar comp. They could have offered that employee a raise instead, and asked him to rewrite his bad code.
- greenyoda 3y ago
  > They could have offered that employee a raise instead, and asked him to rewrite his bad code. Why would an employee deserve a raise if they write bad code that has to be rewritten?
  - mkl95 3y ago
    Usually you don't get a raise because you deserve it, you get it because the company believes keeping you will result in a more positive financial outcome in the next few quarters than replacing you. Plenty of companies give their engineers a chance to pay their tech debt, whether it is bad code, bad infra decisions, etc. If you don't believe some engineer is capable of paying their tech debt, you may as well let them go. Or not give them a raise and hope they leave, which at most places is much cheaper.
diarrhea 3y ago

Splizard 3y ago

Code unambiguously defines 'what' it's doing, so I'm not sure what value technical consistency provides. If anything, consistency hurts the capacity for developers to generalise their understanding of a system. Differences in representation ie. multiple perspectives, help people to develop their understanding of the underlying concepts of the system. This is important because code doesn't effectively record concepts, intents and oughts. These are things recorded within the context of the system. So the more context, usage and interactions available, the more opportunity for learning. I think consistency is one of those things that sounds appealing and offers the illusion of making development 'easier' but boils down to someone deciding by fiat that their poorly communicated representation of the system is correct and nobody else should add their own representations of the system to build up a useful level of context and understanding.

edoloughlin 3y ago
> Code unambiguously defines 'what' it's doing, so I'm not sure what value technical consistency provides Unless it’s really convoluted, most of the time I’m reading code, I’m concerned with why it’s doing something. Technical consistency helps understand the intent by removing distractions and the need to switch paradigms.
tester756 3y ago
Consistent code base makes it easier to read to onboard new people to review if you have e.g 3 different ways to handle error handling then it is a mess. one time you expect exceptions, other time monads, other time int values.
- physicles 3y ago
  It also makes it faster to write code. If the code base has a strong culture around doing X, then whenever you do X, you don’t have to spend time thinking about the best way to do it.
Timon3 3y ago
Hard disagree. Inconsistency increases the amount of time needed for any feature and increases the likelihood of bugs. There is no upside in having to understand each individual piece of functionality from the ground up, because a lot of functionality is simply repeated in slightly different ways. What you call "hurting the capacity for developers to generalise their understanding of a system" is what I would call "building the capacity for developers to generalise their understanding of a system" - consistency is what allows for generalization.

jjk166 3y ago

There is a concept known as Chesterton's Fence - that you shouldn't take down a fence someone else put up until you know why they put it up, or more generally if you don't understand why something was previously considered a good solution to a problem, then it's possible you don't understand the problem it was meant to solve.

mfashby 3y ago
In my experience, so many things were _not_ a good solution to the problem but merely the first thing that looked like it worked. This leaves a trail of garbage behind which it's unclear if it's safe to remove or not, which is a huge cognitive overhead for someone trying to make changes later on. If something is there that's not going to be obvious in 6 months time please please document/comment it. Stop building mysterious fences!
- mike_hock 3y ago
  That's right, but the advice is targeted at you when somebody's already left you a mystery fence. They should have done better and documented it, but they didn't and now you have to deal with it.
- lfowles 3y ago
  Is this Chesterton's Garbage?
  - crabmusket 3y ago
    Chesterton's Roadside Picnic
    - ooterness 3y ago
      I love this analogy. Sometimes it's a perfectly good fence, sometimes it's an invisible spatial anomaly that turns people inside out. ¯\_(ツ)_/¯
- Rexxar 3y ago

tanseydavid 3y ago

In my experience, it is very difficult to get a commitment for the effort to remove deadcode. The problem is essentially invisible to everyone except maybe the dev who is dealing with a high noise-to-signal ratio as a result. If it was a woodworking shop instead of a code repository, management would constantly see what a mess the shop is due to never being cleaned up properly. They would never tolerate this because of the way it looks. The woodshop looks sloppy, half-assed and almost sure to be an inefficient place to do work, if it does not get cleaned up on a regular basis. Because they cannot see deadcode in the repository in a similar manner, it is especially challenging to get them to care.

JonChesterfield 3y ago
You just delete it on the fly as you come across it, since it lives on in the version control system anyway. If your workplace doesn't have version control, leave.
moritzwarhier 3y ago
This might not add much to the discussion, since the post is not really about dead code removal, but your woodworking shop analogy reminds me of the bakery analogy at the beginninf of this old post by Joel Spolsky: https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...