8 ms·

> Except in very minor cases, duplication is virtually always worth fixing. I disagree with the severity of this, and would posit that there are duplications t

> Except in very minor cases, duplication is virtually always worth fixing. I disagree with the severity of this, and would posit that there are duplications that can't be "fixed" by an abstraction. There are many instances I've encountered where two pieces of code coincided to look similar at a certain point in time. As the codebase evolved, so did the two pieces of code, their usage and their dependencies, until the similarity was almost gone. An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. A "wrong abstraction" in that case isn't an ill-fitting abstraction where a better one was available, it's any (even the best possible) abstraction in a situation that has no fitting generalization, at all.

__alias 3y ago
I buy into the same belief as you here, but I guess you could easily argue that you could create a suitable fitting abstraction earlier on with the understanding that you can "detach" them once the point that they're fundamentally different comes
- consilient 3y ago
  The point of abstraction is to reduce the number of concepts in play. If you're still tracking which old concept is "really" being used every time, you haven't actually abstracted over anything, you're just naming things badly.
  - capableweb 3y ago
    > The point of abstraction is to reduce the number of concepts in play. I'm not sure I agree with this. For me, the point of abstraction is divide the number of concepts between the layers you introduce, effectively to hide concepts from the layers where you don't want to have to care about them. Often times, abstractions adds the total number of concepts at play, but hides them beneath/above the layers.
- fluoridation 3y ago
  The problem is that there's an impetus to continue working on top of established facilities, because it's usually incrementally less work than reworking a piece of code into something else. Plus it's difficult to recognize ahead of time when something is about to become a problem, rather than fix something that's already a problem.
peeters 3y ago
I try to think about whether two concepts are innately similar or incidentally similar. Computing compounding interest for a home equity loan and a mortgage might be innately similar. A desired change to one will probably make a desired change to the other. Computing growth of a fruit fly population and computing compounding interesting for a loan might be incidentally similar. Until you change your "computeExponentialGrowth" function to now handle occasional decimations from environmental sources, and anyone looking at the code wonders what the heck that looks like for a loan.
- AnimalMuppet 3y ago
  For a loan, maybe it looks like payments on the principal? But your overall point is very correct. Don't make an abstraction because of coincidence.
- HWR_14 3y ago
  As interest rates go back up, paying off (part of) a mortgage early might come back.
- adammarples 3y ago
  If you've got your abstractions correct, then the exponential growth term and the decimation term will be partial differentials which will compose together nicely
AnimalMuppet 3y ago
In that situation, the correct thing to do is, when the two pieces drift away from each other, to recognize that they are no longer the same abstraction and to break the connection. That may be painful - you have to look at everywhere that abstraction is used and figure out which thing it really is, and change the code to reflect it. But if that's going to happen, then in the early days, a little duplication was probably better.
- deleted 3y ago
  [deleted]
lukeramsden 3y ago
> There are many instances I've encountered where two pieces of code coincided to look similar at a certain point in time. As the codebase evolved, so did the two pieces of code, their usage and their dependencies, until the similarity was almost gone https://connascence.io/ https://connascence.io/
awkward 3y ago
A good example of this is operations type stuff, like the pile of shell scripts or terraform files or whatever that get used to deploy your app. These scripts benefit greatly from a one to one relationship between the thing you're creating and the written text describing it. Not having a situation where changing one thing breaks everything else is a huge help there.
MilStdJunkie 3y ago
I might have to say some unkind things here, but statements like: instead of “duplication is cheaper than the wrong abstraction”, I would say “duplication is cheaper than confusing code littered with conditional logic”. seems like it's looking at this problem from an extremely narrow context. The truth is that the phrase "wrong abstraction" is (more or less) unquantifiable, which makes the original phrase, as employed, sort of like a koan. It addresses the very human tendency to see patterns in noise, and our ability to "transmit" such hallucinations to other humans via natural language and other means. The closest I can get to - given my at-best-apprentice status as a formal programmer - is the quantitative test I developed for CCS (conditional content systems), where the abstraction lies in the SNS[1], and the de-duplication mechanism is applicability[2]. Since each applicability statement carries its own overhead, there's a limit on how much "abstraction" the model can take before it's using quantitatively more keystrokes than duplication. The test goes like this: take the flat text procedures for ALL the configurations, and add it together. Now, take the conditionalized, applicability-laden procedure that unifies the procedure, and measure its file size. If the latter is LARGER than the former, then you're using the wrong SNS/applicability model for rolling up this content. Thing is, this is inevitable if you throw enough dissimilar configurations at a CCS, because each configuration has its own overhead, and eventually that outpaces the content itself. You can address this in a bunch of ways - like adding a containing pseudo-product that has all the configurations inside of it - but the actual real Product Management might not let you build on the applicability like that, because the Product itself isn't sold that way. Any other abstraction isn't available to you, because in the end this is natural language, which - unlike structured language - resists first order abstractions really well. This is one of those instances where, yes, the abstraction of the SNS/Applicability is worse - quantifiably - than duplication. All that complexity would be better handled via version control fork/branch relationships - far outside of the realm of natural language. [1] standard numbering system, a sort of numeric designator of functional systems, the primary way that content is designated as semi-independent modules. [2] conditional "chunks" that turn on and off depending on the applicability statement
- feoren 3y ago
  It'd be wonderful if we could measure the utility of software engineering choices by counting keystrokes or measuring file sizes or putting them in a turbo encabulator and seeing which one has more modial interaction with its magneto-reluctance. Unfortunately, reality is just too complicated, with far too many tradeoffs to be balanced. I'd recommend deep thought and discussion about the domain over looking at a graph of your codebase's sinusoidal repleneration. > All that complexity would be better handled via version control fork/branch relationships Probably not.
mostlylurks 3y ago
> An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. This seems to be the underlying assumption behind most uses of the "duplication is cheaper than the wrong abstraction" quote, but the assumption is simply incorrect. You should almost never try to expand abstractions in this manner. If you don't treat the abstractions relating to the thing you want to change in your codebase as "the" place where you need to make your change, and instead eagerly make new abstractions and throw old ones away as required, you won't really run into this problem. In fact, this predominant mindset where creating abstractions is strongly discouraged leads to the very problem that mindset is based on, as it will simply encourage junior developers and the like to modify the existing abstraction, creating the aforementioned kind of mess where abstractions become complicated through repeated modification, instead of creating new abstractions when appropriate, because creating abstractions has a stigma attached to it. Additionally, if someone has made a "wrong" abstraction based on something silly like two pieces of code simply being similar in terms of their structure and those use cases start to drift apart, you should feel eager to simply split apart the abstraction, be it into bare implementations or two new abstractions, or any other combination. Abstractions are cheap as long as you don't give them special significance.
- sweezyjeezy 3y ago
  I think there's a middle ground here. The original quote does not mean DRY=bad, abstraction=bad. The point is there is a non-zero cost to these things. A bad abstraction can, as you say, accumulate to something terrible through inertia or inexperience. A bad abstraction, even if caught early, was probably not worthwhile - I mean, it took time just to make the original one right? This does not mean that we should be scared of abstraction in general, but in my opinion abstractions that are purely for the sake of reducing duplication should be viewed with an extra level of apprehension.
- jameshart 3y ago
  When an abstraction evolves to a point where it needs to be split into two separate implementations to meet diverging needs… you will need to replace that abstraction with duplication. Which is the right thing to do because that duplication is cheaper than maintaining the wrong abstraction. I think this post makes the mistake of thinking that the only way in which duplication comes up is that it is discovered in the codebase, and we have the choice of abstracting it away or keeping it. On the contrary, duplication can - and should - be consciously introduced to fix bad abstractions when we find them in the codebase.
mdiesel 3y ago
Equality doesn't necessarily mean Equivalence.
feoren 3y ago
You're absolutely right that it's important to look beyond how two modules superficially look right now, and look instead at how they change. However, if you've always defined your abstractions based on what their consumers need rather than what their implementations have, then you shouldn't ever need to stretch them. They're not trying to "cover" both cases, they're trying to solve a problem that both cases have. Your two cases are not implementations of the abstraction, they are consumers of it. If one case grows to not have that problem, it just stops asking for that abstraction. If it grows to have more problems, it just asks for more abstractions. The original abstraction, if based on a common need, doesn't have to change. That's not to say abstractions never change -- they do. But they change because your understanding of the sub-problem they're solving has changed, not because their implementations or consumers have changed.
crabbone 3y ago
So... you kept modifying the two similar pieces of code until they became dissimilar. Why do you think that you wouldn't be able to modify the abstraction if you saw that it doesn't fit anymore?
- convolvatron 3y ago
  I think part of the issue here is that a fair number of programmers work in shops where they have very limited agency. They are tasked with making the minimum defensible change to add a feature or fix a bug. They are not allowed to change the tests or suggest refactoring. So those things just don't occur.
  - crabbone 3y ago
    I see, but wouldn't this lack of agency, or, more precisely the inability to escalate the problem to someone with more edit rights be the actual issue? I could see this also happening in situations where such problems arise at component interface boundaries, where the pressure not to change comes not as an organizational policy but rather inability to influence external components (eg. the OS offers a poor abstraction -- the user-space program developer wouldn't be changing how OS works to rectify things). But, this is again sort of an administrative problem, because, ideally, the user-space program developer should be able to convince the OS developer to change the interface if it's found to be not a good abstraction... But, yes, I can see how in practice that'd be a very difficult thing to do.
    - convolvatron 3y ago
      I guess I was trying to claim that in a lot of places there isn't such a person. yeah - if an application company runs up against a poorly designed OS interface - they usually aren't aware, or just back away slowly. they don't have the scope or mandate to pursue it. that kind of behavior often extends to library dependencies, and even internal interfaces that the company should ostensibly own. its not so much an administrative problem as a desire to limit spending and time on software and do a kind of agile mvp glue job over an unbounded number of external dependencies. that often leads to an unmaintainable hairball. but if the alternative is to hire a bunch of really experienced developers and let them 'do the right thing' for 5 years... even if you do, there is a still a good chance that the output isn't gonna be that great. software people lost a lot of credibility with that model. we've probably swung too far in the other direction
ilyt 3y ago
> There are many instances I've encountered where two pieces of code coincided to look similar at a certain point in time. As the codebase evolved, so did the two pieces of code, their usage and their dependencies, until the similarity was almost gone. An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. Then you split that abstraction again. It's very cheap and very quick. Many people talk about the issue like it was an absolute in the code, but that's wrong approach. If you end up writing 4 functions that are the same, by all means, merge it into one. If then you need to add a parameter only this code path uses and rest doesn't care about, by all means split it back. Moving blocks of code around is cheap.
- oxfordmale 3y ago
  Splitting the abstraction is never cheap and quick, mostly because of politics. With duplicated code you often can assign a single responsible owner to each duplication. However, once abstracted, the code may suddenly be used by a number of different teams. You will need to get this work on their roadmap, increasing the friction to get this done. In many companies, this will also end up in endless discussions about the new approach.
  - esafak 3y ago
    This is a good point. Following Conway's Law, a team may choose to duplicate code or do thing theoretically sub-optimally simply to avoid having to deal with other teams.
  - yellowapple 3y ago
    Solution there would be to make the abstraction "opt-in", such that a team can elect to duplicate or abstract as desired. Also helps if the "main" abstraction is itself composed from smaller abstractions, from which downstream teams could then pick-and-choose rather than having to either fully abstract or fully duplicate.
- jrumbut 3y ago
  I think the key here is the oft repeated but often poorly understood maxim to favor composition ("has a") over inheritance ("is a"). If you have a mixin (or other means of composition) that you use in several places and one diverges, it's easy to remove it. If you use inheritance, it's going to be more painful. A language that offers OOP via prototypes instead of classes like JS can (sometimes) give you the best of both worlds, but it will confuse a lot of devs who aren't familiar with that kind of OO design.
BWStearns 3y ago
Agreed. Abstractions also tend to be more resistant to change, both from a technical level, and a social level. At a technical level an abstraction will have more call sites to worry about in different contexts, the more wrong the initial abstraction the harder it will be to change. The social level is maybe even more problematic. Abstractions seem more important than calling code and will experience more friction in code review. This change friction can also increase with the "wrongness" of the initial abstraction. The starting point makes less sense so a reviewer needs to work more to understand the context. If the abstraction is gnarly enough then it's possible that the reason for the abstraction is almost obscured. Even someone who knows _how_ it works might have lost the forest through the trees and push back on changes that simplify it or improve it if the change is a sufficiently large departure from the initial state. In this case you can often see small incremental changes get added easier but this just makes the shared code a bit gnarlier for next time.
- ljm 3y ago
  This is my beef with naively applied DDD, separation of concerns, and design patterns. Usually what happens is the 'clean' code ideal comes first, and then the implementation is squeezed into it. This then informs the organisation (or architecture) of the rest of the codebase and your software design has become a matter of putting pegs into the right-shaped holes. I have never found that kind of highly abstracted code easier to work with than some simple procedural alternative that is easy to delete and easy to refactor, so long as effort was put into writing it well. Of course, the patterns have a purpose and do help when used nicely - a lot of code you write will fall into some of those patterns even without you explicitly mentioning it. It's just...doing it for the sake of it is a problem.
  - evilduck 3y ago
    We call those architecture astronauts. The pile of abstractions stacks to the moon but when you chase down what's actually happening a 60 file repo ends up holding like 12 lines of actual programming.
- tetha 3y ago
CuriouslyC 3y ago
Duplication can sometimes be useful, for instance if you have many small variations on a central process. Trying to make one process with all the edge cases baked in leads to overly-complex, hard to reason about, expensive software. In my experience, the right way to handle this sort of situation is to create a functional mini-DSL for the process that handles all the implementation details, then create a "default" process which serves as a template. If a process needs slightly different logic, just copy the template, update the DSL to support any new logic, and update the template with the new DSL statements. This approach lets you give semantic meaning to implementation details, and you can see where all the different custom logic is at a glace by looking at all the template copies. As long as the template is only calling out to DSL actions with no internal logic of its own and process flow is correctly encapsulated in the DSL, you should never need to update templates to change behavior, only update the DSL.
- tracker1 3y ago
  This is generally my approach to data ingress/egress (ETL)... I'd rather have a hundred similar, small scripts for each data source than try to create one complex (monstrosity) application to handle them all.
- crazygringo 3y ago
  > is to create a functional mini-DSL Exactly. Is there a formal term for this? Instead of one gigantic function with 50 parameters, you have 100 "template" functions, that all make use 60 different "helper" functions (what you're calling the DSL). Instead of castles-of-logic abstraction, it's nuts-and-bolts or grass-roots abstractions. I've never come across a name for this development style. But it generally works extremely well when building processes for tens/hundreds of data formats or customers or what have you.
  - consilient 3y ago
    > I've never come across a name for this development style. Libraries designed like this are sometimes called "combinator libraries".
  - 3y ago
capableweb 3y ago
As a FYI, just as it's OK to abstract away duplication in code, it's OK to do the opposite, remove abstraction and add duplication. So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really). But, might be more work than it's worth. YMMV.
- patrick451 3y ago
  > As a FYI, just as it's OK to abstract away duplication in code, it's OK to do the opposite, remove abstraction and add duplication. > So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really). This sounds nice in theory, but the reality is that the effort required to make these two kinds of changes is not symmetric. It's about 10 times easier to get a PR approved and merged that combines similar looking code into a function than vise versa. If you any suspicion at all that an abstraction you're making may need to be removed and duplicated in the future, you're better of just never abstracting in the first place. It sucks pushing a change which unwinds an abstraction like that through code review. It's usually a lot easier to just never abstract it in the first place.
codegeek 3y ago
Also, you become a better programmer if you write duplicate code and then learn how to abstract it for cases that make sense. I also don't believe that dupe code is always a bad thing. Like everything else in software engineering, IT DEPENDS.
rdedev 3y ago
You got a good point about code evolution. Has anyone taken a look at it from a biological perspective? Seems like such problems can occur in genetics and nature might have come up with some tricks we can use
bena 3y ago
The answer, as always, is "sometimes". Sometimes duplication is cheaper than the wrong abstraction. And Sometimes it's better to abstract away a duplication rather than let it lie. And that's the mark of becoming a master at the craft. Being able to recognize all of these various slight permutations of state and what to do about them.
- rightbyte 3y ago
  Rule of thumbs really need to be told like this. Or they will be missused. Either by newbies that doesn't know any better or unpleasant programmers that will show their dogmatic beliefs down your throat with the common wisdom as excuse.
bheadmaster 3y ago
There's a very good quote on a programming blog [0] that I enjoyed reading: Repeat yourself to avoid creating dependencies, but don’t repeat yourself to manage them. [0] https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to https://programmingisterrible.com/post/139222674273/write-co...
msluyter 3y ago
I think one example where duplication > abstraction is in tests. I personally find tests that have a ton of extra helper classes/functions to do stuff like set up fixtures or do assertions to be painful to deal with. Taken to an extreme you end up with a mini test framework that obscures the actual test cases and is as hard to understand as the code in question. I'm not against shared test fixtures or some utility functions, but IMHO, it's better to have some duplication but clearer tests.
- echelon 3y ago
  Fully agree. I would add that you should duplicate the common, cross-cutting setup (eg. faked/mocked dependencies that don't matter), but make the test conditions themselves explicit. You get a feel for the correct granularity the more tests you write within the codebase. If you try to be too clever in saving boilerplate, you'll cause pain for future modifications and maintainers. Sometimes fixing "clever" tests takes longer than the code change itself.
- abcdaiojjdfoj 3y ago
  I like it when you have nice, composable utility functions. Ideally each test contains a short preamble setting up the appropriate context for the test to run. The preamble elucidates what the tests are actually testing. It can also serve as documentation on how to use those functions. There will probably be some duplication across tests, but if the utility functions are idempotent/composable, they're usually pretty easy to read/understand and equally mechanical to write/update.
- ezekg 3y ago
  > I personally find tests that have a ton of extra helper classes/functions to do stuff like set up fixtures or do assertions to be painful to deal with. I think it depends on the context. For example, I typically agree, but when I was writing authz tests [0], I ended up writing a DSL so that 1) I'd more more inclined to write the thousands and thousands of tests, and 2) I'd be able to focus on the actual authz assertion and not on verbose setup. I couldn't imagine writing those policy tests without that abstraction. I would have lost my mind with all of the repetition, and would have almost assuredly made mistakes. [0]: https://github.com/keygen-sh/keygen-api/blob/master/spec/policies/license_policy_spec.rb https://github.com/keygen-sh/keygen-api/blob/master/spec/pol...
yellowapple 3y ago
> An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. In that case, my takeaway would be that it ain't the abstraction itself that's wrong, but the unwillingness to get rid of it (or decompose it) when it no longer serves its purpose.
bcrosby95 3y ago
Given a long enough timeline, every abstraction turns wrong. The answer isn't to not abstract, the answer is to tear it out when it turns wrong. That was actually the original point of the popular article that streamlined this view - that we shouldn't be afraid of tearing them out, not that we shouldn't make them in the first place. Most people just read headlines though.
- lolinder 3y ago
  The resistance to tearing out a bad abstraction isn't just cultural: combining two different functions into one is a lossy operation, which makes splitting an abstraction harder than creating it in the first place. While the functions are distinct the call sites are self-documenting. You know which calls are for which purpose because the names are different. After combining them to deduplicate the code, you've lost that information, and to disentangle the abstraction now requires you to infer and reintroduce that lost information. It's not that it can't be done, but there is real friction that doesn't just exist in people's heads.
  - abathur 3y ago
    I think the difficulty of making the right decisions without this lost information is well-observed. I wrote a short post in roughly this idea space last year: https://t-ravis.com/post/doc/what_functions_and_why_functions/ https://t-ravis.com/post/doc/what_functions_and_why_function... It feels like the same thread you're describing, but I guess it's pulling on the other end of it. It's thinking about how to name things in a way that makes it easier to see that the implementations might diverge later, and simplify actually doing so (by preserving more of this intentional context).
- kiitos 3y ago
  Tearing out an abstraction requires a lot of knowledge about the abstraction and how it is used throughout the system. Over time, statistically zero maintainers will have that level of knowledge about any non-trivial system. In contrast, duplicated code may be annoying, but it is usually simple to understand and maintain by anyone with knowledge of the project and the programming language. Leaky abstractions are way way way worse than no abstraction at all.
Gibbon1 3y ago
My problem always is often when writing a function to remove duplication brings up the question of where to put it. If its only called inside one module doesn't matter really. But if not you've created a dependency. Which is bad. I think how much you hate that may depend on your language and the program. Some big enterprise Java monolith is a garbage dump of thousands of small files. So who cares. In C without name spaces and the need for headers you care more.
schwartzworld 3y ago
The problem is that "duplication is cheaper than the wrong abstraction" is basically an excuse that lazy devs use not to engineer their code. The other one I hear a lot is "it's not realistic to reach 100% test coverage / type safety" when submitted code with `any` all over it and zero tests.
seadan83 3y ago
I think this is an example that highlights abstract vs generic code. An abstraction should have hidden those evolutions entirely, which potentially means there would have been two code paths behind the abstraction. Moving that logic to a generic procedure with full parameterization I wouldn't call an abstraction (it's code that has been made generic). Generic code is more complex. DRY is not about making everything that can be generic, generic - it's about making sure a single thing is done in one place (and only that one place, DRY & SRP go together).
devoutsalsa 3y ago
One of the biggest problems with deduplication is you can end up with shared code that's full of corner case handling for different situations. Then your nice shared example becomes a tangled rats nest you can't unravel.

__alias 3y ago

I buy into the same belief as you here, but I guess you could easily argue that you could create a suitable fitting abstraction earlier on with the understanding that you can "detach" them once the point that they're fundamentally different comes

consilient 3y ago
The point of abstraction is to reduce the number of concepts in play. If you're still tracking which old concept is "really" being used every time, you haven't actually abstracted over anything, you're just naming things badly.
- capableweb 3y ago
  > The point of abstraction is to reduce the number of concepts in play. I'm not sure I agree with this. For me, the point of abstraction is divide the number of concepts between the layers you introduce, effectively to hide concepts from the layers where you don't want to have to care about them. Often times, abstractions adds the total number of concepts at play, but hides them beneath/above the layers.
fluoridation 3y ago
The problem is that there's an impetus to continue working on top of established facilities, because it's usually incrementally less work than reworking a piece of code into something else. Plus it's difficult to recognize ahead of time when something is about to become a problem, rather than fix something that's already a problem.

peeters 3y ago

I try to think about whether two concepts are innately similar or incidentally similar. Computing compounding interest for a home equity loan and a mortgage might be innately similar. A desired change to one will probably make a desired change to the other. Computing growth of a fruit fly population and computing compounding interesting for a loan might be incidentally similar. Until you change your "computeExponentialGrowth" function to now handle occasional decimations from environmental sources, and anyone looking at the code wonders what the heck that looks like for a loan.

AnimalMuppet 3y ago
For a loan, maybe it looks like payments on the principal? But your overall point is very correct. Don't make an abstraction because of coincidence.
HWR_14 3y ago
As interest rates go back up, paying off (part of) a mortgage early might come back.
adammarples 3y ago
If you've got your abstractions correct, then the exponential growth term and the decimation term will be partial differentials which will compose together nicely

AnimalMuppet 3y ago

In that situation, the correct thing to do is, when the two pieces drift away from each other, to recognize that they are no longer the same abstraction and to break the connection. That may be painful - you have to look at everywhere that abstraction is used and figure out which thing it really is, and change the code to reflect it. But if that's going to happen, then in the early days, a little duplication was probably better.

deleted 3y ago
[deleted]

lukeramsden 3y ago

awkward 3y ago

A good example of this is operations type stuff, like the pile of shell scripts or terraform files or whatever that get used to deploy your app. These scripts benefit greatly from a one to one relationship between the thing you're creating and the written text describing it. Not having a situation where changing one thing breaks everything else is a huge help there.

MilStdJunkie 3y ago

I might have to say some unkind things here, but statements like: instead of “duplication is cheaper than the wrong abstraction”, I would say “duplication is cheaper than confusing code littered with conditional logic”. seems like it's looking at this problem from an extremely narrow context. The truth is that the phrase "wrong abstraction" is (more or less) unquantifiable, which makes the original phrase, as employed, sort of like a koan. It addresses the very human tendency to see patterns in noise, and our ability to "transmit" such hallucinations to other humans via natural language and other means. The closest I can get to - given my at-best-apprentice status as a formal programmer - is the quantitative test I developed for CCS (conditional content systems), where the abstraction lies in the SNS[1], and the de-duplication mechanism is applicability[2]. Since each applicability statement carries its own overhead, there's a limit on how much "abstraction" the model can take before it's using quantitatively more keystrokes than duplication. The test goes like this: take the flat text procedures for ALL the configurations, and add it together. Now, take the conditionalized, applicability-laden procedure that unifies the procedure, and measure its file size. If the latter is LARGER than the former, then you're using the wrong SNS/applicability model for rolling up this content. Thing is, this is inevitable if you throw enough dissimilar configurations at a CCS, because each configuration has its own overhead, and eventually that outpaces the content itself. You can address this in a bunch of ways - like adding a containing pseudo-product that has all the configurations inside of it - but the actual real Product Management might not let you build on the applicability like that, because the Product itself isn't sold that way. Any other abstraction isn't available to you, because in the end this is natural language, which - unlike structured language - resists first order abstractions really well. This is one of those instances where, yes, the abstraction of the SNS/Applicability is worse - quantifiably - than duplication. All that complexity would be better handled via version control fork/branch relationships - far outside of the realm of natural language. [1] standard numbering system, a sort of numeric designator of functional systems, the primary way that content is designated as semi-independent modules. [2] conditional "chunks" that turn on and off depending on the applicability statement

feoren 3y ago
It'd be wonderful if we could measure the utility of software engineering choices by counting keystrokes or measuring file sizes or putting them in a turbo encabulator and seeing which one has more modial interaction with its magneto-reluctance. Unfortunately, reality is just too complicated, with far too many tradeoffs to be balanced. I'd recommend deep thought and discussion about the domain over looking at a graph of your codebase's sinusoidal repleneration. > All that complexity would be better handled via version control fork/branch relationships Probably not.

mostlylurks 3y ago

> An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. This seems to be the underlying assumption behind most uses of the "duplication is cheaper than the wrong abstraction" quote, but the assumption is simply incorrect. You should almost never try to expand abstractions in this manner. If you don't treat the abstractions relating to the thing you want to change in your codebase as "the" place where you need to make your change, and instead eagerly make new abstractions and throw old ones away as required, you won't really run into this problem. In fact, this predominant mindset where creating abstractions is strongly discouraged leads to the very problem that mindset is based on, as it will simply encourage junior developers and the like to modify the existing abstraction, creating the aforementioned kind of mess where abstractions become complicated through repeated modification, instead of creating new abstractions when appropriate, because creating abstractions has a stigma attached to it. Additionally, if someone has made a "wrong" abstraction based on something silly like two pieces of code simply being similar in terms of their structure and those use cases start to drift apart, you should feel eager to simply split apart the abstraction, be it into bare implementations or two new abstractions, or any other combination. Abstractions are cheap as long as you don't give them special significance.

sweezyjeezy 3y ago
I think there's a middle ground here. The original quote does not mean DRY=bad, abstraction=bad. The point is there is a non-zero cost to these things. A bad abstraction can, as you say, accumulate to something terrible through inertia or inexperience. A bad abstraction, even if caught early, was probably not worthwhile - I mean, it took time just to make the original one right? This does not mean that we should be scared of abstraction in general, but in my opinion abstractions that are purely for the sake of reducing duplication should be viewed with an extra level of apprehension.
jameshart 3y ago
When an abstraction evolves to a point where it needs to be split into two separate implementations to meet diverging needs… you will need to replace that abstraction with duplication. Which is the right thing to do because that duplication is cheaper than maintaining the wrong abstraction. I think this post makes the mistake of thinking that the only way in which duplication comes up is that it is discovered in the codebase, and we have the choice of abstracting it away or keeping it. On the contrary, duplication can - and should - be consciously introduced to fix bad abstractions when we find them in the codebase.

mdiesel 3y ago

Equality doesn't necessarily mean Equivalence.

feoren 3y ago

You're absolutely right that it's important to look beyond how two modules superficially look right now, and look instead at how they change. However, if you've always defined your abstractions based on what their consumers need rather than what their implementations have, then you shouldn't ever need to stretch them. They're not trying to "cover" both cases, they're trying to solve a problem that both cases have. Your two cases are not implementations of the abstraction, they are consumers of it. If one case grows to not have that problem, it just stops asking for that abstraction. If it grows to have more problems, it just asks for more abstractions. The original abstraction, if based on a common need, doesn't have to change. That's not to say abstractions never change -- they do. But they change because your understanding of the sub-problem they're solving has changed, not because their implementations or consumers have changed.

crabbone 3y ago

So... you kept modifying the two similar pieces of code until they became dissimilar. Why do you think that you wouldn't be able to modify the abstraction if you saw that it doesn't fit anymore?

convolvatron 3y ago
I think part of the issue here is that a fair number of programmers work in shops where they have very limited agency. They are tasked with making the minimum defensible change to add a feature or fix a bug. They are not allowed to change the tests or suggest refactoring. So those things just don't occur.
- crabbone 3y ago
  I see, but wouldn't this lack of agency, or, more precisely the inability to escalate the problem to someone with more edit rights be the actual issue? I could see this also happening in situations where such problems arise at component interface boundaries, where the pressure not to change comes not as an organizational policy but rather inability to influence external components (eg. the OS offers a poor abstraction -- the user-space program developer wouldn't be changing how OS works to rectify things). But, this is again sort of an administrative problem, because, ideally, the user-space program developer should be able to convince the OS developer to change the interface if it's found to be not a good abstraction... But, yes, I can see how in practice that'd be a very difficult thing to do.
  - convolvatron 3y ago
    I guess I was trying to claim that in a lot of places there isn't such a person. yeah - if an application company runs up against a poorly designed OS interface - they usually aren't aware, or just back away slowly. they don't have the scope or mandate to pursue it. that kind of behavior often extends to library dependencies, and even internal interfaces that the company should ostensibly own. its not so much an administrative problem as a desire to limit spending and time on software and do a kind of agile mvp glue job over an unbounded number of external dependencies. that often leads to an unmaintainable hairball. but if the alternative is to hire a bunch of really experienced developers and let them 'do the right thing' for 5 years... even if you do, there is a still a good chance that the output isn't gonna be that great. software people lost a lot of credibility with that model. we've probably swung too far in the other direction

ilyt 3y ago

> There are many instances I've encountered where two pieces of code coincided to look similar at a certain point in time. As the codebase evolved, so did the two pieces of code, their usage and their dependencies, until the similarity was almost gone. An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. Then you split that abstraction again. It's very cheap and very quick. Many people talk about the issue like it was an absolute in the code, but that's wrong approach. If you end up writing 4 functions that are the same, by all means, merge it into one. If then you need to add a parameter only this code path uses and rest doesn't care about, by all means split it back. Moving blocks of code around is cheap.

oxfordmale 3y ago
Splitting the abstraction is never cheap and quick, mostly because of politics. With duplicated code you often can assign a single responsible owner to each duplication. However, once abstracted, the code may suddenly be used by a number of different teams. You will need to get this work on their roadmap, increasing the friction to get this done. In many companies, this will also end up in endless discussions about the new approach.
- esafak 3y ago
  This is a good point. Following Conway's Law, a team may choose to duplicate code or do thing theoretically sub-optimally simply to avoid having to deal with other teams.
- yellowapple 3y ago
  Solution there would be to make the abstraction "opt-in", such that a team can elect to duplicate or abstract as desired. Also helps if the "main" abstraction is itself composed from smaller abstractions, from which downstream teams could then pick-and-choose rather than having to either fully abstract or fully duplicate.
jrumbut 3y ago
I think the key here is the oft repeated but often poorly understood maxim to favor composition ("has a") over inheritance ("is a"). If you have a mixin (or other means of composition) that you use in several places and one diverges, it's easy to remove it. If you use inheritance, it's going to be more painful. A language that offers OOP via prototypes instead of classes like JS can (sometimes) give you the best of both worlds, but it will confuse a lot of devs who aren't familiar with that kind of OO design.

BWStearns 3y ago

Agreed. Abstractions also tend to be more resistant to change, both from a technical level, and a social level. At a technical level an abstraction will have more call sites to worry about in different contexts, the more wrong the initial abstraction the harder it will be to change. The social level is maybe even more problematic. Abstractions seem more important than calling code and will experience more friction in code review. This change friction can also increase with the "wrongness" of the initial abstraction. The starting point makes less sense so a reviewer needs to work more to understand the context. If the abstraction is gnarly enough then it's possible that the reason for the abstraction is almost obscured. Even someone who knows _how_ it works might have lost the forest through the trees and push back on changes that simplify it or improve it if the change is a sufficiently large departure from the initial state. In this case you can often see small incremental changes get added easier but this just makes the shared code a bit gnarlier for next time.

ljm 3y ago
This is my beef with naively applied DDD, separation of concerns, and design patterns. Usually what happens is the 'clean' code ideal comes first, and then the implementation is squeezed into it. This then informs the organisation (or architecture) of the rest of the codebase and your software design has become a matter of putting pegs into the right-shaped holes. I have never found that kind of highly abstracted code easier to work with than some simple procedural alternative that is easy to delete and easy to refactor, so long as effort was put into writing it well. Of course, the patterns have a purpose and do help when used nicely - a lot of code you write will fall into some of those patterns even without you explicitly mentioning it. It's just...doing it for the sake of it is a problem.
- evilduck 3y ago
  We call those architecture astronauts. The pile of abstractions stacks to the moon but when you chase down what's actually happening a 60 file repo ends up holding like 12 lines of actual programming.
tetha 3y ago

CuriouslyC 3y ago

Duplication can sometimes be useful, for instance if you have many small variations on a central process. Trying to make one process with all the edge cases baked in leads to overly-complex, hard to reason about, expensive software. In my experience, the right way to handle this sort of situation is to create a functional mini-DSL for the process that handles all the implementation details, then create a "default" process which serves as a template. If a process needs slightly different logic, just copy the template, update the DSL to support any new logic, and update the template with the new DSL statements. This approach lets you give semantic meaning to implementation details, and you can see where all the different custom logic is at a glace by looking at all the template copies. As long as the template is only calling out to DSL actions with no internal logic of its own and process flow is correctly encapsulated in the DSL, you should never need to update templates to change behavior, only update the DSL.

tracker1 3y ago
This is generally my approach to data ingress/egress (ETL)... I'd rather have a hundred similar, small scripts for each data source than try to create one complex (monstrosity) application to handle them all.
crazygringo 3y ago
> is to create a functional mini-DSL Exactly. Is there a formal term for this? Instead of one gigantic function with 50 parameters, you have 100 "template" functions, that all make use 60 different "helper" functions (what you're calling the DSL). Instead of castles-of-logic abstraction, it's nuts-and-bolts or grass-roots abstractions. I've never come across a name for this development style. But it generally works extremely well when building processes for tens/hundreds of data formats or customers or what have you.
- consilient 3y ago
  > I've never come across a name for this development style. Libraries designed like this are sometimes called "combinator libraries".
- 3y ago

capableweb 3y ago

As a FYI, just as it's OK to abstract away duplication in code, it's OK to do the opposite, remove abstraction and add duplication. So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really). But, might be more work than it's worth. YMMV.

patrick451 3y ago
> As a FYI, just as it's OK to abstract away duplication in code, it's OK to do the opposite, remove abstraction and add duplication. > So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really). This sounds nice in theory, but the reality is that the effort required to make these two kinds of changes is not symmetric. It's about 10 times easier to get a PR approved and merged that combines similar looking code into a function than vise versa. If you any suspicion at all that an abstraction you're making may need to be removed and duplicated in the future, you're better of just never abstracting in the first place. It sucks pushing a change which unwinds an abstraction like that through code review. It's usually a lot easier to just never abstract it in the first place.

codegeek 3y ago

Also, you become a better programmer if you write duplicate code and then learn how to abstract it for cases that make sense. I also don't believe that dupe code is always a bad thing. Like everything else in software engineering, IT DEPENDS.

rdedev 3y ago

You got a good point about code evolution. Has anyone taken a look at it from a biological perspective? Seems like such problems can occur in genetics and nature might have come up with some tricks we can use

bena 3y ago

The answer, as always, is "sometimes". Sometimes duplication is cheaper than the wrong abstraction. And Sometimes it's better to abstract away a duplication rather than let it lie. And that's the mark of becoming a master at the craft. Being able to recognize all of these various slight permutations of state and what to do about them.

rightbyte 3y ago
Rule of thumbs really need to be told like this. Or they will be missused. Either by newbies that doesn't know any better or unpleasant programmers that will show their dogmatic beliefs down your throat with the common wisdom as excuse.

bheadmaster 3y ago

There's a very good quote on a programming blog [0] that I enjoyed reading: Repeat yourself to avoid creating dependencies, but don’t repeat yourself to manage them. [0] https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to https://programmingisterrible.com/post/139222674273/write-co...

msluyter 3y ago

I think one example where duplication > abstraction is in tests. I personally find tests that have a ton of extra helper classes/functions to do stuff like set up fixtures or do assertions to be painful to deal with. Taken to an extreme you end up with a mini test framework that obscures the actual test cases and is as hard to understand as the code in question. I'm not against shared test fixtures or some utility functions, but IMHO, it's better to have some duplication but clearer tests.

echelon 3y ago
Fully agree. I would add that you should duplicate the common, cross-cutting setup (eg. faked/mocked dependencies that don't matter), but make the test conditions themselves explicit. You get a feel for the correct granularity the more tests you write within the codebase. If you try to be too clever in saving boilerplate, you'll cause pain for future modifications and maintainers. Sometimes fixing "clever" tests takes longer than the code change itself.
abcdaiojjdfoj 3y ago
I like it when you have nice, composable utility functions. Ideally each test contains a short preamble setting up the appropriate context for the test to run. The preamble elucidates what the tests are actually testing. It can also serve as documentation on how to use those functions. There will probably be some duplication across tests, but if the utility functions are idempotent/composable, they're usually pretty easy to read/understand and equally mechanical to write/update.
ezekg 3y ago
> I personally find tests that have a ton of extra helper classes/functions to do stuff like set up fixtures or do assertions to be painful to deal with. I think it depends on the context. For example, I typically agree, but when I was writing authz tests [0], I ended up writing a DSL so that 1) I'd more more inclined to write the thousands and thousands of tests, and 2) I'd be able to focus on the actual authz assertion and not on verbose setup. I couldn't imagine writing those policy tests without that abstraction. I would have lost my mind with all of the repetition, and would have almost assuredly made mistakes. [0]: https://github.com/keygen-sh/keygen-api/blob/master/spec/policies/license_policy_spec.rb https://github.com/keygen-sh/keygen-api/blob/master/spec/pol...

yellowapple 3y ago

> An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions. In that case, my takeaway would be that it ain't the abstraction itself that's wrong, but the unwillingness to get rid of it (or decompose it) when it no longer serves its purpose.

bcrosby95 3y ago

Given a long enough timeline, every abstraction turns wrong. The answer isn't to not abstract, the answer is to tear it out when it turns wrong. That was actually the original point of the popular article that streamlined this view - that we shouldn't be afraid of tearing them out, not that we shouldn't make them in the first place. Most people just read headlines though.

lolinder 3y ago
The resistance to tearing out a bad abstraction isn't just cultural: combining two different functions into one is a lossy operation, which makes splitting an abstraction harder than creating it in the first place. While the functions are distinct the call sites are self-documenting. You know which calls are for which purpose because the names are different. After combining them to deduplicate the code, you've lost that information, and to disentangle the abstraction now requires you to infer and reintroduce that lost information. It's not that it can't be done, but there is real friction that doesn't just exist in people's heads.
- abathur 3y ago
  I think the difficulty of making the right decisions without this lost information is well-observed. I wrote a short post in roughly this idea space last year: https://t-ravis.com/post/doc/what_functions_and_why_functions/ https://t-ravis.com/post/doc/what_functions_and_why_function... It feels like the same thread you're describing, but I guess it's pulling on the other end of it. It's thinking about how to name things in a way that makes it easier to see that the implementations might diverge later, and simplify actually doing so (by preserving more of this intentional context).
kiitos 3y ago
Tearing out an abstraction requires a lot of knowledge about the abstraction and how it is used throughout the system. Over time, statistically zero maintainers will have that level of knowledge about any non-trivial system. In contrast, duplicated code may be annoying, but it is usually simple to understand and maintain by anyone with knowledge of the project and the programming language. Leaky abstractions are way way way worse than no abstraction at all.

Gibbon1 3y ago

My problem always is often when writing a function to remove duplication brings up the question of where to put it. If its only called inside one module doesn't matter really. But if not you've created a dependency. Which is bad. I think how much you hate that may depend on your language and the program. Some big enterprise Java monolith is a garbage dump of thousands of small files. So who cares. In C without name spaces and the need for headers you care more.

schwartzworld 3y ago

The problem is that "duplication is cheaper than the wrong abstraction" is basically an excuse that lazy devs use not to engineer their code. The other one I hear a lot is "it's not realistic to reach 100% test coverage / type safety" when submitted code with `any` all over it and zero tests.

seadan83 3y ago

I think this is an example that highlights abstract vs generic code. An abstraction should have hidden those evolutions entirely, which potentially means there would have been two code paths behind the abstraction. Moving that logic to a generic procedure with full parameterization I wouldn't call an abstraction (it's code that has been made generic). Generic code is more complex. DRY is not about making everything that can be generic, generic - it's about making sure a single thing is done in one place (and only that one place, DRY & SRP go together).

devoutsalsa 3y ago

One of the biggest problems with deduplication is you can end up with shared code that's full of corner case handling for different situations. Then your nice shared example becomes a tangled rats nest you can't unravel.