6 ms·

I've seen excessive abstraction kill projects time and time again. The crime scene is similar in every case: - What if we ever need to change database server o

by hu3 8y ago

I've seen excessive abstraction kill projects time and time again. The crime scene is similar in every case: - What if we ever need to change database server or driver? Proceeds to write a layer to abstract data access - We might have different login forms one day? LoginFormFactory it is - This code hits our KV Redis/memcache/NoSQL by calling the driver lib directly? Can't have that, I'll write a CacheStorage or DocumentStorage layer. Over time, this defensive mentality produces codebases that are so bloated that no one in the team can confortably grasp all the moving parts despite the project not being rocket science. At that point, devs rather quit or get a substantial raise in order to continue working and ultimately endup leaving to start a new project. But this time, they'll be sure to not make the same mistakes. This time, they'll abstract even more so "when one part of the project becomes bloated, it can be easily rewritten thanks to the abstractions". And the cycle continues.

jstimpfle 8y ago
Which is not to say that we shouldn't wrap dependencies. We absolutely should. But by default, the wrapper should not offer an abstraction that is so flexible that the dependency can be exchanged. Instead, it should start with the actual problem, by offering only abstractions that (a) make sense in the context of the program and (b) translate well to concepts of the dependency/library. For example, hiding POSIX-style select/poll vs Win32 Completion ports behind a common abstraction is something I tried recently, and seems to be HARD. Did not complete. Why? I started with the abstraction without being familiar with what the problem was. All that I knew was that I wanted to experiment with "game engine design".
- Nursie 8y ago
  > Which is not to say that we shouldn't wrap dependencies. We absolutely should. Why? If I'm writing something for a known environment that's unlikely to change, it seems like a waste of time.
  - jstimpfle 8y ago
    Even then, often there will be small bits that don't fit well. By wrapping everything you make the model and assumptions of your application explicit. Code that is written against your own model (which is almost always simpler than what the library offers) will be more understandable. I really like the Dijkstra quote, "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." Usually I even wrap libc and POSIX functionality in my C programs. Advantages include (1) not having to mess with specific types, such as size_t, time_t, DIR*, and so on -- which often don't fit my own model (e.g. dealing with unsigned size_t vs signed integers is painful), and often need small fixes to be portable. (2) compiles a lot faster since I don't need to include 10000s of lines from system header files (3) not wrapping makes huge portions of the project dependent on the library, instead of only the wrapper. (4) You tightly control access to the library, which makes it easier to troubleshoot. You can also make up your own strategy for managing library objects in a central place.
    - pishpash 8y ago
      Yes, a library is already an abstraction, just somebody else's. Thread parent is complaining about bad design that doesn't model the problem, not abstraction.
      - jstimpfle 8y ago
        What do you mean? Bad design in the library? In a wrapper? What's the difference between "design" and "abstraction" here?
    - Nursie 8y ago
      I think this is nonsense and leads to enterprise-y type patterns and hard to understand code. Wrapping things for the sake of it is a recipe for unnecessary verbosity and a hard to learn codebase as things are obfuscated and take longer to trace through. In C, your point 1 makes me think your code is unsafe, if you're having trouble keeping track of types and signedness. Point 2 is fair enough, but I think a lot of C headers are granular enough that it's irrelevant, especially given how fast everything is these days. Gone are the 8 hour builds of old. Point 3 is precisely the type of premature interface-isation that's the problem here. If you're never going to change it, then it doesn't matter whether huge portions of the code rely on the library or your wrapper. It's basically the same thing. Only you've put in extra work to wrap something in another layer for no gain. 4 much the same.
- z3t4 8y ago
  Dependencies should be treated like if it was your own code. After you have replaced some API for the third time you can start thinking about wrapping it.
  - drawkbox 8y ago
    Or if you start with multiple wrapped dependencies from the start, the library is frequently updated and/or you touch the library in many places in your code across multiple projects especially, those are good places for wrappers/facades/adapters. An example would be in games when you submit an achievement or leaderboard score to a third party such as Apple GameCenter, Google Play Game Services, or other systems like Amazon Game Circle, web or custom. These libraries are frequently updated, they change per platform and you will touch them many places in the codebase, so wrapping them in a progression/statistics/achievements wrapper/facade/adapter is smart from the start.
    - z3t4 8y ago
      Even if you have wrapped it you still have to update your code when there are breaking changes. If you follow "don't repeat yourself" (DRY) there will not really be more work updating, compared to if you had wrapped it. (You should however not follow DRY slavishly - but that is a discussion on it's own.)
      - drawkbox 8y ago
        If the wrapper/library is generic enough then it can be less work across projects and codebase and won't need to change with an actively updated library that is wrapped. If the abstraction wrapper is tightly coupled, uses third party library types that can change or leaky then yeah, the effort is moot and you end up with more work. Another case is where you have some sort of messaging system where you want responses/events to be uniform to your system rather than unique to a platform or even a company/product that doesn't match your codebase. An example might be wrapping a game recording library, audio library or something that does not fit well in the codebase or complicates maintenance, even stylistically/standards, or you only need small part of the library like activating it, common types messaging or data level. Game engines themselves are massive wrappers around many wrapped systems.
ses1984 8y ago
>I've seen excessive abstraction kill projects time and time again Agreed. But none of your first few examples seem particularly egregious, if done well, they don't seem like they would sink a project. I think part of the problem is developers underestimating just how hard it is to write an abstraction layer well. The juniors will say, "yeah I could do that in one sprint" then they start to build on top of it, the problems don't start to show until later, maybe when the dev who wrote it is gone. I think the correct way to approach a problem like this is to acknowledge the cost of it up front. They are expensive. They don't just have to work. They have to work well and be readable.
seanmcdirmid 8y ago
Abstraction isn’t always applied for rewrite flexibility reasons, in fact I claim it rarely is. Often it is used primarily as a way to decompose complexity: I want to write this component without deeplh considering specific interdependencies so I’ll just abstract those away and get on with it. The code will only be composed with specific things, but considering all those interactions up front is a PITA. Overly abstract code could just be a sign of a programmer working through a problem that was not well understood (either by everyone or just themselves). The second or third time they do the same task, the code is bound to be less abstract because they are more comfortable in dealing with interdependencies up front. Suggesting refactoring or rewrites can work well in that case.
Pxtl 8y ago
To me the value of abstractjng out those services is testability - if they're abstracted, they can be mocked. The smaller the abstraction, the smaller the surface area you have to mock.
- HereBeBeasties 8y ago
  The smaller the abstraction, the more complex the emergent behaviour is when you combine all the itty-bitty little abstractions together, and the harder it is to write tests that cover real use cases instead of testing implementation details, which is another way of saying "your tests will be brittle". You can band-aid this to an extent with higher-level "integration" tests, but if you'd done things at the right level of abstraction in the first place you would carry less weight around and wouldn't have to maintain a bunch of brittle tests in the first place. This is obviously all shades of grey, but if you're mocking out things that aren't I/O you're probably doing it wrong. If you find yourself vehemently disagreeing with this I'd be interested to know if you've ever had to refactor or simplify a codebase with a bunch of overly-abstracted, itty-bitty things that had very tight coupling via mock behaviour to all their tests, and if so, whether that felt pleasant to you or not. If you haven't then you probably haven't seen the considerable longterm maintenance downsides to this kind of approach and I feel sorry for the poor folk who will inherit your codebase. Also curious is that many codebases I come across that look like this often have terrible copy/paste mock setup across lots of tests, making the issue even worse when you want to change things. Those sorts of codebases often end up with people wrapping the abstractions in other abstractions because they're sufficiently resistant to change as a result that that seems easier. This obviously makes everything even more resistant to change (especially as the wrapper abstraction usually depends deeply on all the behaviour underneath it, and the mock set-up to test the wrapper ends up as an exercise in mentally mismodelling how the other components actually behave).
  - crdoconnor 8y ago
    >The smaller the abstraction, the more complex the emergent behaviour is when you combine all the itty-bitty little abstractions together That's not necessarily true. Abstractions with a small surface area (exposure to their 'outside world' - e.g. via function signatures) that are very deep (they hide a lot of complexity) make more complex behaviour much easier to manage. When the surface area is high and the depth is low is when the overhead of the abstraction tends to exceed its use value.
slack3r 8y ago
Abstraction is useful when the resulting code is logically simpler, much like say category theory makes certain results in algebraic topology seem obvious. Perhaps, you simply mean a different thing when you say "abstraction".
ken 8y ago
I confess I've never seen that happen, but I do think excessive abstraction is killing the argument this article is trying to make. When would anyone ever make an inc_pair function? Or a hierarchy of 'animals that lay eggs'? I guess these are supposed to be examples of something, but I'm not sure what. I wouldn't even call them "abstraction". They seem like completely hypothetical examples of how to apply programming language features to non-problems in bad ways. Without knowing what the designer of these programs is trying to model, it's impossible for me to understand what they're trying to accomplish or why they think applying these language features in this way makes sense at all. Both of these examples sound like they increase the number of lines of code, for no appreciable benefit, and I recall Yegge's old observation that the main issue with any codebase is simply the bulk of it.
- Tarean 8y ago
  On the other hand, the lens library in haskell is extremely abstract. Libraries obviously have different tradeoffs but using lens can increase the onboarding cost significantly for new team members. I am mentioning lens because it has a both function: incBoth = both += 1 Which has a fun type signature: incBoth :: (Bitraversal p, MonadState (p a a) m, Num a) => m ()
  - tome 8y ago
    Hmm, don't you use it as a monadic action like do ... both += 1
- dllthomas 8y ago
  > completely hypothetical examples of how to apply programming language features to non-problems in bad ways. If that were Randall Munroe's next book, I'd buy a copy.
Tarean 8y ago
I don't see the issue with an interfave between the app and the database. As long as the interface is getUserById and not compileSQLQuery.

by hu3 8y ago

jstimpfle 8y ago

Which is not to say that we shouldn't wrap dependencies. We absolutely should. But by default, the wrapper should not offer an abstraction that is so flexible that the dependency can be exchanged. Instead, it should start with the actual problem, by offering only abstractions that (a) make sense in the context of the program and (b) translate well to concepts of the dependency/library. For example, hiding POSIX-style select/poll vs Win32 Completion ports behind a common abstraction is something I tried recently, and seems to be HARD. Did not complete. Why? I started with the abstraction without being familiar with what the problem was. All that I knew was that I wanted to experiment with "game engine design".

Nursie 8y ago
> Which is not to say that we shouldn't wrap dependencies. We absolutely should. Why? If I'm writing something for a known environment that's unlikely to change, it seems like a waste of time.
- jstimpfle 8y ago
  Even then, often there will be small bits that don't fit well. By wrapping everything you make the model and assumptions of your application explicit. Code that is written against your own model (which is almost always simpler than what the library offers) will be more understandable. I really like the Dijkstra quote, "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." Usually I even wrap libc and POSIX functionality in my C programs. Advantages include (1) not having to mess with specific types, such as size_t, time_t, DIR*, and so on -- which often don't fit my own model (e.g. dealing with unsigned size_t vs signed integers is painful), and often need small fixes to be portable. (2) compiles a lot faster since I don't need to include 10000s of lines from system header files (3) not wrapping makes huge portions of the project dependent on the library, instead of only the wrapper. (4) You tightly control access to the library, which makes it easier to troubleshoot. You can also make up your own strategy for managing library objects in a central place.
  - pishpash 8y ago
    Yes, a library is already an abstraction, just somebody else's. Thread parent is complaining about bad design that doesn't model the problem, not abstraction.
    - jstimpfle 8y ago
      What do you mean? Bad design in the library? In a wrapper? What's the difference between "design" and "abstraction" here?
  - Nursie 8y ago
    I think this is nonsense and leads to enterprise-y type patterns and hard to understand code. Wrapping things for the sake of it is a recipe for unnecessary verbosity and a hard to learn codebase as things are obfuscated and take longer to trace through. In C, your point 1 makes me think your code is unsafe, if you're having trouble keeping track of types and signedness. Point 2 is fair enough, but I think a lot of C headers are granular enough that it's irrelevant, especially given how fast everything is these days. Gone are the 8 hour builds of old. Point 3 is precisely the type of premature interface-isation that's the problem here. If you're never going to change it, then it doesn't matter whether huge portions of the code rely on the library or your wrapper. It's basically the same thing. Only you've put in extra work to wrap something in another layer for no gain. 4 much the same.
z3t4 8y ago
Dependencies should be treated like if it was your own code. After you have replaced some API for the third time you can start thinking about wrapping it.
- drawkbox 8y ago
  Or if you start with multiple wrapped dependencies from the start, the library is frequently updated and/or you touch the library in many places in your code across multiple projects especially, those are good places for wrappers/facades/adapters. An example would be in games when you submit an achievement or leaderboard score to a third party such as Apple GameCenter, Google Play Game Services, or other systems like Amazon Game Circle, web or custom. These libraries are frequently updated, they change per platform and you will touch them many places in the codebase, so wrapping them in a progression/statistics/achievements wrapper/facade/adapter is smart from the start.
  - z3t4 8y ago
    Even if you have wrapped it you still have to update your code when there are breaking changes. If you follow "don't repeat yourself" (DRY) there will not really be more work updating, compared to if you had wrapped it. (You should however not follow DRY slavishly - but that is a discussion on it's own.)
    - drawkbox 8y ago
      If the wrapper/library is generic enough then it can be less work across projects and codebase and won't need to change with an actively updated library that is wrapped. If the abstraction wrapper is tightly coupled, uses third party library types that can change or leaky then yeah, the effort is moot and you end up with more work. Another case is where you have some sort of messaging system where you want responses/events to be uniform to your system rather than unique to a platform or even a company/product that doesn't match your codebase. An example might be wrapping a game recording library, audio library or something that does not fit well in the codebase or complicates maintenance, even stylistically/standards, or you only need small part of the library like activating it, common types messaging or data level. Game engines themselves are massive wrappers around many wrapped systems.

ses1984 8y ago

>I've seen excessive abstraction kill projects time and time again Agreed. But none of your first few examples seem particularly egregious, if done well, they don't seem like they would sink a project. I think part of the problem is developers underestimating just how hard it is to write an abstraction layer well. The juniors will say, "yeah I could do that in one sprint" then they start to build on top of it, the problems don't start to show until later, maybe when the dev who wrote it is gone. I think the correct way to approach a problem like this is to acknowledge the cost of it up front. They are expensive. They don't just have to work. They have to work well and be readable.

seanmcdirmid 8y ago

Abstraction isn’t always applied for rewrite flexibility reasons, in fact I claim it rarely is. Often it is used primarily as a way to decompose complexity: I want to write this component without deeplh considering specific interdependencies so I’ll just abstract those away and get on with it. The code will only be composed with specific things, but considering all those interactions up front is a PITA. Overly abstract code could just be a sign of a programmer working through a problem that was not well understood (either by everyone or just themselves). The second or third time they do the same task, the code is bound to be less abstract because they are more comfortable in dealing with interdependencies up front. Suggesting refactoring or rewrites can work well in that case.

Pxtl 8y ago

To me the value of abstractjng out those services is testability - if they're abstracted, they can be mocked. The smaller the abstraction, the smaller the surface area you have to mock.

HereBeBeasties 8y ago
The smaller the abstraction, the more complex the emergent behaviour is when you combine all the itty-bitty little abstractions together, and the harder it is to write tests that cover real use cases instead of testing implementation details, which is another way of saying "your tests will be brittle". You can band-aid this to an extent with higher-level "integration" tests, but if you'd done things at the right level of abstraction in the first place you would carry less weight around and wouldn't have to maintain a bunch of brittle tests in the first place. This is obviously all shades of grey, but if you're mocking out things that aren't I/O you're probably doing it wrong. If you find yourself vehemently disagreeing with this I'd be interested to know if you've ever had to refactor or simplify a codebase with a bunch of overly-abstracted, itty-bitty things that had very tight coupling via mock behaviour to all their tests, and if so, whether that felt pleasant to you or not. If you haven't then you probably haven't seen the considerable longterm maintenance downsides to this kind of approach and I feel sorry for the poor folk who will inherit your codebase. Also curious is that many codebases I come across that look like this often have terrible copy/paste mock setup across lots of tests, making the issue even worse when you want to change things. Those sorts of codebases often end up with people wrapping the abstractions in other abstractions because they're sufficiently resistant to change as a result that that seems easier. This obviously makes everything even more resistant to change (especially as the wrapper abstraction usually depends deeply on all the behaviour underneath it, and the mock set-up to test the wrapper ends up as an exercise in mentally mismodelling how the other components actually behave).
- crdoconnor 8y ago
  >The smaller the abstraction, the more complex the emergent behaviour is when you combine all the itty-bitty little abstractions together That's not necessarily true. Abstractions with a small surface area (exposure to their 'outside world' - e.g. via function signatures) that are very deep (they hide a lot of complexity) make more complex behaviour much easier to manage. When the surface area is high and the depth is low is when the overhead of the abstraction tends to exceed its use value.

slack3r 8y ago

Abstraction is useful when the resulting code is logically simpler, much like say category theory makes certain results in algebraic topology seem obvious. Perhaps, you simply mean a different thing when you say "abstraction".

ken 8y ago

I confess I've never seen that happen, but I do think excessive abstraction is killing the argument this article is trying to make. When would anyone ever make an inc_pair function? Or a hierarchy of 'animals that lay eggs'? I guess these are supposed to be examples of something, but I'm not sure what. I wouldn't even call them "abstraction". They seem like completely hypothetical examples of how to apply programming language features to non-problems in bad ways. Without knowing what the designer of these programs is trying to model, it's impossible for me to understand what they're trying to accomplish or why they think applying these language features in this way makes sense at all. Both of these examples sound like they increase the number of lines of code, for no appreciable benefit, and I recall Yegge's old observation that the main issue with any codebase is simply the bulk of it.

Tarean 8y ago
On the other hand, the lens library in haskell is extremely abstract. Libraries obviously have different tradeoffs but using lens can increase the onboarding cost significantly for new team members. I am mentioning lens because it has a both function: incBoth = both += 1 Which has a fun type signature: incBoth :: (Bitraversal p, MonadState (p a a) m, Num a) => m ()
- tome 8y ago
  Hmm, don't you use it as a monadic action like do ... both += 1
dllthomas 8y ago
> completely hypothetical examples of how to apply programming language features to non-problems in bad ways. If that were Randall Munroe's next book, I'd buy a copy.

Tarean 8y ago

I don't see the issue with an interfave between the app and the database. As long as the interface is getUserById and not compileSQLQuery.