19 ms·

Abstraction is expensive

224 points by pclmulqdq 4y ago

warrenm 4y ago
This sounds like an application risk-aversion/-seeking that Kahneman highlights in "Thinking, Fast and Slow" Specifically... each of these individual decisions (wrt to abstraction, in this context) was made to locally optimize (speed, simplicity, etc) some issue or other But when taken in gestalt, they're overall not only not optimal, but downright bad Most of those decisions are made by people far to close to a tiny portion of the problem to understand the implications of their choices Alternatively, it's an application of the blind mice finding an elephant story: yes, it's like a rope, a fan, a tree, a hose, and on and on But it's none of those, it's an elephant
- projektfu 4y ago
  The title needs work too. People will undoubtedly assume that reducing abstraction will make things cheaper.
  - sghio4Q2 4y ago
    Indeed.
  - cryptonector 4y ago
    Eh, some people might do that, but, really, titles like that beg you to read the article (or at least it's why I read it). The article absolutely does not say anything like "reducing abstraction will make things cheaper".
  - danShumway 4y ago
    Sometimes reducing abstractions will make things cheaper. Some regular traps that I see people run into with abstractions: - building around scenarios that they can be pretty confident that they will never need to handle. - building abstractions that are larger and more complicated than the thing they are abstracting (this tends to happen when people build abstractions of abstractions). - building abstractions before they have a proper understanding of what they're abstracting and what they'll need to encapsulate. In many of those cases, reducing abstractions (even just temporarily) reduces complexity. When I'm building purely personal projects, I don't use abstractions to help me deal with directory structure on Windows, because I don't use Windows, and that would be additional complexity for no benefit -- so it's simpler for me to just work with the lower level OS paths. When I start writing an abstraction I look at the amount of documentation I'm generating, and if I'm generating more documentation than it would take to explain the underlying system, I look to see if there are concepts that I can remove. I worked at a company where our build process became considerably simpler when we stopped using high-level build tools like Gulp/Grunt and switched to writing simple Node scripts, because it was easier to debug what those scripts were doing. We simplified that process even further by occasionally just dipping into Bash scripts. Working with low-level concepts for a while often also gives you better understanding of what you need to abstract. I've worked with codebases where the abstractions all get built first, and it's not uncommon for those abstractions to be built around tasks that are pretty simple and easy to do with lower-level code, at the same time that the abstractions completely ignore the really difficult tasks that are very annoying to do. And once abstractions get baked in, it was time consuming and difficult and expensive to pull them out and rewrite them. Going back to the scripts above, our build process got better because we made it simple to begin with -- a set of scripts, rather than a large established pipeline -- and then as we identified pain points, we started abstracting those pain points away. That allowed us to not waste time rewriting abstractions over and over and instead to have targeted small interfaces that helped us with the actual painful parts of building and deployment. It is surprisingly common for software to be over-abstracted to the point where it is more complicated to deal with than it would be otherwise. I mean, heck, this comes up in web development all the time, it is one of the primary criticisms people have of the JS ecosystem -- that it overcomplicates development. In many cases those complications exist for reasons, they solve real problems that people have had. But also in many cases, someone's individual blog doesn't need any of that, and it would be cheaper and easier for them to build something smaller and simpler. If I can build a site that is one HTML file and one CSS file, and I know that's all I'm going to need, then it's overkill to try and set up a bunch of abstractions on top of that.
- hinkley 4y ago
  You’re speaking to impedance mismatch which is one of the classic blunders. Systems that don’t use the same jargon as the user base end up having substantial bugs, especially ones the authors insist are features. The abstractions are a bad fit for the problem domain and they break things. I’m in a project that did that massively, at the hands of both architectural astronauts and another common blunder: people who seemed to think the concerns of the user base are beneath them tend to create a fantasy world for themselves where they can pretend they work on something more esoteric than the petty concerns of the people who pay their salaries. Because of the bullshit (and more importantly, the top-heavy sources of that bullshit) we’ve lost a lot of the better people who could have fixed that situation. Meanwhile we are also trying to expand into a new industry, and I can’t help but wonder if we would have been in trouble even if our abstractions had matched our core competency. I would have needed all of those missing people to make the refactors necessary to do that work, so the agents of chaos and delusion are a little vindicated. I have a couple of things I want to button up but they are all fast approaching. I don’t expect to be here in six months. I’ve started fantasizing about my exit interview. In which I will probably suggest that my team is too powerful and isolated from the end user and so needs to be disbanded, its members dispersed into groups one and two steps closer to the users. So as to concretize some of the code and confront them with the day to day struggles of internal customers, dealing with the batshit parts of the code.
nine_k 4y ago
Lack of abstraction is also expensive: try writing something large in assembly. I'd say that lack of a language / system of notions adequate to the subject area is expensive. The desire to describe things in a way that's efficient for a particular class of problems leads to invention of various frameworks. Say, Rails makes you hugely productive at solving a particular type of problems (see [Shopify]), though it's less than helpful if you try to apply it to unfitting problems (see [Twitter]). [Shopify]: https://tomaszs2.medium.com/how-shopify-handled-1-27-million-requests-s-during-black-friday-cyber-monday-8a4b1d0dd1f9 https://tomaszs2.medium.com/how-shopify-handled-1-27-million... (Sorry for a Medium link) [Twitter]: https://www.theregister.com/2012/11/08/twitter_epic_traffic_saved_by_java/ https://www.theregister.com/2012/11/08/twitter_epic_traffic_...
- ghaff 4y ago
  The headline is sort of misleading. It is (appropriately) primarily about inappropriate abstractions. As you say, some abstractions are unavoidable. Even assembly language is an abstraction.
  - cryptonector 4y ago
    That's why real programmers write binary object code directly.
    - nine_k 4y ago
      Don't tell them about the lost art of loading microcode on boot.
    - fnordpiglet 4y ago
      I wire my magnetic core with artisanal copper.
      - choeger 4y ago
        You use copper? These kids today...
pca006132 4y ago
The problem is that application performance doesn't only depend on the thing you want to do, but also depends on the low level implementation details and workload, which the programmer may not fully understand or in control of. Some operation may be optimal when executed alone but exists better alternative for batch processing. Some programs may be optimized for throughput but is really bad for low latency application. The designer have to consider these when designing their system, and compilers/library writers can't help much except write a more comprehensive documentation which the programmer will probably not read :). I think abstraction guides implementation, so it is very beneficial to think about what will be the bottleneck in the application, and if a certain abstraction will hinder the performance or prohibit future optimization. And although a low level abstraction can in theory allow for better performance if the programmer spend enough time to optimize for it, they often won't and a simpler interface (with sane implementation and perhaps escape hatches for performance tuning) may allow for better performance with much less effort.
- AnimalMuppet 4y ago
  > I think abstraction guides implementation, so it is very beneficial to think about what will be the bottleneck in the application I think this way, but in terms of the "performance" of writing the application. What makes it hard to write this program? For example, in high-performance computing, you may need to control memory alignment (for SIMD and cache reasons). That makes it harder to write a program that just implements an algorithm. So I think you should identify what it is that makes the program hard to write, and pick abstractions that help with that, that give you easier ways of controlling or managing that hard part.
mattacular 4y ago
One of my favorite quotes about software engineering, sorry not sure who to attribute it to: "First you learn the value of abstraction, then you learn the cost of abstraction, then you're ready to engineer"
- pphysch 4y ago
  Junior, mid, senior? Explains why the mid phase is brief.
fnordpiglet 4y ago
A CS prof of mine liked to say: Every problem in computer science can be solved with abstraction other than too much abstraction.
cloogshicer 4y ago
Even though many people call abstraction 'the essence of programming', it is rarely fully understood. Maybe no human on the planet really understands it. I've long had an essay in me about this topic that I really need to put to paper soon. Here is the brief version of that essay: * Programming is about building theories/models of the problem. Source code largely has no value by itself [1] * It is very difficult, if not impossible, to transfer these models/theories between humans (see the essay I linked below for details) * Because of this difficulty, programming is essentially teaching (when writing program code, we're teaching other people of the problem domain, and models that fit that problem domain) * Another way to phrase this: All programming is building user interfaces. When you're writing, say, a function, you're writing a more abstract UI for the next programmer or yourself * The tools we have for writing these UIs are terrible - taken the above example of writing a function, what are the tools you have to communicate this idea to the next person? A single string of characters (the function name). In a typed language you get a bit of extra info because of the type information (I believe this to be the main advantage of type systems). * The big question is: What would a better UI for communicating abstractions look like? * I don't know the answer to that question but I have a hunch that it has to be bi-directional. If you've ever worked with a GUI library that has a visual editor, you know how awful they are, unless there is also a representation of the same GUI in code. This bi-directional mapping of code and GUI makes it very easy to understand the two different ways of looking at the problem. I think something like this is needed down to the very lowest levels of abstractions. * Another way to phrase this: Imagine two different scientific models for the same problem. For example, for the model of an atom, its protons and neutrons, there is the more simple Bohr Model, which is completely wrong, given our current knowledge, but still very useful in many modern calculations. But in certain situations, a more accurate model is needed, which takes quantum mechanical effects into account. I see an analogy for programming here: In most cases, a simplified model suffices, but as more performance is needed, a more complex model is required. The question is, how can we easily teach someone the more difficult model, once they've understood the simple one? And the other way around (which is often also not easy, since simple is not the same as easy). If you have any thoughts on what I've written, any at all, I'd love to hear from you (I'll watch this thread, or find my email in my bio). [1] https://hiringengineersbook.com/post/autonomy/ https://hiringengineersbook.com/post/autonomy/
agentultra 4y ago
For the mathematically minded, this is not about abstractions. This is about technology choices! One more interpretation of an overloaded word that means a very specific thing.
- vlunkr 4y ago
  Agreed, they should probably include their definition of 'abstraction' in the post.
  - adamrezich 4y ago
    I would've thought that everyone who has done any programming at all would be familiar with the term as used in the article—why would this not be the case?
    - agentultra 4y ago
      I think that's part of the problem with the use of the word in programming circles: it means different things depending on whose talking.
    - vlunkr 4y ago
      Choosing the right database technology for your application has nothing to do with abstractions as far as I understand the term. Even if you're just going with the dictionary definition, that doesn't makes sense. An ORM is an abstraction, not a database.
      - PeterCorless 4y ago
        Here's an example of a database abstraction choice: the data model. Do you treat data as a series of normalized row-based data tables with joins optimized for strongly consistent OLTP? [PostgreSQL] Or as a denormalized row-based key-key-value single table optimized for eventually consistent OLTP? [ScyllaDB] Or as a denormalized column-based single table optimized for OLAP? [Pinot] Or as a denormalized key-value optimized for in-memory caching? [Redis] And so on for document stores, or property graphs, etc. Each of these design choices then gives way to the abstractions of implementation.
Zigurd 4y ago
In most cases time and capital are more expensive and abstraction, well-used, is the cheapest technical debt.
photochemsyn 4y ago
One takeaway for dealing with what the author calls misalignment seems to be that abstractions should be reversible, that is, easily rolled back to the base layer, without disrupting other parts of the system. This facilitates swapping out one abstraction for another. Another way of putting it is that if you lay out all the dependencies in a system, it should look more like a tree than a graph, with the abstractions closer to the leaves than to the root.
- vlovich123 4y ago
  Path dependency makes that often difficult / impractical.
BurningFrog 4y ago
I find this seemingly trivial insight very useful: Everything has costs and benefits, and they need to be weighed against each other. If you look for it, you'll see tons of arguments like "this has costs, so it's bad" and "this had benefits, so it's good". Both are missing half of the analysis!
- danielmarkbruce 4y ago
  Absolutely. Most political "debates" suffer from this. "Everything is a trade-off" is similar - trivial yet profound.
- akira2501 4y ago
  I've often described the type of Engineering I do as "the art of compromise."
yason 4y ago
Abstraction has a slight cost but doesn't inherently mean it's very expensive. The cost of abstraction is generally worth buying because it allows you to write code in more independent blobs which reduces complexity and thus bugs and potential bugs. But using abstractions to "solve" a hard problem is dearly expensive. By "solving" I mean pushing up the hard parts one layer at a time until you eventually can't avoid solving it for real. At best, maybe you're lucky and someone else has to solve it, by working through all the layers of abstractions you built on your way to negotiate yourself out of the hard spot. But those abstractions are expensive because they don't reduce complexity nor offer any tangible benefit except keeping their creators in their comfort zones.
falcolas 4y ago
I don't know if this will help (or even resonate) with anyone else, but it's helpful to me to view abstractions as a form of DRY. All developers have created functions to de-duplicate their code. And all developers have consequently seen how the more code a function de-duplicates, the larger and more cumbersome a function becomes; how many more if statements and safety checks come into play. Now imagine how complex - how many if statements and safety checks and introspection - have to go in to replacing hand-written SQL with custom constructors with an ORM.
- Arch-TK 4y ago
  It's unfortunate that you have only ever experienced abstractions which grow in size over time. The best abstractions out there are ones which operate well together to solve a large set of problems in the smallest amount of complexity (local to any given abstraction) as well as the smallest amount of total complexity at any given abstraction layer, as well as the smallest amount of total solution complexity. Now yes, this end goal is an idealized world, when you build on top of crusty old APIs and crusty old software you inevitably end up with crusty abstractions, especially difficult to deal with is the real world. That being said, there are plenty of examples out there where complexity at all three levels of detail that I outlined above has been painstakingly minimized. It usually takes a lot more effort to produce a simple solution to a problem than a complex one. This really requires the "alignment" the article talks about. Speaking of "alignment", ORM is one of the best examples of complete and total misalignment. The sets-of-tuples model of relational databases and the graph-of-objects model of object oriented systems are so seriously at odds with each other that there isn't a single ORM out there which successfully fully resolves the mismatch without severe abstraction leaks. (No, the fact that you can successfully use an ORM without abstraction leaks at the cost of severe performance degradation doesn't really matter.) On the other hand, large parts of the design of a project such as Plan 9 are extremely aligned.
  - falcolas 4y ago
    You're right, I have never seen an abstraction which hasn't grown with time. I think that's because either an abstraction project either ends, or receives constant feature requests. Kind of a grow-or-die mentality as applied to software development.
agentultra 4y ago
I’d argue that abstraction is cheap. You should do it more. As often as possible. Consider a world where the software industry state-of-the-art is completely nascent and isolated to every owner of a computing system. You buy these huge refrigerator-sized machines and get a bunch of manuals with them but nothing else. You have no operating system. No compiler. Nothing. The expense for this project can only be funded by a large university or government. The timeline to deliver is measured in years. In todays world? We write programs that generate new programs, that emulate entire classes of machines, and we ship software projects in days, weeks, and months. It’s so vastly complex that it’s a wonder it works at all let alone so well. Abstraction is what makes it all work without toppling over when someone, somewhere in the stack, makes a slight change.
uvdn7 4y ago
And no abstraction is perfect, as by definition an abstraction hides some details from the layer beneath. A good abstraction is one that allows you to not look under the hood most of the time. One can be happily writing code in their favorite programming language until you want better performance and started looking into cpu caches, branch prediction, etc. which are "breaks" the nice abstractions provided by the OS and high level programming languages.
gavinray 4y ago
ScyllaDB is, ironically, maybe one of the worst examples the author could have come up with for "abstraction" in the article. If folks aren't familiar with their work/internal tech, go check out some of their repos like Seastar. They have some of the most talented systems programmers on the planet writing thin veneers over kernel and hardware API's to squeeze every ounce out of performance. https://github.com/scylladb/seastar https://github.com/scylladb/seastar You want to talk about getting nerd-sniped to work somewhere if you're into performance/low-latency systems and databases =P I know it's beside the point, but I just had to share because I thought that was funny
NoZZz 4y ago
Hey talking heads.
revskill 4y ago
I think many misunderstand/misinterpret abstraction with magic. Abstraction is naming things that share common properties and structures. Magic is expensive because it adds debts in debugging. Non-abstraction is also expensive because it adds debts in development and refactoring. Good abstraction pays later, it reduces debts, but you need to buy it first. It's not expensive if it's well produced and consumed.
chousuke 4y ago
I like abstractions when they are transparent enough that it's easy to tell how it could be implemented one layer down. There might be many implementation details that you hide under the abstraction, but if the interface is so abstract that I can't envision a straightforward implementation of it just based on the interface, there's probably something wrong with the abstraction. Additionally, if the behaviour of the implementation conflicts with the simplified model communicated by the interface, that'll also cause issues.
- taeric 4y ago
  Similar to liking them for being easy to see what they mean one layer down, it is also nice to know what they mean one layer up. Your program has to inhabit a middle layer between what it is you want, and how it is that it will be executed. Sometimes, we can get lucky and a declarative statement of what we want works. Often, that isn't the case.
fdgsdfogijq 4y ago
I think its a meme now to hate abstraction. In big tech I see it as a justification to build parallel systems that do almost the same thing. Abstraction done well is where technical leverage comes from. Abstraction done poorly is where technical debt/bad technology comes from. Management prefers the safe route of gating their engineers from abstraction to get the work done with certainty, the trade off being less technical leverage. Its a corporate engineering meme to pin tech workers into being replaceable cogs pumping out low abstraction widgets
- BlargMcLarg 4y ago
  I wish it was a meme. The things people complained about decades ago are still happening today, and the practitioners are all too eager to turn a 100 line program into a 1000 line one, while taking weeks to test if it does what it should. The average loud dev is a GoF fanatic in love with inaccessible reflection. Double points if their tools are still stuck in a time where stack traces weren't extended to deal with such practices better. And with everything being sliced into pieces: the fun has only just begun
  - fdgsdfogijq 4y ago
    Call me blunt, but ive come to the conclusion most people arent intelligent enough to create complex abstractions
    - BlargMcLarg 4y ago
      I don't think intelligence or lack thereof is the problem. Rather, it seems people are paralyzing themselves by enjoying the endless discussions, stakeholder indecision, or boredom. All leading to these grotesque things which lack words in the right places, but have plenty in the wrong places. The average web dev CRUD job isn't interesting enough to warrant that much intelligence.
      - trashtester 4y ago
        Intelligence is definitly a factor. In fact, more often than not when I encounter a person or team that seem to spend their time on everything else except addressing their most important problems, the reason for the apparent procastination is that they simply do not even know how to START solving the problem. This is not exclusive for developers. It happens in management, too. Which is a second source of the problems you describe.
kahon65 4y ago
Not if ChatGPT creates it.
gpspake 4y ago
Abstractions are inevitable in software so the important thing is to think about, recognize, identify, and manage them. Sometimes it's appropriate to "unroll" bad abstractions in a codebase. quotes I live by (Mostly accurate from my memory): Repeated code is better than the wrong abstraction - Sandi Metz. Always know what the abstraction is, its value, and its cost - Kent Dodds.
AtNightWeCode 4y ago
I think it is too common that you have to work with abstractions that hides functionality in the underlying tech.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
LAC-Tech 4y ago
I know this is about back-end, but it partly describes my issues with front end - the industry standard abstractions save you very up front development time, but they increase the project complexity massively.
- another_devy 4y ago
  I think front end is possessed with abstraction and DRY. Everything has to be a library or npm module, People keep on writing their on versions of same stuff most of the times because they didn’t liked the name of the function or order of arguments. Complexity in front end is definitely increased by obsession of abstraction while it should’ve reduced it
ChicagoDave 4y ago
Abstractions are at their worst when they cross bound contexts. A customer in a billing system is not the same as a customer in a sales pipeline system. By merging disparate purposes for seemingly technical benefit, we often create complexity where none need exist. Domain Driven Design helps wall off inappropriate and costly abstractions.
anfilt 4y ago
This post makes me wish more research was done on exo-kernels: https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel The concept makes dealing with some the topics in this post much more logical and easier in my opinion.
feoren 4y ago
Reading the article and most of these comments, I've concluded that no two people are using the word "abstraction" exactly the same. I don't really understand what the author's definition of an "abstraction" is. He seems to call a poor database schema an "abstraction misalignment"; is anything not abstraction misalignment? It feels like we're in an old-folks home where everyone is talking but nobody is conversing. Nobody is agreeing on what "abstraction" means. A better title for this article is "mistakes are expensive", or possibly "bad design is expensive". But of course then nobody reads it, because everyone loves to hate on whatever they've decided "abstraction" means for them.
pickingdinner 4y ago
But it's all abstractions. The only issue is who gets to decide. The programmer controls the abstractions they're allowed to code, but everything predefined that they are forced to work with are "specs" that someone else abstracted for them. But even the programmers who worked on the specs are dealing with their own set of "specs" forced upon them by someone else. And these form the abstraction layers. And if you're working on accounting software, then many of the abstractions you're working with are defined by accounting practices and the IRS. Your job becomes to translate them, and the logic, into software that can automate them, etc. Whenever you're forced to deal with something, it's expensive. Whenever you're forced to do something, it's expensive, especially when considering the cost of getting it wrong.
qwertyuiop_ 4y ago
I think he has a better take on it. https://m.youtube.com/watch?v=hOrpppzEX14 https://m.youtube.com/watch?v=hOrpppzEX14
titzer 4y ago
The most expensive abstractions are usually do to bad layering, leading to abstraction inversion. This is the phenomenon where a layer of the system ends up reimplementing the abstractions of a lower level that have been imprudently hidden by a layer in-between. This is why considering all layers together needs to be a constant consideration.
- gwbas1c 4y ago
  I hit that when working with NHibernate (an early ORM in C# that was a port of Java's Hibernate.) I found that it was easier to write the ^%$#$^ query myself than to deal with that monster. What was crazier was that the senior members of the team were so afraid of a database that they couldn't conceive of writing a SQL query. I got stuck on trying to store an enum as an int in a column for about 3 days. Eventually I decided to read the book; and walked away realizing that it was only really useful for extremely complicated schemas. For simple schemas the learning curve was so high that writing boring SQL code was easier / faster. (Now, Microsoft's entity framework is often "good enough" as an ORM, and has a rather gentle learning curve. It helps that C# has a way to express SQL-like queries on a collection of objects and translate them into SQL.)
habibur 4y ago
Bad title. The article clearly says : Abstraction *misalignment* is expensive.
khiqxj 4y ago
no duh. now lets get back to the real world where all the "abstractions" are poor abstractions AND inefficient.
user5678 4y ago
nathias 4y ago
premature abstraction is a very common anti-pattern
college_physics 4y ago
your abstraction is more expensive than mine
manv1 4y ago
"You can't delegate understanding." - Charles Eames Abstraction means you don't have to think about details. Unfortunately, details matter in software. As a simple example, how many projects use Hibernate, then run into massive performance problems? Pretty much all of them...because the devs don't bother to learn how their data store works. Then they iterate over 150,000,000 rows with their ORM instead of using, you know, SQL. It's always a tradeoff, but again, "you can't delegate understanding."

224 points by pclmulqdq 4y ago

This sounds like an application risk-aversion/-seeking that Kahneman highlights in "Thinking, Fast and Slow" Specifically... each of these individual decisions (wrt to abstraction, in this context) was made to locally optimize (speed, simplicity, etc) some issue or other But when taken in gestalt, they're overall not only not optimal, but downright bad Most of those decisions are made by people far to close to a tiny portion of the problem to understand the implications of their choices Alternatively, it's an application of the blind mice finding an elephant story: yes, it's like a rope, a fan, a tree, a hose, and on and on But it's none of those, it's an elephant

projektfu 4y ago
The title needs work too. People will undoubtedly assume that reducing abstraction will make things cheaper.
- sghio4Q2 4y ago
  Indeed.
- cryptonector 4y ago
  Eh, some people might do that, but, really, titles like that beg you to read the article (or at least it's why I read it). The article absolutely does not say anything like "reducing abstraction will make things cheaper".
- danShumway 4y ago
  Sometimes reducing abstractions will make things cheaper. Some regular traps that I see people run into with abstractions: - building around scenarios that they can be pretty confident that they will never need to handle. - building abstractions that are larger and more complicated than the thing they are abstracting (this tends to happen when people build abstractions of abstractions). - building abstractions before they have a proper understanding of what they're abstracting and what they'll need to encapsulate. In many of those cases, reducing abstractions (even just temporarily) reduces complexity. When I'm building purely personal projects, I don't use abstractions to help me deal with directory structure on Windows, because I don't use Windows, and that would be additional complexity for no benefit -- so it's simpler for me to just work with the lower level OS paths. When I start writing an abstraction I look at the amount of documentation I'm generating, and if I'm generating more documentation than it would take to explain the underlying system, I look to see if there are concepts that I can remove. I worked at a company where our build process became considerably simpler when we stopped using high-level build tools like Gulp/Grunt and switched to writing simple Node scripts, because it was easier to debug what those scripts were doing. We simplified that process even further by occasionally just dipping into Bash scripts. Working with low-level concepts for a while often also gives you better understanding of what you need to abstract. I've worked with codebases where the abstractions all get built first, and it's not uncommon for those abstractions to be built around tasks that are pretty simple and easy to do with lower-level code, at the same time that the abstractions completely ignore the really difficult tasks that are very annoying to do. And once abstractions get baked in, it was time consuming and difficult and expensive to pull them out and rewrite them. Going back to the scripts above, our build process got better because we made it simple to begin with -- a set of scripts, rather than a large established pipeline -- and then as we identified pain points, we started abstracting those pain points away. That allowed us to not waste time rewriting abstractions over and over and instead to have targeted small interfaces that helped us with the actual painful parts of building and deployment. It is surprisingly common for software to be over-abstracted to the point where it is more complicated to deal with than it would be otherwise. I mean, heck, this comes up in web development all the time, it is one of the primary criticisms people have of the JS ecosystem -- that it overcomplicates development. In many cases those complications exist for reasons, they solve real problems that people have had. But also in many cases, someone's individual blog doesn't need any of that, and it would be cheaper and easier for them to build something smaller and simpler. If I can build a site that is one HTML file and one CSS file, and I know that's all I'm going to need, then it's overkill to try and set up a bunch of abstractions on top of that.
hinkley 4y ago
You’re speaking to impedance mismatch which is one of the classic blunders. Systems that don’t use the same jargon as the user base end up having substantial bugs, especially ones the authors insist are features. The abstractions are a bad fit for the problem domain and they break things. I’m in a project that did that massively, at the hands of both architectural astronauts and another common blunder: people who seemed to think the concerns of the user base are beneath them tend to create a fantasy world for themselves where they can pretend they work on something more esoteric than the petty concerns of the people who pay their salaries. Because of the bullshit (and more importantly, the top-heavy sources of that bullshit) we’ve lost a lot of the better people who could have fixed that situation. Meanwhile we are also trying to expand into a new industry, and I can’t help but wonder if we would have been in trouble even if our abstractions had matched our core competency. I would have needed all of those missing people to make the refactors necessary to do that work, so the agents of chaos and delusion are a little vindicated. I have a couple of things I want to button up but they are all fast approaching. I don’t expect to be here in six months. I’ve started fantasizing about my exit interview. In which I will probably suggest that my team is too powerful and isolated from the end user and so needs to be disbanded, its members dispersed into groups one and two steps closer to the users. So as to concretize some of the code and confront them with the day to day struggles of internal customers, dealing with the batshit parts of the code.

nine_k 4y ago

Lack of abstraction is also expensive: try writing something large in assembly. I'd say that lack of a language / system of notions adequate to the subject area is expensive. The desire to describe things in a way that's efficient for a particular class of problems leads to invention of various frameworks. Say, Rails makes you hugely productive at solving a particular type of problems (see [Shopify]), though it's less than helpful if you try to apply it to unfitting problems (see [Twitter]). [Shopify]: https://tomaszs2.medium.com/how-shopify-handled-1-27-million-requests-s-during-black-friday-cyber-monday-8a4b1d0dd1f9 https://tomaszs2.medium.com/how-shopify-handled-1-27-million... (Sorry for a Medium link) [Twitter]: https://www.theregister.com/2012/11/08/twitter_epic_traffic_saved_by_java/ https://www.theregister.com/2012/11/08/twitter_epic_traffic_...

ghaff 4y ago
The headline is sort of misleading. It is (appropriately) primarily about inappropriate abstractions. As you say, some abstractions are unavoidable. Even assembly language is an abstraction.
- cryptonector 4y ago
  That's why real programmers write binary object code directly.
  - nine_k 4y ago
    Don't tell them about the lost art of loading microcode on boot.
  - fnordpiglet 4y ago
    I wire my magnetic core with artisanal copper.
    - choeger 4y ago
      You use copper? These kids today...

pca006132 4y ago

The problem is that application performance doesn't only depend on the thing you want to do, but also depends on the low level implementation details and workload, which the programmer may not fully understand or in control of. Some operation may be optimal when executed alone but exists better alternative for batch processing. Some programs may be optimized for throughput but is really bad for low latency application. The designer have to consider these when designing their system, and compilers/library writers can't help much except write a more comprehensive documentation which the programmer will probably not read :). I think abstraction guides implementation, so it is very beneficial to think about what will be the bottleneck in the application, and if a certain abstraction will hinder the performance or prohibit future optimization. And although a low level abstraction can in theory allow for better performance if the programmer spend enough time to optimize for it, they often won't and a simpler interface (with sane implementation and perhaps escape hatches for performance tuning) may allow for better performance with much less effort.

AnimalMuppet 4y ago
> I think abstraction guides implementation, so it is very beneficial to think about what will be the bottleneck in the application I think this way, but in terms of the "performance" of writing the application. What makes it hard to write this program? For example, in high-performance computing, you may need to control memory alignment (for SIMD and cache reasons). That makes it harder to write a program that just implements an algorithm. So I think you should identify what it is that makes the program hard to write, and pick abstractions that help with that, that give you easier ways of controlling or managing that hard part.

mattacular 4y ago

One of my favorite quotes about software engineering, sorry not sure who to attribute it to: "First you learn the value of abstraction, then you learn the cost of abstraction, then you're ready to engineer"

pphysch 4y ago
Junior, mid, senior? Explains why the mid phase is brief.

fnordpiglet 4y ago

A CS prof of mine liked to say: Every problem in computer science can be solved with abstraction other than too much abstraction.

cloogshicer 4y ago

Even though many people call abstraction 'the essence of programming', it is rarely fully understood. Maybe no human on the planet really understands it. I've long had an essay in me about this topic that I really need to put to paper soon. Here is the brief version of that essay: * Programming is about building theories/models of the problem. Source code largely has no value by itself [1] * It is very difficult, if not impossible, to transfer these models/theories between humans (see the essay I linked below for details) * Because of this difficulty, programming is essentially teaching (when writing program code, we're teaching other people of the problem domain, and models that fit that problem domain) * Another way to phrase this: All programming is building user interfaces. When you're writing, say, a function, you're writing a more abstract UI for the next programmer or yourself * The tools we have for writing these UIs are terrible - taken the above example of writing a function, what are the tools you have to communicate this idea to the next person? A single string of characters (the function name). In a typed language you get a bit of extra info because of the type information (I believe this to be the main advantage of type systems). * The big question is: What would a better UI for communicating abstractions look like? * I don't know the answer to that question but I have a hunch that it has to be bi-directional. If you've ever worked with a GUI library that has a visual editor, you know how awful they are, unless there is also a representation of the same GUI in code. This bi-directional mapping of code and GUI makes it very easy to understand the two different ways of looking at the problem. I think something like this is needed down to the very lowest levels of abstractions. * Another way to phrase this: Imagine two different scientific models for the same problem. For example, for the model of an atom, its protons and neutrons, there is the more simple Bohr Model, which is completely wrong, given our current knowledge, but still very useful in many modern calculations. But in certain situations, a more accurate model is needed, which takes quantum mechanical effects into account. I see an analogy for programming here: In most cases, a simplified model suffices, but as more performance is needed, a more complex model is required. The question is, how can we easily teach someone the more difficult model, once they've understood the simple one? And the other way around (which is often also not easy, since simple is not the same as easy). If you have any thoughts on what I've written, any at all, I'd love to hear from you (I'll watch this thread, or find my email in my bio). [1] https://hiringengineersbook.com/post/autonomy/ https://hiringengineersbook.com/post/autonomy/

agentultra 4y ago

For the mathematically minded, this is not about abstractions. This is about technology choices! One more interpretation of an overloaded word that means a very specific thing.

vlunkr 4y ago
Agreed, they should probably include their definition of 'abstraction' in the post.
- adamrezich 4y ago
  I would've thought that everyone who has done any programming at all would be familiar with the term as used in the article—why would this not be the case?
  - agentultra 4y ago
    I think that's part of the problem with the use of the word in programming circles: it means different things depending on whose talking.
  - vlunkr 4y ago
    Choosing the right database technology for your application has nothing to do with abstractions as far as I understand the term. Even if you're just going with the dictionary definition, that doesn't makes sense. An ORM is an abstraction, not a database.
    - PeterCorless 4y ago
      Here's an example of a database abstraction choice: the data model. Do you treat data as a series of normalized row-based data tables with joins optimized for strongly consistent OLTP? [PostgreSQL] Or as a denormalized row-based key-key-value single table optimized for eventually consistent OLTP? [ScyllaDB] Or as a denormalized column-based single table optimized for OLAP? [Pinot] Or as a denormalized key-value optimized for in-memory caching? [Redis] And so on for document stores, or property graphs, etc. Each of these design choices then gives way to the abstractions of implementation.

Zigurd 4y ago

In most cases time and capital are more expensive and abstraction, well-used, is the cheapest technical debt.

photochemsyn 4y ago

One takeaway for dealing with what the author calls misalignment seems to be that abstractions should be reversible, that is, easily rolled back to the base layer, without disrupting other parts of the system. This facilitates swapping out one abstraction for another. Another way of putting it is that if you lay out all the dependencies in a system, it should look more like a tree than a graph, with the abstractions closer to the leaves than to the root.

vlovich123 4y ago
Path dependency makes that often difficult / impractical.

BurningFrog 4y ago

I find this seemingly trivial insight very useful: Everything has costs and benefits, and they need to be weighed against each other. If you look for it, you'll see tons of arguments like "this has costs, so it's bad" and "this had benefits, so it's good". Both are missing half of the analysis!

danielmarkbruce 4y ago
Absolutely. Most political "debates" suffer from this. "Everything is a trade-off" is similar - trivial yet profound.
akira2501 4y ago
I've often described the type of Engineering I do as "the art of compromise."

yason 4y ago

Abstraction has a slight cost but doesn't inherently mean it's very expensive. The cost of abstraction is generally worth buying because it allows you to write code in more independent blobs which reduces complexity and thus bugs and potential bugs. But using abstractions to "solve" a hard problem is dearly expensive. By "solving" I mean pushing up the hard parts one layer at a time until you eventually can't avoid solving it for real. At best, maybe you're lucky and someone else has to solve it, by working through all the layers of abstractions you built on your way to negotiate yourself out of the hard spot. But those abstractions are expensive because they don't reduce complexity nor offer any tangible benefit except keeping their creators in their comfort zones.

falcolas 4y ago

I don't know if this will help (or even resonate) with anyone else, but it's helpful to me to view abstractions as a form of DRY. All developers have created functions to de-duplicate their code. And all developers have consequently seen how the more code a function de-duplicates, the larger and more cumbersome a function becomes; how many more if statements and safety checks come into play. Now imagine how complex - how many if statements and safety checks and introspection - have to go in to replacing hand-written SQL with custom constructors with an ORM.

Arch-TK 4y ago
It's unfortunate that you have only ever experienced abstractions which grow in size over time. The best abstractions out there are ones which operate well together to solve a large set of problems in the smallest amount of complexity (local to any given abstraction) as well as the smallest amount of total complexity at any given abstraction layer, as well as the smallest amount of total solution complexity. Now yes, this end goal is an idealized world, when you build on top of crusty old APIs and crusty old software you inevitably end up with crusty abstractions, especially difficult to deal with is the real world. That being said, there are plenty of examples out there where complexity at all three levels of detail that I outlined above has been painstakingly minimized. It usually takes a lot more effort to produce a simple solution to a problem than a complex one. This really requires the "alignment" the article talks about. Speaking of "alignment", ORM is one of the best examples of complete and total misalignment. The sets-of-tuples model of relational databases and the graph-of-objects model of object oriented systems are so seriously at odds with each other that there isn't a single ORM out there which successfully fully resolves the mismatch without severe abstraction leaks. (No, the fact that you can successfully use an ORM without abstraction leaks at the cost of severe performance degradation doesn't really matter.) On the other hand, large parts of the design of a project such as Plan 9 are extremely aligned.
- falcolas 4y ago
  You're right, I have never seen an abstraction which hasn't grown with time. I think that's because either an abstraction project either ends, or receives constant feature requests. Kind of a grow-or-die mentality as applied to software development.

agentultra 4y ago

I’d argue that abstraction is cheap. You should do it more. As often as possible. Consider a world where the software industry state-of-the-art is completely nascent and isolated to every owner of a computing system. You buy these huge refrigerator-sized machines and get a bunch of manuals with them but nothing else. You have no operating system. No compiler. Nothing. The expense for this project can only be funded by a large university or government. The timeline to deliver is measured in years. In todays world? We write programs that generate new programs, that emulate entire classes of machines, and we ship software projects in days, weeks, and months. It’s so vastly complex that it’s a wonder it works at all let alone so well. Abstraction is what makes it all work without toppling over when someone, somewhere in the stack, makes a slight change.

uvdn7 4y ago

And no abstraction is perfect, as by definition an abstraction hides some details from the layer beneath. A good abstraction is one that allows you to not look under the hood most of the time. One can be happily writing code in their favorite programming language until you want better performance and started looking into cpu caches, branch prediction, etc. which are "breaks" the nice abstractions provided by the OS and high level programming languages.

gavinray 4y ago

ScyllaDB is, ironically, maybe one of the worst examples the author could have come up with for "abstraction" in the article. If folks aren't familiar with their work/internal tech, go check out some of their repos like Seastar. They have some of the most talented systems programmers on the planet writing thin veneers over kernel and hardware API's to squeeze every ounce out of performance. https://github.com/scylladb/seastar https://github.com/scylladb/seastar You want to talk about getting nerd-sniped to work somewhere if you're into performance/low-latency systems and databases =P I know it's beside the point, but I just had to share because I thought that was funny

NoZZz 4y ago

Hey talking heads.

revskill 4y ago

I think many misunderstand/misinterpret abstraction with magic. Abstraction is naming things that share common properties and structures. Magic is expensive because it adds debts in debugging. Non-abstraction is also expensive because it adds debts in development and refactoring. Good abstraction pays later, it reduces debts, but you need to buy it first. It's not expensive if it's well produced and consumed.

chousuke 4y ago

I like abstractions when they are transparent enough that it's easy to tell how it could be implemented one layer down. There might be many implementation details that you hide under the abstraction, but if the interface is so abstract that I can't envision a straightforward implementation of it just based on the interface, there's probably something wrong with the abstraction. Additionally, if the behaviour of the implementation conflicts with the simplified model communicated by the interface, that'll also cause issues.

taeric 4y ago
Similar to liking them for being easy to see what they mean one layer down, it is also nice to know what they mean one layer up. Your program has to inhabit a middle layer between what it is you want, and how it is that it will be executed. Sometimes, we can get lucky and a declarative statement of what we want works. Often, that isn't the case.

fdgsdfogijq 4y ago

I think its a meme now to hate abstraction. In big tech I see it as a justification to build parallel systems that do almost the same thing. Abstraction done well is where technical leverage comes from. Abstraction done poorly is where technical debt/bad technology comes from. Management prefers the safe route of gating their engineers from abstraction to get the work done with certainty, the trade off being less technical leverage. Its a corporate engineering meme to pin tech workers into being replaceable cogs pumping out low abstraction widgets

BlargMcLarg 4y ago
I wish it was a meme. The things people complained about decades ago are still happening today, and the practitioners are all too eager to turn a 100 line program into a 1000 line one, while taking weeks to test if it does what it should. The average loud dev is a GoF fanatic in love with inaccessible reflection. Double points if their tools are still stuck in a time where stack traces weren't extended to deal with such practices better. And with everything being sliced into pieces: the fun has only just begun
- fdgsdfogijq 4y ago
  Call me blunt, but ive come to the conclusion most people arent intelligent enough to create complex abstractions
  - BlargMcLarg 4y ago
    I don't think intelligence or lack thereof is the problem. Rather, it seems people are paralyzing themselves by enjoying the endless discussions, stakeholder indecision, or boredom. All leading to these grotesque things which lack words in the right places, but have plenty in the wrong places. The average web dev CRUD job isn't interesting enough to warrant that much intelligence.
    - trashtester 4y ago
      Intelligence is definitly a factor. In fact, more often than not when I encounter a person or team that seem to spend their time on everything else except addressing their most important problems, the reason for the apparent procastination is that they simply do not even know how to START solving the problem. This is not exclusive for developers. It happens in management, too. Which is a second source of the problems you describe.

kahon65 4y ago

Not if ChatGPT creates it.

gpspake 4y ago

Abstractions are inevitable in software so the important thing is to think about, recognize, identify, and manage them. Sometimes it's appropriate to "unroll" bad abstractions in a codebase. quotes I live by (Mostly accurate from my memory): Repeated code is better than the wrong abstraction - Sandi Metz. Always know what the abstraction is, its value, and its cost - Kent Dodds.

AtNightWeCode 4y ago

I think it is too common that you have to work with abstractions that hides functionality in the underlying tech.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

LAC-Tech 4y ago

I know this is about back-end, but it partly describes my issues with front end - the industry standard abstractions save you very up front development time, but they increase the project complexity massively.

another_devy 4y ago
I think front end is possessed with abstraction and DRY. Everything has to be a library or npm module, People keep on writing their on versions of same stuff most of the times because they didn’t liked the name of the function or order of arguments. Complexity in front end is definitely increased by obsession of abstraction while it should’ve reduced it

ChicagoDave 4y ago

Abstractions are at their worst when they cross bound contexts. A customer in a billing system is not the same as a customer in a sales pipeline system. By merging disparate purposes for seemingly technical benefit, we often create complexity where none need exist. Domain Driven Design helps wall off inappropriate and costly abstractions.

anfilt 4y ago

This post makes me wish more research was done on exo-kernels: https://en.wikipedia.org/wiki/Exokernel https://en.wikipedia.org/wiki/Exokernel The concept makes dealing with some the topics in this post much more logical and easier in my opinion.

feoren 4y ago

Reading the article and most of these comments, I've concluded that no two people are using the word "abstraction" exactly the same. I don't really understand what the author's definition of an "abstraction" is. He seems to call a poor database schema an "abstraction misalignment"; is anything not abstraction misalignment? It feels like we're in an old-folks home where everyone is talking but nobody is conversing. Nobody is agreeing on what "abstraction" means. A better title for this article is "mistakes are expensive", or possibly "bad design is expensive". But of course then nobody reads it, because everyone loves to hate on whatever they've decided "abstraction" means for them.

pickingdinner 4y ago

But it's all abstractions. The only issue is who gets to decide. The programmer controls the abstractions they're allowed to code, but everything predefined that they are forced to work with are "specs" that someone else abstracted for them. But even the programmers who worked on the specs are dealing with their own set of "specs" forced upon them by someone else. And these form the abstraction layers. And if you're working on accounting software, then many of the abstractions you're working with are defined by accounting practices and the IRS. Your job becomes to translate them, and the logic, into software that can automate them, etc. Whenever you're forced to deal with something, it's expensive. Whenever you're forced to do something, it's expensive, especially when considering the cost of getting it wrong.

qwertyuiop_ 4y ago

I think he has a better take on it. https://m.youtube.com/watch?v=hOrpppzEX14 https://m.youtube.com/watch?v=hOrpppzEX14

titzer 4y ago

The most expensive abstractions are usually do to bad layering, leading to abstraction inversion. This is the phenomenon where a layer of the system ends up reimplementing the abstractions of a lower level that have been imprudently hidden by a layer in-between. This is why considering all layers together needs to be a constant consideration.

gwbas1c 4y ago
I hit that when working with NHibernate (an early ORM in C# that was a port of Java's Hibernate.) I found that it was easier to write the ^%$#$^ query myself than to deal with that monster. What was crazier was that the senior members of the team were so afraid of a database that they couldn't conceive of writing a SQL query. I got stuck on trying to store an enum as an int in a column for about 3 days. Eventually I decided to read the book; and walked away realizing that it was only really useful for extremely complicated schemas. For simple schemas the learning curve was so high that writing boring SQL code was easier / faster. (Now, Microsoft's entity framework is often "good enough" as an ORM, and has a rather gentle learning curve. It helps that C# has a way to express SQL-like queries on a collection of objects and translate them into SQL.)

habibur 4y ago

Bad title. The article clearly says : Abstraction *misalignment* is expensive.

khiqxj 4y ago

no duh. now lets get back to the real world where all the "abstractions" are poor abstractions AND inefficient.

user5678 4y ago

nathias 4y ago

premature abstraction is a very common anti-pattern

college_physics 4y ago

your abstraction is more expensive than mine

manv1 4y ago

"You can't delegate understanding." - Charles Eames Abstraction means you don't have to think about details. Unfortunately, details matter in software. As a simple example, how many projects use Hibernate, then run into massive performance problems? Pretty much all of them...because the devs don't bother to learn how their data store works. Then they iterate over 150,000,000 rows with their ORM instead of using, you know, SQL. It's always a tradeoff, but again, "you can't delegate understanding."