8 ms·

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20. Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.

BackBlast 5y ago
Specs are often over engineered. This is part of the problem. Too much effort in rowing the boat, not enough effort in figuring out the right direction to point it first.
sokoloff 5y ago
With the benefit of hindsight, I think you can conclude that the system was over-engineered. If you’re trying to answer only “was it over-engineered?” I think you can answer it with hindsight. If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.
noisy_boy 5y ago
My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you are wondering about those same things, trust me, there is a reason/background - its not because the people who did it that way were stupid".
- mgkimsal 5y ago
  Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'. Examples Having a 'user info' table with 190+ boolean columns for 'country' ('US','CA','DE','IT', etc) in case someone wants to indicate they're 'from' two countries. Joining views on views of joined views which are themselves built on joins of other views is, likely, not a terribly sound data decision. (X: "It worked fine last year - you must have broken something." Me: "well... last year you had 450 users, and now there are 37000 users - this isn't a performant way of doing this". X: "Can't be true - the person who wrote it is a DBA")
  - hinkley 5y ago
    > Can't be true - the person who wrote it is a DBA Not for long…
    - mgkimsal 5y ago
      heh... well... he was a DBA, and wrote all the application code as well. testing on localhost with 20 sample users, what could possibly go wrong? :)
  - noisy_boy 5y ago
    > Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'. I agree on the point regarding the decision being 'stupid' but in this case it was not because of inexperience. On the contrary, it was because an overruling decision by an experienced manager. So the point is that it is not always technical or due to inexperience (though that does happen) - from what I've seen, it is quite common for such things to happen due to hierarchical/ego/political issues as well.
mumblemumble 5y ago
It's a question of framing. Nothing is ever over- or under-engineered. It's over- or under-engineered for a purpose. What that distinction recognizes is that it's possible for something to have been well-engineered at one time, while still being over-engineered today. Header files in C and C++ are an example of this phenomenon. They solved a very real problem with technical constraints from 40, 50 years ago, both in terms of computers' capabilities and compiler technology, but, since those problems don't exist anymore, they function as over-engineering.
- selcuka 5y ago
  Agreed, but "nothing is ever over- or under-engineered" is a bit too bold. There is definitely over engineering just for the sake of over engineering, and under engineering caused by incompetency.
- blt 5y ago
  What is the less-engineered alternative to header files?
  - mumblemumble 5y ago
    Symbolic imports. Basically, the way every programming language newer than C++ or Objective-C does it.
typicalbender 5y ago
This is why commit messages are so important. Documentation that is external to the code its referencing will always drift but a commit message is a snapshot in time that is attached to the thing it is referring to. In my experience not enough people take commit messages seriously and just basically give a summary of what changed not why it was changed.
- baby 5y ago
  Commit messages get lost too easily. It’s the worse way to document reasoning around the code. There’s many proposals to document code outside of code and none of them has ever worked. Just comment your code.
lumost 5y ago
I think the lesson here is that with great product market fit everything is under engineered, with poor product market fit everything is over engineered. Statistically you are more likely to be building an over engineered product with poor product market fit than you are to be building an under engineered product with great market fit. Now is over or under engineering a bad thing? that depends on how many resources the company can muster, your customer tolerance to bugs/failures, and whether you can build a small subset of features well. In practice I've observed products with fewer, but well built features tend to succeed more so than buggy featureful products. Large companies may opt to overbuild everything simply so that they can prove that the product was bad rather than having to decide whether it was product or execution that failed.
- mountainriver 5y ago
  +1 to less better built features. What I primarily see kill products is going after too many features and they all suck. Often when you do one thing really well you may find there is more value in the primitives
  - rch 5y ago
    Or going after too many users. The 'next billion' mindset is misguided.
wellpast 5y ago
> If you weren't there, you don't know. Technically speaking, for any given situation there's no way you can be wrong. But this post describes a distinct and, by my experience, dominant industry phenomenon. And the stakes are as high is OP says (dead products). Industry engineers in the large are massively divorced from product outcome. This should be no surprise, as it is a seller's market: Delivering product simply and directly can be, depending on perspective, somewhat of a mundane and Sisyphean activity. Engineering "stuff" is far more intellectually engaging, and there's no market pressure (for engineers) against over-engineering. If my company fails to deliver a product outcome, whether a viable company or startup, in this market there is no recourse at all on my salary let alone my hireability. I say "depending on perspective" because if you can find a way to be more intellectually and emotionally stimulated by shipping stable product over time, then ime you _can_ unite your joie de vivre with market outcomes. But that does not at all describe most engineers in industry who generally at best shed off some market value while spinning unnecessary webs of toy abstractions (ie liabilities).
what_is_orcas 5y ago
I politely disagree. I'm currently walking away from a code-base (and the company who let it be built) that was obviously built by someone who's mental model of how the code works was incorrect. At some very basic levels, it under-performs compared to literally every other example of this sort of code I've ever seen (and it's extremely common), which results in a subpar user-experience and an EC2 instance way bigger than necessary to run this sort of code. A lot of code running doesn't need to run in the first place, and due to the incorrect mental model, there's a substantial amount of code that doesn't run. Nobody really understands how bad it is, though, because it's barely tested (and the engineer who wrote most of the code has argued that we don't need tests, for various reasons) and all of the engineers on the team who didn't build it initially, have told me that they're not familiar enough with how this sort of product should work to understand that it doesn't work like that (again, this is a super common kind of product). There's a lot of other anti-patterns (I started keeping a list in my first week) but I think these are the most damning. This code is in production. Nobody at the org brought up these issues earlier in development (like saying "hey, nobody else's product does [annoying thing that doesn't need to happen], why does ours?"), which leads me to believe that the whole technical org is at fault. It sucks and I can't wait to be done with this experience.
- im_down_w_otp 5y ago
  That doesn't sound like over-engineering. That sounds like bad-engineering.
  - what_is_orcas 5y ago
    I don't disagree, but I think that the reason that this fits here is because the poorly engineered software is unnecessarily complex (not in the "someday we might need this" way, but in ways that I'm having a hard time articulating without giving away the product because I know my coworkers are on HN). In fact, it was the ridiculous complexity that drove me to discover the disparities between the original developer's mental-model and the working-model. It just so happens that I've worked on this sort of thing before (as have thousands of others) that I was pretty quickly able to understand where the original developer went wrong (trust me, I'm just an ordinary developer, not a 10x elite hacker or anything).
activitypea 5y ago
In my experience, the situation you're describing more often results in "sloppy" code -- it seems convoluted, non-obvious, poorly thought out until you realize when the pivots happened. Overengineered code usually looks neat at first glance, maybe even impressive, especially to non-technical stakeholders. Then when you actually dive into it, you notice a bunch of wrong abstractions that introduce layers of indirection without offering any sort of valuable flexibility. "This metaclass that implements a context manager could've just been a slightly longer than average function" type stuff. A codebase that pivoted multiple times makes more sense the more time you spend in it, an overengineered one makes less.
PragmaticPulp 5y ago
> I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. The specs and the decisions surrounding the code are usually part of the over-engineering. > Everything was built to spec. Even with reasonable specs, I've worked with, and later managed, engineers who had a strong drive toward over-engineering solutions. I've also done it myself, and I suspect most people reading this have done the same. The problem is that many of us engineers became developers because we enjoy building things and we enjoy solving complex problems. It can be exciting to imagine a particularly complex version of a problem and how elegant and fun a complex solution would be. It can be as simple as a programmer who tries to write their own CMS when the spec could be solved with a simple blogging platform. If you're an ambitious programmer, writing a CMS is fun! Installing Wordpress is blech. I didn't truly understand the ubiquity of the problem until managing ambitious engineers who wanted to do complex things. At times it felt like 1/4 of my job was just asking "If we wanted to simplify this solution, what would that look like?" Repeat for a few rounds and eventually the team has shed a lot of unnecessary complexity without sacrificing the spec at all.
btilly 5y ago
I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Can you prove it in a court of law? No. But you still can often easily diagnose it. And you'll usually be correct. For example the conditions under which https://wiki.c2.com/?SecondSystemEffect https://wiki.c2.com/?SecondSystemEffect happens are well-known, and the pattern has been repeated often in the half-century since it was named. If it is a second system, and you can find complicating abstractions in it that get used exactly once in practice, then with better than 90% odds it was over-engineered. And the farther you dig, the stronger the evidence will become.
SkyPuncher 5y ago
> Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. I would argue this is a case of over-engineering. Though, it extends beyond pure engineering. This is poor resource management and business de-risking. An effective approach here would have been: "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern".
- beachy 5y ago
  I've been in exactly this position, in fact I conceived and created the situation at my last company. I found it uncanny how on point OP was. The trouble with "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern" is that when you're building an integration marketplace, you need partners, and they don't want to be your research bitches. They just want to build their integrations once, and gtfo. If they think you're signing then up for a revolving wheel of experimentation, they'll just politely stay away until someone else has done the hard yards. Sure if you're a Microsoft or a Google you'll have any number of willing partners who will put up with anything to be the pioneers in your integrations marketplace. But otherwise, they're using your integrations marketplace purely for the benefits, and they don't want to be building on sand.
radley 5y ago
> Everything was built to spec. First line in the second paragraph of every dead company's obituary.
d0m3 5y ago
Well how do you know something is overengineered? It's when you know it will never (or too far in the future) need to scale accordingly to the engineering.

BackBlast 5y ago

Specs are often over engineered. This is part of the problem. Too much effort in rowing the boat, not enough effort in figuring out the right direction to point it first.

sokoloff 5y ago

With the benefit of hindsight, I think you can conclude that the system was over-engineered. If you’re trying to answer only “was it over-engineered?” I think you can answer it with hindsight. If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.

noisy_boy 5y ago

My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you are wondering about those same things, trust me, there is a reason/background - its not because the people who did it that way were stupid".

mgkimsal 5y ago
Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'. Examples Having a 'user info' table with 190+ boolean columns for 'country' ('US','CA','DE','IT', etc) in case someone wants to indicate they're 'from' two countries. Joining views on views of joined views which are themselves built on joins of other views is, likely, not a terribly sound data decision. (X: "It worked fine last year - you must have broken something." Me: "well... last year you had 450 users, and now there are 37000 users - this isn't a performant way of doing this". X: "Can't be true - the person who wrote it is a DBA")
- hinkley 5y ago
  > Can't be true - the person who wrote it is a DBA Not for long…
  - mgkimsal 5y ago
    heh... well... he was a DBA, and wrote all the application code as well. testing on localhost with 20 sample users, what could possibly go wrong? :)
- noisy_boy 5y ago
  > Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'. I agree on the point regarding the decision being 'stupid' but in this case it was not because of inexperience. On the contrary, it was because an overruling decision by an experienced manager. So the point is that it is not always technical or due to inexperience (though that does happen) - from what I've seen, it is quite common for such things to happen due to hierarchical/ego/political issues as well.

mumblemumble 5y ago

It's a question of framing. Nothing is ever over- or under-engineered. It's over- or under-engineered for a purpose. What that distinction recognizes is that it's possible for something to have been well-engineered at one time, while still being over-engineered today. Header files in C and C++ are an example of this phenomenon. They solved a very real problem with technical constraints from 40, 50 years ago, both in terms of computers' capabilities and compiler technology, but, since those problems don't exist anymore, they function as over-engineering.

selcuka 5y ago
Agreed, but "nothing is ever over- or under-engineered" is a bit too bold. There is definitely over engineering just for the sake of over engineering, and under engineering caused by incompetency.
blt 5y ago
What is the less-engineered alternative to header files?
- mumblemumble 5y ago
  Symbolic imports. Basically, the way every programming language newer than C++ or Objective-C does it.

typicalbender 5y ago

This is why commit messages are so important. Documentation that is external to the code its referencing will always drift but a commit message is a snapshot in time that is attached to the thing it is referring to. In my experience not enough people take commit messages seriously and just basically give a summary of what changed not why it was changed.

baby 5y ago
Commit messages get lost too easily. It’s the worse way to document reasoning around the code. There’s many proposals to document code outside of code and none of them has ever worked. Just comment your code.

lumost 5y ago

I think the lesson here is that with great product market fit everything is under engineered, with poor product market fit everything is over engineered. Statistically you are more likely to be building an over engineered product with poor product market fit than you are to be building an under engineered product with great market fit. Now is over or under engineering a bad thing? that depends on how many resources the company can muster, your customer tolerance to bugs/failures, and whether you can build a small subset of features well. In practice I've observed products with fewer, but well built features tend to succeed more so than buggy featureful products. Large companies may opt to overbuild everything simply so that they can prove that the product was bad rather than having to decide whether it was product or execution that failed.

mountainriver 5y ago
+1 to less better built features. What I primarily see kill products is going after too many features and they all suck. Often when you do one thing really well you may find there is more value in the primitives
- rch 5y ago
  Or going after too many users. The 'next billion' mindset is misguided.

wellpast 5y ago

> If you weren't there, you don't know. Technically speaking, for any given situation there's no way you can be wrong. But this post describes a distinct and, by my experience, dominant industry phenomenon. And the stakes are as high is OP says (dead products). Industry engineers in the large are massively divorced from product outcome. This should be no surprise, as it is a seller's market: Delivering product simply and directly can be, depending on perspective, somewhat of a mundane and Sisyphean activity. Engineering "stuff" is far more intellectually engaging, and there's no market pressure (for engineers) against over-engineering. If my company fails to deliver a product outcome, whether a viable company or startup, in this market there is no recourse at all on my salary let alone my hireability. I say "depending on perspective" because if you can find a way to be more intellectually and emotionally stimulated by shipping stable product over time, then ime you _can_ unite your joie de vivre with market outcomes. But that does not at all describe most engineers in industry who generally at best shed off some market value while spinning unnecessary webs of toy abstractions (ie liabilities).

what_is_orcas 5y ago

I politely disagree. I'm currently walking away from a code-base (and the company who let it be built) that was obviously built by someone who's mental model of how the code works was incorrect. At some very basic levels, it under-performs compared to literally every other example of this sort of code I've ever seen (and it's extremely common), which results in a subpar user-experience and an EC2 instance way bigger than necessary to run this sort of code. A lot of code running doesn't need to run in the first place, and due to the incorrect mental model, there's a substantial amount of code that doesn't run. Nobody really understands how bad it is, though, because it's barely tested (and the engineer who wrote most of the code has argued that we don't need tests, for various reasons) and all of the engineers on the team who didn't build it initially, have told me that they're not familiar enough with how this sort of product should work to understand that it doesn't work like that (again, this is a super common kind of product). There's a lot of other anti-patterns (I started keeping a list in my first week) but I think these are the most damning. This code is in production. Nobody at the org brought up these issues earlier in development (like saying "hey, nobody else's product does [annoying thing that doesn't need to happen], why does ours?"), which leads me to believe that the whole technical org is at fault. It sucks and I can't wait to be done with this experience.

im_down_w_otp 5y ago
That doesn't sound like over-engineering. That sounds like bad-engineering.
- what_is_orcas 5y ago
  I don't disagree, but I think that the reason that this fits here is because the poorly engineered software is unnecessarily complex (not in the "someday we might need this" way, but in ways that I'm having a hard time articulating without giving away the product because I know my coworkers are on HN). In fact, it was the ridiculous complexity that drove me to discover the disparities between the original developer's mental-model and the working-model. It just so happens that I've worked on this sort of thing before (as have thousands of others) that I was pretty quickly able to understand where the original developer went wrong (trust me, I'm just an ordinary developer, not a 10x elite hacker or anything).

activitypea 5y ago

In my experience, the situation you're describing more often results in "sloppy" code -- it seems convoluted, non-obvious, poorly thought out until you realize when the pivots happened. Overengineered code usually looks neat at first glance, maybe even impressive, especially to non-technical stakeholders. Then when you actually dive into it, you notice a bunch of wrong abstractions that introduce layers of indirection without offering any sort of valuable flexibility. "This metaclass that implements a context manager could've just been a slightly longer than average function" type stuff. A codebase that pivoted multiple times makes more sense the more time you spend in it, an overengineered one makes less.

PragmaticPulp 5y ago

> I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. The specs and the decisions surrounding the code are usually part of the over-engineering. > Everything was built to spec. Even with reasonable specs, I've worked with, and later managed, engineers who had a strong drive toward over-engineering solutions. I've also done it myself, and I suspect most people reading this have done the same. The problem is that many of us engineers became developers because we enjoy building things and we enjoy solving complex problems. It can be exciting to imagine a particularly complex version of a problem and how elegant and fun a complex solution would be. It can be as simple as a programmer who tries to write their own CMS when the spec could be solved with a simple blogging platform. If you're an ambitious programmer, writing a CMS is fun! Installing Wordpress is blech. I didn't truly understand the ubiquity of the problem until managing ambitious engineers who wanted to do complex things. At times it felt like 1/4 of my job was just asking "If we wanted to simplify this solution, what would that look like?" Repeat for a few rounds and eventually the team has shed a lot of unnecessary complexity without sacrificing the spec at all.

btilly 5y ago

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Can you prove it in a court of law? No. But you still can often easily diagnose it. And you'll usually be correct. For example the conditions under which https://wiki.c2.com/?SecondSystemEffect https://wiki.c2.com/?SecondSystemEffect happens are well-known, and the pattern has been repeated often in the half-century since it was named. If it is a second system, and you can find complicating abstractions in it that get used exactly once in practice, then with better than 90% odds it was over-engineered. And the farther you dig, the stronger the evidence will become.

SkyPuncher 5y ago

> Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. I would argue this is a case of over-engineering. Though, it extends beyond pure engineering. This is poor resource management and business de-risking. An effective approach here would have been: "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern".

beachy 5y ago
I've been in exactly this position, in fact I conceived and created the situation at my last company. I found it uncanny how on point OP was. The trouble with "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern" is that when you're building an integration marketplace, you need partners, and they don't want to be your research bitches. They just want to build their integrations once, and gtfo. If they think you're signing then up for a revolving wheel of experimentation, they'll just politely stay away until someone else has done the hard yards. Sure if you're a Microsoft or a Google you'll have any number of willing partners who will put up with anything to be the pioneers in your integrations marketplace. But otherwise, they're using your integrations marketplace purely for the benefits, and they don't want to be building on sand.

radley 5y ago

> Everything was built to spec. First line in the second paragraph of every dead company's obituary.

d0m3 5y ago

Well how do you know something is overengineered? It's when you know it will never (or too far in the future) need to scale accordingly to the engineering.