158 ms·

> The developer's incentive to maximize their own lock-in factor and billable hours are powerful forces. In my experience these are rarely, if ever, the reason

> The developer's incentive to maximize their own lock-in factor and billable hours are powerful forces. In my experience these are rarely, if ever, the reasons for over engineering.

JauntyHatAngle 3y ago
I never though so either, but then I worked at a place that had a stale product, product teams powerless and developers rejecting most features/writing random code all the time. It took me a month to realise that the staff developers very much revelled in and protected their bad code and bizarre domain choices. It was so far gone that there was no way to get rid of them and the product was just slowly dying and burning the remaining cash. Then the mergers happened and they all got let go, only retaining the name for brand power and the entire stack was quietly moved over to another similar product which was rebranded. Separately, there were absolutely very large consultancies that had a programming style/rules based on making their implementations difficult to read/modify, needing to call their COE back in to fix their code or add features - with it being very hard to modify. Talking entire codebase structured with ridiculous levels of abstraction and annoying code style. Bad integrations requiring their tooling to work and make sense of etc. They target traditional orgs where the management just wants to get a project through and then bleed them over years.
- whatshisface 3y ago
  >It took me a month to realize that the staff developers very much reveled in and protected their bad code and bizarre domain choices. I think personality can account for this without any reference to incentives, which come in to explain how this personality problem can be so common among successful engineers.
neilv 3y ago
Lock-in factor and billable hours can be motivations for contractors. But, for contemporary software developers in general, I'd guess more often it's either resume-driven development (e.g., add the big complex new framework keyword to your resume) or not yet having enough experience to know what complexity is worthwhile for a situation.
- adonovan 3y ago
  In my experience most over-engineering can be explained by a lack of understanding. When we design it, we don't know what matters and our guesses are wrong; and when we modify it later, we don't have time to figure out how it really works. Both of those problems can be fixed by simply spending more time to understand, but time is money.
  - j4yav 3y ago
    I think, related, is the hope that switching to a new framework will somehow solve everything annoying about the old stack, and let you undo past mistakes. In reality, though, if you're lucky enough not to end up supporting two stacks at once for a long time, you end up making lots of new mistakes again.
  - PH95VuimJjqBqy 3y ago
    Not a lack of understanding, a misalignment of values. I had a team once implement the MediatR pattern for a < 5k LoC codebase (I'm guessing at its size but it was made larger due to the overhead of the MediatR pattern). When I asked them to remove it, it became a political fight that went to the VP because they were convinced that sort of flexibility was a good idea. Fast forward a year and they have a new technical leader and he thought I wanted that complexity until we had a conversation and when I mentioned I didn't like it he confessed to me he wanted to rip it all out. People value the wrong thing too often.
    - randomdata 3y ago
      > they were convinced that sort of flexibility was a good idea. Believing flexibility is needed still suggests a lack of understanding.
      - Terr_ 3y ago
        I've often say "design for deletion": Most of those long-term "flexibility someday" needs are best-met by making sure the inflexible modules or flows can be clearly identified and ripped out for replacement. This leads to a certain kind of decoupling, although with a higher tolerance for coupling that can kept in check by static analysis.
jauntywundrkind 3y ago
Today's best post on complexity to me was the I accidentally saved my company half a million dollars, which is a story filled yes with a lot of poor developers but much much worse is a story filled with Conway's Law style lessons. Of madcap organizations & wild legacy systems that we live atop & typically just have to make do with. https://news.ycombinator.com/item?id=38069710 https://news.ycombinator.com/item?id=38069710 https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-million-dollars/ https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-... So much of the grug-brained argumentation is same-side violence, is developers roasting developers. Assigning bad motives & declaring immoral bad people among us probably takes already same-side disdain & amplifies it, forments real & powerful hatred. Yes, various petty or criminal over engineering happens some, yes. But I usually think there are simpler Hanlon's Razor explanations that aren't even the individual's fault, are just the story of an org: an organizational body where so many constituent members of the body-whole have so little idea what others are doing or have done, and access to so few who can offer informed situational-appropriate wisdom for whatever dark end a squad finds itself tasked with. In some ways, this has been the actual story of open source. We have not been end user facing. We have grow communities of practitioners with individual knowledge & experience that ports & can be compared & discussed with others, from outside our company. We get peership that companies can rarely afford or find after they grow mid-sized. The Influx folks just talked about their 3.0, & replacing their custom built engine with Flight, DataFusion, Arrow, and Parquet. Maybe they might miss one box or two on neat nitty gritty optimizations (maybe this guy elsewhere in this thread, insisting on diy'ing binary encodings because protobuf doesn't have 16-bit ints can find some deficiencies, https://news.ycombinator.com/item?id=38078133 https://news.ycombinator.com/item?id=38078133), but the nice to haves of ecosystem integration seems like an intangible, but one that lets little good things out there plug in & help out, and that seems invaluable. https://news.ycombinator.com/item?id=38013714 https://news.ycombinator.com/item?id=38013714 https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ https://www.influxdata.com/blog/flight-datafusion-arrow-parq... The top comment to me epitomizes the worst of grug brainedness. It's smug & certain of itself, a strong & divisive Hanlon's violation of high degree. That kind of blue-on-blue attitude to me is far worse than 99% of the people who end up over engineering. Few organizations have real defense, lack coherency, to help themselves away from complexity, or they have such tight constraints that they overprune & disaffect those who do care and have good ideas. These are organizational issues, and having an org that can somehow see itself & what's afoot is the huge challenge. Stop hating on devs, please.
lawn 3y ago
My hobby projects are always over-engineered despite my best efforts. There's absolutely no monetary incentives here so anecdotally I agree with you.
- Defletter 3y ago
  Yup, but it goes both ways. I end up over-eningeering as some feeble attempt to avoid technical debt, only to realise my over-eningeering /is/ the technical debt. Or I end up with relatively simplistic / specialised code that needs an entire rewrite and migration process any time something is added. Either way, it's a bunch of rewritten code mixed with paralysing anxiety about writing bad code.
  - Reefersleep 3y ago
    Get used to, and even good at rewriting. I enjoy it, and I realize that I may never have the best idea of what's appropriate at one point in time, but rather accumulate an approximation over time as I reshape the code to the best of my knowledge. And I develop tools and practices to aid me in refactoring faster and with more confidence. Most importantly, keep throwing yourself into it.
  - mmmm2 3y ago
    I think we as developers underestimate the value of a 100 line Python script. Everything could be hard coded and inflexible, but it's still easy to refactor because you can keep the whole thing in your head. "Bad" design can be fine if it's kept simple.
  - nuancebydefault 3y ago
    How refreshing it is to read these very honest findings about oneself's abilities!
- jongjong 3y ago
  I also did a lot of hobby projects and open source work. Still, in my early career, I was over-engineering everything including my own hobby projects. I think I was trying to put my signature on the work and I unwittingly achieved this through unnecessary complexity. I was thinking of good code as something I had to invent, but now I feel like it's more like something I have to discover.
bryanrasmussen 3y ago
In my experience over-engineering also often doesn't have the result of locking in devs, although their billable hours may increase in the short term.

> The developer's incentive to maximize their own lock-in factor and billable hours are powerful forces. In my experience these are rarely, if ever, the reasons for over engineering.

JauntyHatAngle 3y ago

I never though so either, but then I worked at a place that had a stale product, product teams powerless and developers rejecting most features/writing random code all the time. It took me a month to realise that the staff developers very much revelled in and protected their bad code and bizarre domain choices. It was so far gone that there was no way to get rid of them and the product was just slowly dying and burning the remaining cash. Then the mergers happened and they all got let go, only retaining the name for brand power and the entire stack was quietly moved over to another similar product which was rebranded. Separately, there were absolutely very large consultancies that had a programming style/rules based on making their implementations difficult to read/modify, needing to call their COE back in to fix their code or add features - with it being very hard to modify. Talking entire codebase structured with ridiculous levels of abstraction and annoying code style. Bad integrations requiring their tooling to work and make sense of etc. They target traditional orgs where the management just wants to get a project through and then bleed them over years.

whatshisface 3y ago
>It took me a month to realize that the staff developers very much reveled in and protected their bad code and bizarre domain choices. I think personality can account for this without any reference to incentives, which come in to explain how this personality problem can be so common among successful engineers.

neilv 3y ago

Lock-in factor and billable hours can be motivations for contractors. But, for contemporary software developers in general, I'd guess more often it's either resume-driven development (e.g., add the big complex new framework keyword to your resume) or not yet having enough experience to know what complexity is worthwhile for a situation.

adonovan 3y ago
In my experience most over-engineering can be explained by a lack of understanding. When we design it, we don't know what matters and our guesses are wrong; and when we modify it later, we don't have time to figure out how it really works. Both of those problems can be fixed by simply spending more time to understand, but time is money.
- j4yav 3y ago
  I think, related, is the hope that switching to a new framework will somehow solve everything annoying about the old stack, and let you undo past mistakes. In reality, though, if you're lucky enough not to end up supporting two stacks at once for a long time, you end up making lots of new mistakes again.
- PH95VuimJjqBqy 3y ago
  Not a lack of understanding, a misalignment of values. I had a team once implement the MediatR pattern for a < 5k LoC codebase (I'm guessing at its size but it was made larger due to the overhead of the MediatR pattern). When I asked them to remove it, it became a political fight that went to the VP because they were convinced that sort of flexibility was a good idea. Fast forward a year and they have a new technical leader and he thought I wanted that complexity until we had a conversation and when I mentioned I didn't like it he confessed to me he wanted to rip it all out. People value the wrong thing too often.
  - randomdata 3y ago
    > they were convinced that sort of flexibility was a good idea. Believing flexibility is needed still suggests a lack of understanding.
    - Terr_ 3y ago
      I've often say "design for deletion": Most of those long-term "flexibility someday" needs are best-met by making sure the inflexible modules or flows can be clearly identified and ripped out for replacement. This leads to a certain kind of decoupling, although with a higher tolerance for coupling that can kept in check by static analysis.

jauntywundrkind 3y ago

Today's best post on complexity to me was the I accidentally saved my company half a million dollars, which is a story filled yes with a lot of poor developers but much much worse is a story filled with Conway's Law style lessons. Of madcap organizations & wild legacy systems that we live atop & typically just have to make do with. https://news.ycombinator.com/item?id=38069710 https://news.ycombinator.com/item?id=38069710 https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-million-dollars/ https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-... So much of the grug-brained argumentation is same-side violence, is developers roasting developers. Assigning bad motives & declaring immoral bad people among us probably takes already same-side disdain & amplifies it, forments real & powerful hatred. Yes, various petty or criminal over engineering happens some, yes. But I usually think there are simpler Hanlon's Razor explanations that aren't even the individual's fault, are just the story of an org: an organizational body where so many constituent members of the body-whole have so little idea what others are doing or have done, and access to so few who can offer informed situational-appropriate wisdom for whatever dark end a squad finds itself tasked with. In some ways, this has been the actual story of open source. We have not been end user facing. We have grow communities of practitioners with individual knowledge & experience that ports & can be compared & discussed with others, from outside our company. We get peership that companies can rarely afford or find after they grow mid-sized. The Influx folks just talked about their 3.0, & replacing their custom built engine with Flight, DataFusion, Arrow, and Parquet. Maybe they might miss one box or two on neat nitty gritty optimizations (maybe this guy elsewhere in this thread, insisting on diy'ing binary encodings because protobuf doesn't have 16-bit ints can find some deficiencies, https://news.ycombinator.com/item?id=38078133 https://news.ycombinator.com/item?id=38078133), but the nice to haves of ecosystem integration seems like an intangible, but one that lets little good things out there plug in & help out, and that seems invaluable. https://news.ycombinator.com/item?id=38013714 https://news.ycombinator.com/item?id=38013714 https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ https://www.influxdata.com/blog/flight-datafusion-arrow-parq... The top comment to me epitomizes the worst of grug brainedness. It's smug & certain of itself, a strong & divisive Hanlon's violation of high degree. That kind of blue-on-blue attitude to me is far worse than 99% of the people who end up over engineering. Few organizations have real defense, lack coherency, to help themselves away from complexity, or they have such tight constraints that they overprune & disaffect those who do care and have good ideas. These are organizational issues, and having an org that can somehow see itself & what's afoot is the huge challenge. Stop hating on devs, please.

lawn 3y ago

My hobby projects are always over-engineered despite my best efforts. There's absolutely no monetary incentives here so anecdotally I agree with you.

Defletter 3y ago
Yup, but it goes both ways. I end up over-eningeering as some feeble attempt to avoid technical debt, only to realise my over-eningeering /is/ the technical debt. Or I end up with relatively simplistic / specialised code that needs an entire rewrite and migration process any time something is added. Either way, it's a bunch of rewritten code mixed with paralysing anxiety about writing bad code.
- Reefersleep 3y ago
  Get used to, and even good at rewriting. I enjoy it, and I realize that I may never have the best idea of what's appropriate at one point in time, but rather accumulate an approximation over time as I reshape the code to the best of my knowledge. And I develop tools and practices to aid me in refactoring faster and with more confidence. Most importantly, keep throwing yourself into it.
- mmmm2 3y ago
  I think we as developers underestimate the value of a 100 line Python script. Everything could be hard coded and inflexible, but it's still easy to refactor because you can keep the whole thing in your head. "Bad" design can be fine if it's kept simple.
- nuancebydefault 3y ago
  How refreshing it is to read these very honest findings about oneself's abilities!
jongjong 3y ago
I also did a lot of hobby projects and open source work. Still, in my early career, I was over-engineering everything including my own hobby projects. I think I was trying to put my signature on the work and I unwittingly achieved this through unnecessary complexity. I was thinking of good code as something I had to invent, but now I feel like it's more like something I have to discover.

bryanrasmussen 3y ago

In my experience over-engineering also often doesn't have the result of locking in devs, although their billable hours may increase in the short term.