3 ms·
> What if there is no "complexity trap", just disagreements about how best to design programs? While I think this post asks reasonable questions, and has some
by dahart 8y ago
> What if there is no "complexity trap", just disagreements about how best to design programs?
While I think this post asks reasonable questions, and has some healthy skepticism, the complexity trap is very real, in my experience. After 20 years of professional software development, I can safely say I've watched literally tens of millions of dollars wasted over-engineering solutions to problems that were anywhere from smaller than advertised to did not actually exist.
I also freely admit that I personally tend to deep dive on cool/elegant/fast/small solutions where my time to achieve the optimal program is never recouped by running the program.
There absolutely is some under-engineering too, I've done some, and seen some in the wild. But on the whole my samples are that over-engineering is the dominant problem. That might not be true for people who hire contractors; from that point of view, getting less than you think you paid for is probably the most common experience.
- Raidion 8y agoWell, I feel like the environment of the job coupled with the developer causes over-engineering, rather than true technical concerns. I'm fairly young for my new role and architecting (from the ground up) some fairly big systems. I'm really struggling not to over complicate problems and I think it's really easy to do for a couple reasons. The selfish architect I'm working very hard not to be would rather have his work be complex. This builds ego if it works and you can always argue "it's a complex problem" if it doesn't. He'd rather face the shame of maintaining software that's overengineered as that's a very low visibility cost. No one bats an eye at estimates to update an existing system if it's needed. On the other hand, failure to account for a use case you deemed 'a bit complex for MVP' is super visible, even if it's a miscommunication between product and development. This selfish guy doesn't have eyes for business needs, he just falls in love with solving a problem, when he starts to elaborate on capabilities that it could have, product manager will just nod and say "oh yea, that might be nice", so you over-engineer to make sure that pathway is open, even if the product lady forgot about that possible feature before she left the meeting. He's also afraid of showing up at a meeting saying we solved the problem with a boring tweak on an existing implementation and having someone show him up by asking questions about why he didn't just use a software platform or language that felt foreign and complex when he read a few articles about it on reddit. Idk, felt good to vent some of this stuff, but those are the things I have to make sure I'm aware of, otherwise I'm just gonna build shit to the ceiling.
- dahart 8y agoI agree with all of that. My first reaction is that you’re probably in startup land where you’re mainly building new things. People do bat an eye at estimates to update old and large existing systems. It’s the new and small ones that are easy to update. If you’re even talking about MVPs, you’re probably not seeing much over-engineering yet. Or, you might have some and not know it yet, it might take a couple of years before you realize you paid large opportunity costs solving issues that seemed important. Having to decide whether to solve an issue before vs after MVP can be hard, but if you know you need to solve it either way, you’re way ahead of solving something that you aren’t absolutely sure you’ll need. A common but not-common-enough (IMO) programming philosophy is to only write the minimum code necessary to solve problems you actually tangibly have, not for any problems you think you’ll have, only for problems that absolutely require solving now and people are screaming for them. Extreme programming and Agile and lots of other software development frameworks all have variations on this tenet. The biggest over-engineering mistakes I’ve seen are when teams decide to rewrite from scratch a code base that they deem too crufty and too difficult to maintain and update anymore. Devs make this call far too easily, and fail to take into account how many things still work and how much effort it took to get there. I’ve been witness to two very large such rewrites that, many years later, both companies admitted were colossal mistakes.
- hinkley 8y agoWhat really burns people is making irreversible changes and finding out you chose wrong. Some people avoid this by trying not to make any decisions at all, and that’s more than enough fuel for long a journey into overengineering. Someone (possibly Michael Feathers or Uncle Bob) called these people cowards and I tend to agree. These days I’m very aware that smart and fool aren’t at opposite ends of the same ruler. I’m much more concerned with being wise instead of clever. Clever requires too much busywork for everybody, and it’s not conducive to teamwork.
- shrimpx 8y agoWould you say that these over-engineered solutions actually grasped the problems, and over-engineered the solutions? Or did they not grasp the original problems to begin with, and solved tangential problems instead? In my experience it's often the latter that leads to accrued accidental complexity and technical debt. Edit: grammar.
- dahart 8y agoI’ve seen both of those and more. Actually I’d say that from my point of view, there is no such thing as fully grasping the problem while over-engineering it. I’d categorize that as slightly not grasping the whole problem. The complexity of the solution and the budget for solving the problem are part of the problem, not something separate and distinct. I think I’ve more often seen tangential and imagined problems than people doing the right thing the wrong way. To be fair, it can actually be very hard to know whether a problem is really truly a problem that needs to be solved. Some tangential and imagined problems seem important and real, and you don’t discover until later you needed to solve something else and not the thing you did. Some examples: At a games company, the two mathiest programmers decided we needed to be using geometric algebra instead of matrices. They spent years working on it, and the code infected the rigging and animation and rendering systems, all kinds of things. Most of the programmers had to interface with it, but didn’t know geometric algebra, so it slowed development. It only solved minor issues. One of the claims was we’d have no gimbal lock, but people already solve that with matrices in one or two functions, it’s a non issue. To top it off, the math all got a bit slower, and we had to maintain the matrix math anyway to interface with the hardware and external libraries. At a film company all the programmers decided they were tired of the renderer and they needed to start from scratch because making updates was getting slow and difficult. They scrapped the renderer, started over, and it took many multiples longer than they estimated. When they were done, they had roughly the same complexity and ugliness as before, and making updates didn’t get significantly easier. Some mistakes were made in the rush to rewrite everything. Personally, I built a multi-user whiteboard website, and I over-engineered the rendering and the multi-user undo system, convinced they both needed to be perfect. In the mean time, discovered that very few people care about those things or ever hit corner cases, and I missed the opportunity to write proper integrations with other tools and import/export tools — the things the customers needed in order to adopt my software for their workflows.