21 ms·

Overengineering can kill a product

• 0xdky 5y ago
  Wonder if the current hiring and rewarding contributes to over engineering? How would you look if you just invoked APIs to get your work done versus designed and implemented an end to end solution. Many times, I have seen over engineered solutions come from promotion attached initiatives.
• _drimzy 5y ago
  These are just fantastical musings of a product manager, who is pretty far from engineering, and thinks that their products fail because of engineers, not because they didn't get their product right, and/or think that every engineer who doesn't produce a fully functional Facebook with news feed and friends, in 39 days is overengineering their product and are far removed from users!
  • engineeringwoke 5y ago
    and then your product manager smiles during the meeting when you say this, and then bursts out laughing when the call is over
• r_singh 5y ago
  Over-engineering can inhibit or delay your work (art work) from becoming a product, but it cannot kill your product alone. In some cases over-engineering may even lead you to stumble upon interesting discoveries or to innovate.
• jcun4128 5y ago
  I'm wondering about this now I guess it is old fashioned now to use environment variables and bare EC2 servers, managing your own APIs and websockets/DB on same server as opposed to breaking everything out. You need to use cloud formation and "oh did you know there is an AWS service for that?" Then you are using 5 services instead of 2. This twelve factor app concept. Don't know when is the right time to do this/at what scale.
  • GaelFG 5y ago
    my two cents : until you hit scaling wall (and when you will congratulations, you are either successfull or a video streaming platform), a big server to upgrade is the best way to go and focus on building product. Then you hit performance problems, any sysadmin could help you handle 2x/4x/10x scaling with simple separations of service and maybe some hours of downtime. In the meantime you probably have weeks/months to think about going really crazy with your infrastructure.
    • jcun4128 5y ago
      > until you hit scaling wall What kind of numbers am I looking for? Thousands of users? Ops/sec? I realize it is partly due to what your thing is doing eg. website vs. a complicated app. Yeah it is a good problem to have assuming you have cash flow.
      • n0w 5y ago
        When your service can no longer process requests as fast as they come in, you've hit a wall. Until then the simple solution is to allocate more resources to your service (i.e. scale vertically).
        • maccard 5y ago
          There's obviously a balance to be struck but the more often you do something the easier it is to do. If you have a simple app with a db, backend, frontend and proxy, vertically scaling every time you hit that wall is going to be very painful. A little complexity goes a long way - using a managed system adds a very small amount of complexity in exchange for some breathing room when you need it. The last thing you want when your service is in a death spiral is to start thinking about the practicality of migrating your db and taking the hours of downtime/working overnight/weekends to do it at a convenient time for your customers.
      • GaelFG 5y ago
        Of course it may vary a lot but i would say a hundred of thousands of users (100 000). To explain the number, I assume a user will spend 1% of it's time on your app (that mean the average user spend around 15min per day (EVERY DAY) on your app, that may not sound impressive but it aldready is) and an nginx server on a powerfull machine could handle around 1k connection at the same time. If you want to dig about an exemple of number far less abstract I remember stack overflow published their settings : - https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-... - https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... The 2013 article state they could handle 148,084,883 page load per day with a single database server (but they did use two) for exemple.
• nisa 5y ago
  Complexity kills your product - Overengineering is just one instance of complexity - Technical Debt like having state and data all over the place is another one - I quit my last job and I happily blame this article for convincing me to quit: https://itnext.io/the-origin-of-complexity-8ecb39130fc https://itnext.io/the-origin-of-complexity-8ecb39130fc - coordination causes complexity and this killed me - we had everything not once but twice or more in different places - just one example - it's much more complex but me and my colleagues were grinded between an old codebase that nobody understood anymore and kubernetes, ci, etc.pp on the other side because management sold this without understanding that you need a process and time for digesting and applying the concepts in the team.
  • baby 5y ago
    I interpret “early over engineering” not as adding complexity necessarily. It could also be about decreasing complexity but by taking too much time. Early on you’re supposed to rush to your MVP and add tech debt, not spend too much time to design something pretty and modular. Later on once you know your company can survive multiple quarters, then you can spend more time refactoring.
  • jasfi 5y ago
    I agree, complexity is the real underlying problem. That is why good frameworks and libraries help so much.
    • 123pie123 5y ago
      >That is why good frameworks and libraries help so much. until they change - again.
      • jeffkeen 5y ago
        Gotta keep up with the updates
        • amelius 5y ago
          My dependencies can't keep up.
• zeitgeist_y2k 5y ago
  The most important thing is to find the right balance. This article goes into one direction. But to be honest, most of the time I see it shift into the other direction: in product driven orgs the drive to implement features and bring them to market quickly is more important than engineering quality. But in the end you end up with something where implementing new features takes so much time because of the complexity that built up because you started to implement a product where the specs where unknown in the beginning and only materialized later. That's the point when you should re-engineer your system. But yeah... it's all about balancing and finding the sweet spot.
  • whizzter 5y ago
    To put more succinctly, the least code wins. What we're talking about 2 opposite problems that cause the same end problem, too much code. Overengineered what-if everywhere often causes way too complex systems where even simple things become hard, de-engineering is nigh impossible because it'd break things by taking away functionality (unless an alternative manifests itself). Non-engineering hackjobs where we basically have N copy-pastes or more or less the same functionality everywhere, cleaning up anything will require a fair bit of testing to avoid breaking anything in the process, possible but causes a lot of uncertainty and risky if you won't see it unless failing in prod (because it goes hand in hand with bad deployment practices). As mentioned in the article and above already, you need people with experience and to give those the business knowledge to find the correct balance.
• revskill 5y ago
  Just use cron job as escape hatch for the backend is always a workable solution to me.
• FriedrichN 5y ago
  Another problem is that these overly complex systems are often very fragile when something changes. In a very theoretical situation there could be a case where someone with an overengineered client consumes your JSON API and their client breaks when you add a field to a certain service response. Something that should have been no problem suddenly causes total breakage of your software and then you'll have to alter that very complex piece of software. Something that could've been prevented if you kept things simple and robust and simple chose to ignore extra fields or headers you weren't using anyway.
  • jrochkind1 5y ago
    That's indeed the irony. The over-engineering happened in an attempt to be resilient to future change, but the outcome can often be the opposite. We've all been there, I think?
  • 9dev 5y ago
    > In a very theoretical situation I've had this exact ticket in the past: A customer built a complex Java client for our public API which assumed no surplus fields in the responses, and as we added a new field, their entire application broke down, causing huge losses for the customer. I wasted so much time on explaining how our stability promise does not extend to added fields!
    • damethos 5y ago
      I do not understand how is this over-engineering. Over-engineering means that you worry too much for future/edge cases that might or might not come and you want them covered too early. The fields situation looks like bad engineering unless the reason that the "no surplus field" has an explanation which falls under "covering future needs" that escapes me at the moment.
• quickthrower2 5y ago
  YAGNI
  • boyadjian 5y ago
    So true. YAGNI is the cure for overengineering
• crate_barre 5y ago
  If you find yourself over-engineering due to stuff you think you should be doing, things that ‘real developers’ do, understand you are vulnerable to being pimped. It’s no different than anyone else handing over their common sense and self worth in pursuit of an abstract form of validation (so abstract that you can internalize the validation cycle even in the absence of a physical superior). If you find yourself defending your bad over-engineering and those that sold you the lifestyle, understand you are fully pimped and are now defending the pimp, even to your own detriment. Happens to all of us from time to time, snap out of it. Don’t get pimped by ‘faangs’ or ‘notable person of interest’. Don’t listen to everything I said either, lest you want to get pimped by me. Unless you can wholeheartedly make an objective argument for why you used a pattern, a tech stack, a process, in plain simple words, sans ‘that’s how the big boys do it’, sans ‘this makes me a real developer’, you are simply pimped and spewing out pimped out thoughts of your overlord pimps. Be ready to be wrong and backtrack and own the mistake and correct course, but don’t you dare hold on to it, or I’ll probe to see who is pimping you out. I was never more free from my overlord developer pimps until the day I realized they evangelized impractical solutions. Never hoe’ing for anyone ever again.
  • crispyambulance 5y ago
    I share this sentiment BUT... It's also important to try stuff out, fail, recover, and try again. That "Code Complexity vs. Experience" graph in the article is not completely a joke. Very few people can tunnel through the complexity hump without years of failures and successes behind them. Moreover, you might not have a choice. You might find yourself dropped into an obstacle course of complexity that other people created and that you have to keep running following their arcane patterns and practices-- while at the same time implementing new features and refactoring it into something workable before it becomes completely intractable. I think almost everyone faces this problem (except for maybe the most orderly and elite workplaces?).
    • crate_barre 5y ago
      My message was mostly for those that hide behind mistakes via an appeal to authority. Trying new things is a risk, which is fine, but to own the the success of the risk means you must also pay the collateral of owning the failure. The message is for those who don’t put up the collateral and hide behind ‘this is what everyone(the pros /sarcasm) is doing’ or ‘this is how it’s done’, and never reflect objectively. No one would argue against trying things, it’s where all creativity and innovation comes from. I argue against dysfunction, the whole ‘the ship is not sinking’, when in fact it is. Anyway, perhaps I’m speaking too personally, because I am on a literal Titanic right now, so apologies for that.
• ChrisMarshallNY 5y ago
  I've written a lot of complex stuff. In fact, I'm doing it right now. There needs to be a "sweet spot," where we have enough complexity to achieve goals (which can be business and cultural, as well as technical), and as much simplicity as possible. A lot of folks think that using dependencies makes the project "simpler." It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc. Unfortunately, you can't do complex stuff, simply. Some level of complexity is always required. Managing that complexity is where we get to play adults. T.A.N.S.T.A.A.F.L.
  • hobs 5y ago
    Complexity is also one of the best moats in the business, if you have a "gross" problem with many edge cases there's few people who will want to eat your lunch, and most next generation up and comers often tout their simplicity compared to your old complex product because they certainly cant argue feature parity ;)
  • pkolaczk 5y ago
    Right, using too dependencies don't make things simpler. They make them easier as long as the dependencies can do exactly what you need. However sometimes you come to a point where the dependency can't do something that's needed (often by the time I realized a third party lib can't do sth, I could have implemented it by myself without the dependency) or where you've accumulated so many dependencies that they conflict with each other or where connecting two different libraries is the really hard part. No silver bullets.
  • jrochkind1 5y ago
    "as simple as possible but no simpler". Figuring out the sweet spot in architectural design seems to me still an art/craft, that comes from intuition and by experience. [As far as dependencies... using dependencies may sometimes be simpler than the alternative(s), and sometimes not, sure. It depends on the nature of the dependencies, the problem, and the alternatives. :) ]
  • Griffinsauce 5y ago
• a_square_peg 5y ago
  On the other hand, simplifying an already over-engineered product is almost next to impossible simply because many jobs depend on it. Maybe I'm cynical but I'm beginning to think that software complexity grows until it justifies all the head counts in the department.
  • darkerside 5y ago
    This is the same phenomenon as the Peter Principle, where people Rose to the level of their incompetence. It sounds like a joke at first, but, of course they do. People get promoted when they excel. When you are in over your head, you stop getting promoted. Of course, over time, people grow into their roles, and regain their competence. Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
    • Griffinsauce 5y ago
      > Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage. I'm developing a principle of radical simplicity to attempt to combat this: always keep things absolutely as simple as they can be. It is always easy to add complexity later, never to remove it. So the only conscientious choice you can make is to keep things as simple as possible while satisfying the requirements. You should also critically evaluate requirements that introduce complexity. Also, I don't think people mention this enough here: complexity == bugs.
      • FpUser 5y ago
        Ability to keep things "simple" (basically produce code / architecture that is easy to understand) when solving problems in business domains (which usually are very complex in real live) is an art that few can successfully practice.
        • axiosgunnar 5y ago
          Exactly. It's actually very very hard to write simple code, no matter the problem at hand. It's easy to write convoluted, complicated code.
• coldcode 5y ago
  Making overly complex products developed in an absence of engineering input is also a major issue, at my last employer this was endemic to everything product threw at us. If you have no concept of how difficult something is to build you have no reason to not require it; by the time it gets to engineering its either too late to change or results in endless CRs to fix.
• uldo 5y ago
  At first I thought that the article will be about putting so many features, that “in the end it will be able to send e-mail”. Because this is what is killing products - adding, and adding and redoing design as if users really were begging for it. And then comes new product that is leaner and more simple and takes over the old product and cycle starts from the beginning. I think that in capitalism it is not possible to not overengineer a product.
• scottious 5y ago
  The insidious thing about over engineering is that it's usually committed by very experienced engineers. Experienced engineers rarely under-engineer, that tends to be fixed very early in one's career. As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way. Best real-world example: I inherited a project that was an giant "microservice" with 50-ish endpoints, and a Mongo database and dozens of collections. After probably 3 months of wrestling with this thing I had a realization: "this whole thing can be just a simple command line tool and one collection in Mongo". That reduced the code by almost half and it became so much easier to work with. It's frustrating that it could have just started this way.
  • jrochkind1 5y ago
    OP suggests there's a hump in the curve of experience vs over-engineering; more experience correlates to over-engineering until it gets to a certain point, at which even more experience leads to less over-engineering. Is it true? I don't know, I think it matches my... experience. Part of it is that the "experience" needed to avoid over-engineering is helped if it's not just engineering experience, but domain experience too. If someone is constantly jumping industries/domains, it might take longer to get there. I think they still would eventually.
    • jaywalk 5y ago
      My own personal learning curve matches that graph almost perfectly. I'm probably not as close to the right side as I'd like to think I am, but I'm always trying to move in that direction.
    • magicalhippo 5y ago
      Often I'll tend to over-engineer the initial solution, but as I think about all the complex edge cases that needs to be handled etc I often find myself asking "does the customer really need all this". After discussing it with the customer, a simpler solution often emerges, where they change the requirements slightly allowing for a much simpler solution that might even solve the actual needs better. While it's not just down to experience, I'd say it has a strong influence on being able to see beyond the given requirements towards a better outcome.
• MattKimber 5y ago
  I think there's a very specific form of overengineering afflicting products currently, which I'd classify as "endless revisiting". This is where companies build something which works well enough, but then get trapped in a cycle of endlessly tweaking that one thing. Inevitably the amount of code churn in this one area combined with the need for some semblance of backward compatibility results in something that is fragile, complex and slow. Plus the annoyance as a user that whenever you use this thing, all your tools are in a slightly different place and work slightly differently to how you left them. IMO there's a need for better balance between "it works well enough, leave it alone" and "we haven't fully optimised this workflow yet" in product development.
  • spaceywilly 5y ago
    Yeah, I see this happening a lot as well. A prime example is Spotify. Their app is done. It has all the features it needs. Instead of just focusing on making those features work more reliably, they seem intent on doing a big UI redesign every few months, adding bloat to the app and making it more frustrating to use. It’s really rare to find companies that just build a thing and stop adding features once the core functionality is done.
    • yCombLinks 5y ago
      I'm amazed how every version has been worse than the one before. The best version was out over a decade ago.
    • jspash 5y ago
      See also: Dropbox, 1Password, Gmail. The list goes on.
• m12k 5y ago
  From a UI/usability perspective, one of the dangers with overengineering is the "wall of options" issue, where all users - even the majority of them that just need something simple - still need to read and understand all these advanced options. As a product manager and UI designer you have a couple ways to deal with this. You can choose an opinionated subset of features that make sense for a given niche and target only that demographic. You can go the corporate way, keep the wall of options and just require training for users. Or you can try the balancing act - choose a sane subset of features as the default, and hide the more advanced options, so they don't bother normal users, but still have them as possible options. There are many important choices regarding how much to hide, and where, and how to make it discoverable, and how to make it possible to gradually dig deeper, and for users to self-identify as someone that needs to dig deeper - and those details are often as much art as science. But get it right, and you've got one of those rare killer apps that both newbies and experienced users enjoy.
  • diordiderot 5y ago
    I feel like figma does the balancing act well
    • satyrnein 5y ago
      I must not be the target user. I don't want to tinker with gradients, I just want a button with some text in it. Yet, I see options for the former, but I had to carefully align two different elements for the latter. I'm sure this can all be solved with modeling out your design system or whatever, but as a product manager who needed to make a quick mockup, I found it a little overwhelming.
  • kwertyoowiyop 5y ago
    And if you get it right, people just say “That software is so simple, I could have written it in a weekend!” :-)
• yosito 5y ago
  Six months ago I left a company that was working on an overengineered product. Even worse than it being overengineered was that it was also under documented. Working on anything was a pain, because the CTO wanted everything to follow his well thought out, and frankly very cleverly engineered design patterns, but he couldn't clearly communicate what those patterns were. And the entire company amounted to transforming and cleaning data sets using in-house tools, which could easily be done with existing tools too. Both myself and the other senior engineer on the team left at the same time. I felt bad leaving them, because they were trying to grow and had a ton of funding and deals with FANG companies but they were struggling to find engineers that the CTO thought were smart enough. I didn't want to burn bridges, so I didn't end up telling them that the problem wasn't a lack of qualified engineers, it was an over-engineering CTO who struggled to communicate.
  • abraxas 5y ago
    Being ins a similar role myself, how do I ensure that engineers stay happy working on the project that we're working on? I'm finding myself actually doing the opposite of the CTO you mentioned and pushing them towards adopting more off-the-shelf components instead of maintaining homegrown stuff but I think I'm causing a degree of upheaval by doing this. Their justifications for push back however, often smell of sunken cost fallacy to me.
    • deleted 5y ago
      [deleted]
    • Lhiw 5y ago
      Off the shelf software isn't an instant win. You're signing up for lock-in, a set way of doing something, a boundary you can NEVER cross, and domain and language you will never be able to change. This language will leak into your software and may not be a good fit for the end user you're trying to serve. That's a lot of trade offs to avoid maintaining the subset of features you need from said software in house, being able to leverage internal knowledge, being able to streamline all your environments and tooling to suit your needs instead of catering to the needs of said software. This isn't an argument one way or another, it's just pointing out that there are trade offs you need to make consciously or otherwise. In your position I would be extremely concerned about pushing off the shelf software onto Devs that either lack the clout to be comfortable pushing back or lack the communication skills to clearly articulate all the trade offs really taking place. It's very easy for a boss figure to push through requirements with off the cuff pointed questions. The reports often need to push back with orders of magnitude more research and thoughtfulness than went into the question or suggestion.
• gravyjones 5y ago
  This overengineering happened on my team recently. I built a REST backend and the front end team was tasked with building 15 admin CRUD interfaces in react which would talk with the backend. A junior front end guy convinced the senior front end guy to build an entirely new dynamic CRUD layout system. I saw what they were doing and knew it wouldnt work out. People got upset at me for raising concerns so I kept my mouth shut. 6 months later the project was not even 25% complete, the senior dev left for another company, and the junior dev got shuffled to another job in the company.
• locallost 5y ago
  The biggest consequence for me was not mentioned: that your carefully planned design very soon becomes an obstacle to something a user actually wants done, at which point you say "we can't do that". Which is one of the worst things you can do. In this sense almost all of the systems I interact with, modify etc. are over engineered.
• bronlund 5y ago
  One would think a lot of investors learned their lesson after Juicero :D
• izolate 5y ago
  The "super simple code" at both ends of the graph aren't equivalent. The latter is more "simplest code possible". Overly simple, hacky, narrowly spec'd code produces the same tech debt as overengineering. Anybody who's worked in a move-fast-break-things type startup will know how much engineering resources are wasted on rewrites/bugfixes due to this. Ultimately, as with many things in life, you need to find the right balance.
  • jeremyjh 5y ago
    I think the graph needs to continue a few more years, but the author hasn't lived that yet. The "simplify" mindset is also something you can take too far without experience.
• 9dev 5y ago
  Interestingly, dang (HN mod) once mentioned in a feature request comment that he has a mental model of a complexity budget. This very closely fits my own perception: You can only add so much complexity to a system before it starts to break down under the load. Estimating that budget, and how much new features will consume, makes a good engineer in my opinion.
• kwertyoowiyop 5y ago
  Posts arguing against over-engineering are catnip for the HN crowd! In other news, kittens and puppies are adorable.
• lbriner 5y ago
  As others have said, it is unecessary complexity. Over-engineering is an ambiguous term. For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all the RAM you are storing them in and therefore you might want to make them smaller/store them off-board? Of course, you can come back and refactor later but it is so much easier for the person who writes that code to understand what can be done on day 1 rather than assuming it is cheaper to refactor later only when needed. I think a better take is for people to accept that no app lasts forever. If we build that assumption into our worldview, it will help us make better decisions. I still think some engineers think there is some perfection that means the app will live forever.
  • bluGill 5y ago
    > use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and you need to design your array to be cache friendly. Of course the above assumes your map/dictionary is on the hot path (seems unlikely), your code is too slow (probably subjectively, but in some real time applications this can be objectively measured), and you are writing in an efficient language. IF All of the above are true, only then should you go to the complexity of using an array to do the job of a map/dictionary.
    • noisy_boy 5y ago
      Also depends on expected changes to the no. of items. If you think it can shoot up, then you are better served by sticking to map/dictionary (and paying the minor penalty, assuming not in hot path) compared to starting with array for low-volume and then having to make code changes to map/dictionary to handle increased volume.
• lkrubner 5y ago
  On a related note, see "Don’t waste $1 million on devops infrastructure that you’ll never need": http://www.smashcompany.com/technology/high-availability-is-not-compatible-with-mvp http://www.smashcompany.com/technology/high-availability-is-... This is a true story about an entrepreneur who was suffering a bad case of perfectionism. Instead of showing his MVP to potential customers, he just kept investing in the software, chasing an almost impossible ideal of high availability
• Dave3of5 5y ago
  Overengineering is a non technical problem but of course most tech companies only interview devs on their technical skills. I've found is that once a dev has an idea of how something is suppose to work it's hard to get them to think in a different way about that thing. So when that thing doesn't work out for some reason they just keep adding exceptions and adding exceptions until you have the horribly over-engineered solution.
• somehnacct3757 5y ago
  I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20. Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.
  • BackBlast 5y ago
    Specs are often over engineered. This is part of the problem. Too much effort in rowing the boat, not enough effort in figuring out the right direction to point it first.
  • sokoloff 5y ago
    With the benefit of hindsight, I think you can conclude that the system was over-engineered. If you’re trying to answer only “was it over-engineered?” I think you can answer it with hindsight. If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.
  • noisy_boy 5y ago
    My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you are wondering about those same things, trust me, there is a reason/background - its not because the people who did it that way were stupid".
• mberning 5y ago
  In the enterprise I see this all the time. Step one of the project is lets look at kubernetes or whatever is hot lately. Even for something stupid with 10k users max. What they actually need is a 30 line terraform script and a preconfigured AMI.
  • what_is_orcas 5y ago
    Ha ha. This is why I used to throw things into simple kubernetes setups (that I just ran the same terraform scripts to create) and just tell the management "yeah, now it's ready for all of your features". I argued for a while, then realized I didn't have to.
• gampleman 5y ago
  One thing that bugs me is the notion that "Software rewrites are something you should never do", which is a mantra so often repeated that it has acquired the status of self-evident truth, despite the only evidence being (usually) presented is an example of a web browser from 20 years ago! (Which incidentally spawned Mozilla, so not exactly a complete loss; especially from the POV of society rather than shareholders, but I digress). Having rewritten a bunch of systems (sometimes several times) I can attest that it will not always lead to the death of the company. The trick is of course having modular enough systems that they can be rewritten from scratch in a reasonable amount of time. It can also be a great way to increase the simplicity of the system as typically the old version was designed with a very imperfect understanding of the problem and no operational experience servicing it; further learning were usually crudely patched on top and you often end up in a conceptual hodge-podge where words mean subtly different things depending on the context and translation layers need to be inserted between the contexts etc. Often a (good) rewrite starts by clarifying the conceptual model. I like the saying "clear writing reflects clear thinking", and in programming there is a lemma "clear thinking produces clear code".
  • cfn 5y ago
    I suspect that the main issue with rewrites is that the users or product managers see it as an oportunity to add new features or redo old ones extensively. In the end the scope of the rewrite is no longer a rewrite but a new product that is incompatible with the original it was supposed to replace. I have seen this happen a couple of times. A straight rewrite for technical reasons and well defined scope does not suffer these issues.
    • Quarrelsome 5y ago
      one huge issue with a lot of technical and product debt is that any re-write gets saddled with a huge dam of expectation bursting. Many people who have been told they can't have their feature for years because the volition of the software team has been close to zero (hence the rewrite) for so long suddenly push their demands onto the new product. Its hard for a re-write to focus on an MVP as a consequence. Arguably it can be better to float a completely different boat and see if it swims but that can result in a product positioning problem where you then have two versions of the same product but with an uneven feature set.
• leroman 5y ago
  A critical job of a system designer is to see into the future and know which components need to be made custom so that the business can have the flexibility to make the changes where it needs them. Over engineering is a term that is as useful as saying - product requirements list was too big - features were never communicated clearly (tree swing comic..) - right people / people with relevant experience were not hired.. I make sure to always "Engineer Just Enough TM"
• melenaboija 5y ago
  Interesting interview of Lex Friedman to Kevin Systrom co-founder of instagram where he talks about the beginning of the platform and overengineering. https://youtu.be/3pvpNKUPbIY https://youtu.be/3pvpNKUPbIY
• pdenton 5y ago
  Oh boy, does this ring a bell with me. I've already written 5575LOC according to cloc (and threw away 7449LOC in the process) and I'm far from finished writing the code to email the data from a contact form in PHP. But it ticks all the boxes! It's all OOP SOLID principles 99,6713% typed (according to psalm) Purposely written to be unit-testable in PHPUnit strict mode (but no actual harness yet) ...I'll show myself out
  • jamil7 5y ago
    Types and tests sound pretty sensible to me.
• AtNightWeCode 5y ago
  The best designs are so simple that people do not even understand that it is a design. That is one reason why it looks like senior devs write simple code.
• lngnmn2 5y ago
  Engineering here must be in quotes, because real engineering is about simplicity and justrightness, like any craft.
• wheelerof4te 5y ago
  I often find myself prefering languages with few abstractions even when some more complex language might be "better". Reason is that with "bare" languages like C, you can choose your own design more freely. Case in point: Serenity OS. Author of that OS replicated entire libc and still refuses to use C++'s STL. He created his own abstractions that he feels confident using. That is true engineering.
  • baby 5y ago
    You should check golang, it’s much more safe than C and has very few ways of doing the same thing. It makes every codebase look like simple code. It’s always a pleasure to read or refactor. That’s the only language that managed to do that imo.
    • wheelerof4te 5y ago
      I tried Golang while I was experimenting on Linux. Fun little language, but I haven't got used to it's type notation. What is the state of Golang's Windows support?
• danhoule 5y ago
  I agree with this statement; it's better to put some updates slowly over time but not to the point that it will transform the whole product.
• am391 5y ago
  The issue here isn't so much over or under-engineering, but rather "Are we building the right thing?" or "Are we building the thing right?" In a startup you don't know if you're building the right thing so trying to build it right is premature optimization. Since you have limited resources you really have to focus on ensuring that you're building the right thing...if you aren't it doesn't matter how well designed or built it is no one is going to use it. Once you've validated you're building the right thing you can start focusing on building it right, but by they you probably know where the pain points are and where you need to spend the effort. The trick with this though is that everyone needs to be aligned on that approach and be honest about the fact that corners may be being cut to get something working fast. Where I've seen this go badly is where a crappy initial version was built to get to market fast, but then no time was made available to address the defencies in the initial release.
• f322323jiojio32 5y ago
  German developers love to overengineer everything. They will even write a hello world app using Kubernetes and microservices, just because it is good for CV.
• dbg31415 5y ago
  So many things can kill your product. And I agree that having an engineering-led product can be especially prone to the dangers of over-engineering... but... Not having users / customers can kill your product. Not building the right features can kill your product. Not doing enough testing can kill your product. Doing too much testing can kill your product. Having toxic / inexperienced / unmotivated staff can kill your product. Having a bad marketing plan can kill your product. Not having enough staff can kill your product. Not having enough funding can kill your product. Technical debt of all kind can kill your product. Bad data schemas can kill your product. Under-engineering can kill your product. Over-engineering can kill your product. ... This is in no way a complete list, but from my experience the items on this list are ordered with the ones most likely to kill your product put at the top.
  • throwawayboise 5y ago
    Most products/startups fail. This is part of the reason why. But all of this is survivable if you have customers and revenue. That needs to always be the primary focus.
  • convolvatron 5y ago
    dont forget BigCo noticing you have an interesting idea and yanking the rug out from under you.
• rubiquity 5y ago
  I suspect under-producting has killed far more products than over-engineering. The ratio of decision making power to decision making abilities is way out of balance for most Product Managers. Even at the big tech companies I feel most PMs are unimaginative MBA types that can optimize but not innovate and have no grasp of the concept of opportunity cost. In terms of power structures, Product Managers decisions largely go unchecked in a lot of places. Engineering decisions face significantly more scrutiny, especially in places that work in short sprint cycles.
  • engineeringwoke 5y ago
    A product manager's decisions go unchecked most of the time because engineers and responsibility for business concerns are like oil and water. That's why PM exists, no? If engineers could just talk to marketing, sales, etc. then there wouldn't be a need for them, but alas, that is not the case.
    • _drimzy 5y ago
      No one is denying the need for PMs. OP is pointing out that PMs have too much decision making power, with too little accountability in most organizations. Your argument is akin to, "PMs can't code, so alas we need engineers, and that's why shitty engineering exists, and there is no way to make it better"! Nope! We need product practices, akin to engineering practices, with 360° feedback and analysis, and the product management should be held accountable for their decisions!
      • engineeringwoke 5y ago
        Right. And I'm saying that is because engineers don't want the responsibility that they often request because the entire point of PM's existence is offloading that responsibility. They seem more than happy to complain about it though
        • _drimzy 5y ago
          This is still missing the point. The way to improve PM accountability isn't engineers fixing them. It's the organization's and leadership responsibility to ensure PMs are held accountable. Engineers would be far happier if they don't have to do good engineering and can do away with shitty software without accountability. But there are checks and balances to improve engineering quality, and those aren't created by PMs. It's the engineering org that champions good engineering practices and accountability and post mortems. Same should be done in PM organizations.
• GuB-42 5y ago
  While "overengineering" is usually defined as making things too complex, I think it is a bit misleading, that's because simplifying, removing or making things cheaper is also engineering. In fact there are many well paid engineers who work full time simplifying things. As much as I hate the cult of Elon Musk, I have to admit that's one aspect of engineering he really gets, and I suggest you listen to his interviews with Tim Dodd / Everyday Astronaut, where he gets technical, trust me, it is not the usual bullshit. All that to say that "overengineering" is bad doesn't mean you shouldn't have a lot of engineers working hard on a problem. In fact, a lot of "ovengineering" is the result of not enough engineering. They picked a (complex) solution without thinking instead of really studying the problem.
• yuriy-yarosh 5y ago
  Seems too oversimplified and biased. Probably missing few core points: 1. Without good development practices due to lack of Experience or plain old Rational Thought the only possible outcome is Operational Deficiency. Every Best Practice is context-dependent, thus having Deficient Resources makes them inapplicable in certain cases. Some teams can really suck with the same tech stack when others flourishing using it. 2. Basic organizational anti-patterns, like Mushroom Management, and broken retrospective lead to rediculous outcomes. Even plain old Micro-services and Micro-frontends can be a basis of Stovepiping and applying Mushroom Management. Usually, again, due to Lack of Competence and Sheer Hubris. 3. “Premature Optimization” only used in context of over-engineering by those who didn’t read the book, but use Halo-effect cognitive bias to project compensated qualities onto the term itself. There are a lot of Psychological Compensational Needs under the hood. It’s like “Why Agile has nothing to do with Discipline ?” or “Why senior developers turning the project into a sandbox due to the lack of self-fulfillment ?” or “Why most of the MVP’s lack Concise and Validated Definition of Viability ?” Complex doesn’t mean Hard or Expensive. Simple doesn’t mean Easy or Cheap. Too often “over-engineering” is just an organizational and psychological issue and not an Engineering one. Stop operating on Feelings. Six Sense of the Fifth Body Anchor Point is not a reliable Key Performance Indicator.
• zackmorris 5y ago
  Engineering within the wrong paradigm can kill your product. I see far more problems on a day-to-day basis with patterns and anti-patterns taken too far. For example, I never use the factory pattern, because it leads one down the Java road where everything ends up an object with mutable state. Which isn't scalable over some metric (like a million lines of code) because a human brain can't trace execution, even with a debugger. A far better pattern generally is to take a functional approach of accepting data, swizzling it, and returning the resulting data without mutability or side effects. Another really bad pattern is when execution suspends and resumes somewhere else (goto hell). Any project which uses queuing, eventual consistency, promises, nonblocking streams, even basic notifications or things as complex as monads will encounter this nondeterminism and inability to statically analyze code. Note that pretty much all web development suffers from some aspect of this due to its async and multi-signaling nature. So what I do now, which I don't see much these days, is solve problems abstractly in a spreadsheet (pure functional programming), in the shell (the Actor model) or as a flowchart (declarative and data-driven design), and then translate that to whatever crappy language/framework I have to use for the project. I find that today, roughly 90% of developer effort goes to discovery, refactoring and testing of ill-conceived code. Only 10% is "actual work" and that's probably a stretch. Which is incredibly heartbreaking for me to see, since I grew up on software like HyperCard, FileMaker and Microsoft Access which solved much of this in the 1980s and 90s in a no-code fashion. One of the very first "languages" I used was a visual programming environment called Visual Interactive Programming (VIP) for the Macintosh by Mainstay, which unfortunately today I can find almost nothing about, to show what an impact it had on computer science hah: https://duckduckgo.com/?q=Visual+Interactive+Programming+VIP+by+Mainstay+Macintosh&t=h_&ia=web https://duckduckgo.com/?q=Visual+Interactive+Programming+VIP... With mainstream languages and frameworks like Node.js and React, and their predecessors like Ruby on Rails and Angular, I just keep thinking to myself "never have I seen so much code do so little". It's all overengineered man! Some better alternatives to the status quo: Simple Made Easy: https://www.youtube.com/watch?v=LKtk3HCgTa8 https://www.youtube.com/watch?v=LKtk3HCgTa8 Object-Oriented Programming is Bad: https://www.youtube.com/watch?v=QM1iUe6IofM https://www.youtube.com/watch?v=QM1iUe6IofM
• lonelyasacloud 5y ago
  Fwiw, my experience is that generally a lot of blame for over engineeering tends to be lumped on engineering when in reality its usually a wider business failure. Every competent engineers I have met has been capable of grasping that: - They shouldn't rely on the use of crystal balls. - Complexity should be minimised. - If it can be done today, it can be done tomorrow. So if a business provides its engineers with: - A process that helps them develop an intuitive understanding of their customers' needs. - At least one other similarly capable person to work with so they know they're not the only person going to be looking after the project off into the future. - A procurement process for off-the-shelf solutions that is less painful for them than rolling their own. - Time to test and document where projects should go if the initial version is successful. - And crucially, confidence that they'll be given enough time to do the additional work if it becomes necessary. Then the business, at least in my experience, will be in a pretty good position to prevent almost all over engineering.
• HEmanZ 5y ago
  "Stop overengineering" was the excuse I heard back on a rough project when I insisted we add logging beyond just the returned HTTP status code to our service before shipping to 200M+ people. But that would take an extra week or two in the current system and that's just too much investment for some silly over engineering. Had we gotten decent logging in place early, we could have saved a terrible change that got passed our canary rings, which got us called to see the CEO and made news. So, I learned early that "overengineering" can also be a management excuse for cutting corners that shouldn't be cut.
  • _drimzy 5y ago
    Overengineering is typically a term thrown around when a manager wants to ship a system in half the time it actually would take to make a half-decent system!
• m3kw9 5y ago
  Over engineering also have a way to burn people out when a very vocal engineer say it is imperative that this and that should be scalable and secure. It delays actionable areas before people even want to hack you or use you
• dominotw 5y ago
  Overengineers will find an edgecase or an hypothetical future scenario to justify their overengineering. unless there is an org level commitment to not using "what if in future" type scenarios, overengineers will win everytime.
• jugg1es 5y ago
  I never understand why people criticize using interfaces for most functionality. How else do you write effective unit tests? Interfaces with dependency injection make unit tests far easier to write. Smartly designed interfaces also make code easier to read and understand. Lastly, adding new interfaces is trivially easy, so why not?
  • Jensson 5y ago
    Yeah. Similarly, walls of text are bad. I use line breaks after every sentence to fix that. I see no reason why not. This way you can clearly tell apart sentences. Adding new lines doesn't require any more time than just writing a space. So why not just do it always? By following this rule I never wrote a wall of text again!
  • wheelerof4te 5y ago
    What is an Interface? What is a dependency injection? How can you explain them to someone who never needed to use them? Do I need unit tests when my code is dead simple, or written in Rust? Etc...
• hiram112 5y ago
  I'm seeing this a lot with several of the "devops" or cloud architects (or whatever the preferred title is these days for the guys who manage the apps running on the servers) that we've hired in the last year. I've noticed two distinct types: * The "AWS cert" admins who have a dozen different cloud certs, but little practical experience. Every problem is to be solved using some over-priced, over-engineered conglomeration of cloud service. It doesn't matter if it's just an internal app to be temporarily used by a few dozen users for 6 months, they immediately begin following some "best practice" guide by setting up some complex multi-region, multi-subnet, read-replicated, load-balanced, auto-scaled, content-delivery-networked, containerized, Cloud-Formated, hourly-snapshotted, hot-swappable, OAuth secured and social media-login-enabled, automated environment that would be appropriate for only some retail giant's Black Friday operations, not a single CRUD app, to be used only temporarily by 10 users in HR dept. * The "automation expert" who takes the requirements to set up and maintain a few environments (e.g. dev, test, and production) that might need to be re-created only a few hours, 1-2x per year, and instead spends weeks crafting some monstrosity in Ansible or Cloud Formation or Terraform, complete with all sorts of required infrastructure servers that themselves bigger and more complex than the actual working environment itself. And what's worse is that none of these frameworks like Cloud Formation are ever 100% anyway, so you can't just "push a button" and create a new environment. Instead, there are a dozen holes in the different tools that need to be manually plugged when run, so it's not like a developer or junior devops person who doesn't know the environment inside and out or understand the undocumented quirks could use it themselves anyway, if and when the original guy leaves the company.
• antoniuschan99 5y ago
  I found this to be the most important line in the article (author was referring to when Hiring Senior Devs, but it applies to everyone in the firm): Stick with those who put the user and simplicity ahead of simply technology solutions. Users are #1. It’s a constant battle between diving into the code/technology and reaching out to users and customers.
• d0m3 5y ago
  The concept of overengineering is good on paper, but in practice it's being overused and not understood precisely. I see devs freaking out when you use the word abstraction now... Suggest extracting business logic from a React component and you don't know anymore if someone is gonna raise the "overengineering" flag. If you start a new project, do you not use any library or framework at the beginning? Obviously you take decisions according to how much the project/feature is expected to scale. If your estimation was too low, you might cripple your development at some point, and need a rewrite or at least some significant refactoring. If it was too high, I guess you overengineered. What matters is asking yourself the question, and of course as engineers we like to challenge ourselves into building the best possible solution, but we equally need to consider how likely it is that such a solution is never needed.
• awillen 5y ago
  I own an ecommerce company, and I'm using a huge (>$100M funding raised) company that's supposed to be a tech-enabled 3PL. It was a terrible choice, and their idiotically overengineered solutions are a big reason why. The most egregious one is that instead of weighing packages to find out their weight, they have an algorithm that estimates the weight. My packages are designed to come in at just under a pound (it's a big cutoff for shipping prices). Needless to say, slight problems with their algorithm can and do lead them to overbill me. A ton of work when they could just use a scale to do the job much, much better (and I'm sure save time overall with all the refunds of overbilling factored in).
• ronbarr 5y ago
  Duh
• keyle 5y ago
  What is overengineering? Code or design that solves problems you don’t have. That's beautifully put.
• drewcoo 5y ago
  > overengineering has killed more products than the absence of good development practices But as the author defines overengineering, it is clearly not a good development practice. Further, there is no evidence provided to support the claim. Maybe a truer lesson is that the authors work suffers under too much scrutiny. We're all better off imagining a similar thesis and then imagining it is actually true.