15 ms·

Why software ends up complex

188 points by kiyanwang 6y ago

underdeserver 6y ago
This is what project maintainers or code owners are for. It's the maintainer's responsibility to reduce complexity, and because it's hard to quantify the maintenance cost of additional features, the maintainer should have the final say.
- TobiasA 6y ago
  I have never heard of code ownership being a positive thing. Isn't this more an issue of encouraging internal knowledge sharing and confidence in refactoring?
- deleted 6y ago
  [deleted]
justapassenger 6y ago
Biggest underlying reason for why software ends up complex, is because real world domain in which software operates, is also complex.
- pontus 6y ago
  That together with the fact that nobody gets promoted and nobody collects revenue by simplifying or refactoring code.
  - arbaal 6y ago
    Funny that I'm nobody again. I work for a big german retailer. A big part of my work includes simplifying and refactoring code, so that we have lower maintenance cost and can move faster as a organisation unit. This was also a big factor in my last promotion. We also increased profit by lowering our runtime cost by some of those optimizations.
    - senko 6y ago
      If I were a betting man, I would bet that the main reason why you're allowed to do that is because you lowered runtime costs by some of those optimizations, not because of the cleaner code you can move faster.
      - Aeolun 6y ago
        You can trade on previous achievements to spend time improving your codebase (unless you have a micromanager that tracks every hour).
      - ThePadawan 6y ago
        ? not parent, but I am really surprised by that mindset. A large part of my career has been telling my boss "we should invest 5h of cleanup time here to make sure that in the future we won't have those 5h worth of obstacles here before building a new feature on top of it". In industries where requirements change on a weekly or monthly basis, agility is worth a lot.
        senko 6y ago
        I completely agree. My bet would be the bosses usually don't. So in a lot of places you need another "excuse" (or as another poster mentioned, spend the time saved to invest more into maintainability but cover it up).
  - hugey010 6y ago
    People do get promoted and paid for simplifying and refactoring code. Heck, I got promoted and am currently being paid to do just that. It often involves adding value in other ways simultaneously, but what you said is just false.
    - pontus 6y ago
      I guess it's dangerous to use words like "nobody" or "everyone" or "always" or "never" on the overly pedantic Hacker News. I would hope that the sentiment of my point came across either way. If not, let me clarify: it's generally preferred to spend engineering effort on building new features and optimizing code to improve performance etc. over e.g. rewriting code from scratch to simplify it's maintenance. I would venture to guess that an overwhelming amount of engineering effort is spent on effectively adding complexity to code.
      - nthj 6y ago
        Whoever is able to demonstrate the value of a sequence of work to the business will get their work prioritized. Product’s job is to shape and predict the value of adding new features. Customer complaints about slow reports help Customer Support justify performance optimizations. If engineers want the business to pay them for simplifying maintenance, they can identify a collection of support tickets that have a compelling root cause, and propose refactoring that system to eliminate the bugs and maintenance issues. What I usually see is us engineers calling it “tech debt” and complaining we never take time for it, which is about as effective as the bookkeeper complaining about the paperwork that debt payments create for him. Businesses love debt. It’s a useful financial tool. The business doesn’t get the issue.
  - ratww 6y ago
    Not directly. But simplifying and refactoring can help you understand the code better than doing routine maintenance. This helps you solve bugs and write new features faster and with more stability, and also give better input during meetings. So, indirectly, yes: you can get promoted and collect revenue by simplifying and refactoring.
- ratww 6y ago
  I don't know if I agree with that. Most of the complexity and bugs I see in software are not because of the problem domain, but rather because of over-abstraction, under-abstraction and abstraction leaks, and also because of limitations and complexities introduced by the programming model or environment. (unless you consider that "supporting five operating systems and the language must be X" is part of "essential complexity") Of course, the more complex your domain is, the bigger is the program. But the non-essential complexity that exists due to the bureaucracy of languages/libraries/frameworks is a much bigger factor in adding complexity, bug and lines of code. Some examples: - Manual allocation and deallocation of memory is a good example of something that we might think as essential, since it's intertwined with our domain code, but turns out to be unnecessary (even though the replacement has downsides). The billion-dollar problem (nulls) is another one. - Supporting multiple environments/browsers/platforms. Competition is good, but the cost is steep for third-party developers. Using multiplatform frameworks partially solves but also has drawbacks: performance, limitations, bugs, leaky abstractions. If you need to be closer to metal, then different OSs have different threading models or API styles. Sometimes they don't even expose the main loop to you. You need to work around those limitations. - In most environments we still don't have a nice way of handling async operations without leaking the abstraction. The current solution is adding "isLoading" data everywhere (or creating some abstraction around both buttons and the fetching mechanism). Concurrent Mode in React is probably the best thing we have so far. - Most modern Javascript toolchains need multiple parsing steps: in the transpiler (to convert to ES5), in the bundler (to detect dependencies), in the linter, in the prettifier, and in tests. Compatibility between them is not guaranteed, and you might get conflicts which have to be solved by finding a middle ground, and that sometimes take more time than writing features. - Dogmatism is another issue. I remember in one workplace years ago there was a "ORM only" rule and most of us would work the SQL and then convert to Rails ActiveRecord (or worse: Arel). In the end it was a complete waste of time and the results were impossible to maintain. - I also think that the old Peter Norvig quote that "design patterns are missing language features" still stands. Go has proven that it's possible to have "dumb, simple code", but in other languages our best practices involve adding non-essential complexity to products. The only exception to that in my experience is SQL: if a query is too big is not due to some bureaucracy of the language, but rather due to the complexity of the domain itself.
LC_ALL 6y ago
Building features on top of other features is often zero cost. Code becomes a many layered cake consumed by the end user. In the web development stack the simplest of features like text on a screen is the achievement of decades of technological progress. Text may be localized, shaped with HarfBuzz, run through libicu's BIDI algorithm, encoded with a nontrivial encoding, wrapped in markup language, nested inside of multiple layers of network headers and corresponding metadata, sent over the wire as a series of 0s and 1s, and then painstakingly unpacked in reverse order. This is clearly complicated and clearly works. Many different actors operating quasi-independently. You can imagine the difficulty when one actor in a time crunch tries to design a similarly complicated cake stitched together with parts homemade, parts open sourced and parts paid for.
nchelluri 6y ago
I agree wholeheartedly on the Brooks' "conceptual integrity" thing. I really enjoyed his book The Design of Design. It takes effort to maintain cohesion and sound architecture, but it pays off for future development.
dstick 6y ago
Is this missing half the article? The page just stops on mobile and there’s no footer, nothing. I count 4 paragraphs in total.
- nchelluri 6y ago
  I believe that you're seeing the whole article; it is short.
deleted 6y ago
[deleted]
k__ 6y ago
I had the impression the main problem was missing cognitive flexibility of developers. Some people just do things like they always have. They don't bother to understand what people before did and then bend everything to their will.
- throwaway201103 6y ago
  Another twist on this is that some people are passive-aggresively lazy. They may understand that what they are being asked to do has longer term negative consequences, but they don't want to invest the effort in trying to change any minds. They just do what they are asked, and when things get complicated they can say "I did what you asked for; you didn't ask me to simplify anything" This is seen more in rigidly hierarchical organizations where debate and ideas from the lower ranks tend to be quashed. Think of the boss who says some variant of "you're paid to do what I ask, not to talk about it"
- SulfurHexaFluri 6y ago
  This downplays the difficulty of "bothering" to understand things. Most developers work on applications of insane complexity that they couldn't ever hope to understand. Almost every change is done in some amount of ignorance of the surroundings.
Blikkentrekker 6y ago
I'd say that the only reason that software seems too complex, rather than as complex as it needs to be, is because every programmer thinks he can rewrite it in a simpler way, but when he's done, it's as complex that which he has rewritten. I've seen it happen so many times, and I've done it. It's the very same principle that leads to almost every construction project running behind schedule — a man simply underestimates the complexity of nigh every task he endeavors to complete.
- nchelluri 6y ago
  I see your points, and I see the merit of "rewrite syndrome", and lean strongly towards automated-test backed refactoring, and all in all I disagree with your thesis. Sometimes, software patches and new features get tacked on and tacked on and the system loses all semblance of cohesion or integrity. Thinking of the system as a whole, iterating with the confidence brought by tests of some sort, one can begin to detangle all the unncessary intermixing and duplicate work and begin to make the system sensible.
- Aeolun 6y ago
  Depends. If you can re-evaluate the requirements when you start your rewrite, you can likely consolidate a lot of the features. If the requirements stay exactly the same, then yeah, there’s no point.
  - Blikkentrekker 6y ago
    Well, when Wayland started they went in on the assumption that they could cast away quite a bit that “no one was using” or that “wasn't necessary”. And then they had to include more of what they cast away because they underestimated the number of consumers of things they personally weren't using. libinput originally actually did not have a way to disable and configure pointer acceleration I believe because the developed thought there was no reason to ever turn it off. He was not a gamer and was largely ignorant of how essential being able to disable it is for the level of accuracy required for video games.
    - 6y ago
AnonC 6y ago
> This means that supporters can always point to concrete benefits to their specific use cases, while detractors claim far more abstract drawbacks. I don’t agree with this. If you have a salesperson/accountant who grew into being CEO and is just a bean counter who doesn’t (want to) understand the real costs of training new developers on increased complexity in a system or being able to maintain a complex system over decades, this could possibly be true. But in that case, your engineering manager isn’t a real engineering manager either, and is just another bean counter in disguise. I get that senior management may not always (want to) understand the nuances in building, maintaining and supporting a complex system, but it’s not abstract. It costs real money that all these types can feel the pinch of. The real reason why software is complex or can get complex is because the underlying domain and its requirements and constraints are complex, combined with layers of complexity added on the technical side to enable certain things (perhaps easy configurability, scalability, reliability, etc.). There are self-inflicted wounds too, where complexity is prematurely added. But that’s not the full story in all cases.
tonetheman 6y ago
Software (like the universe) is entropy over time.
- lhpz 6y ago
  Let me expand on this thermodynamical metaphor. Open systems that may exchange matter and/or energy with their environment, can maintain or even decrease their own entropy. This is how living organisms exist, without violating the second law of thermodynamics. If you see a software system as a living organism, then the natural entropy increase (new feature requests?) may be contained if enough energy (developer time?) is invested. Of course this kind of investment may contravene some other commercial software law, like being profitable.
khaledh 6y ago
Complexity is a fact of the world we live in. The human body is extremely complex, but all organs and systems in the human body work in harmony and with astounding efficiency. We need to stop fighting complexity in software and embrace it instead. The goal shouldn't be to pursue simplicity at the cost of ignoring complexity of the domain being modelled, but to acknowledge that we're modelling a complex system, and design it such that subsystems work together in harmony. This article in particular touches on an important point which resonates with me. It's what I call "displacement oriented design". I view a code base as a puzzle that grows in size over time; you can't grow a puzzle just by adding to it and leaving all other pieces intact. As you expand the size of the puzzle you need "displace" some of the existing pieces, reshape them, and fit them with the new pieces.
- nchelluri 6y ago
  I agree with some of your points and disagree with others. The human body is exceptionally awesome, but also so very fragile. I think we can do better with well-thought-out design, and a part of that that I embrace is the value of simplicity. My thoughts can be best said with a quote (especially the part about a complexity budget): > I used to tolerate and expect complexity. Working on Go the past 10 years has changed my perspective, though. I now value simplicity above almost all else and tolerate complexity only when it's well isolated, well documented, well tested, and necessary to make things simpler overall at other layers for most people. For example, the Go runtime is relatively complex internally but it permits simple APIs and programming models for users who then don't need to worry about memory management, thread management, blocking, the color of their functions, etc. A small number of people need to understand the runtime's complexity, but millions of people can read & write simple Go code as a result. More importantly, Go users then have that much more complexity budget to work with to build their actual application. I would've never built Perkeep had I needed to fight both its internal complexity and the complexity imposed on me by other contender languages/environments at the time. > > All that is to say, simplicity is not only refreshing, but it also enables. Go made me feel productive in a way I hadn't felt in many years where everything just felt like it was getting more complex. Ever since finding Go, I've been regularly hunting for other technologies that provide simplicity as a feature. https://bradfitz.com/2020/01/30/joining-tailscale https://bradfitz.com/2020/01/30/joining-tailscale
jameskilton 6y ago
I feel that a lot of people misunderstand "complexity" vs "complicated". There's nothing wrong with complex. It's the nature of life that things are complex. Complicated though is almost always a negative. Complex code is fine, it's probably solving a real problem. Complicated code is not, it's just hard to work with.
- breck 6y ago
  Rich Hickey had a great talk on this that introduced me to the word "complecting" (https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...) Relatedly, I have a simple new maths for counting complexity, so that you can compare two "complex" solutions and pick the less "complicated" one: https://github.com/treenotation/research/blob/master/papers/paper3/countingComplexity.pdf https://github.com/treenotation/research/blob/master/papers/... SVG form: https://treenotation.org/demos/complexityCanBeCounted.svg https://treenotation.org/demos/complexityCanBeCounted.svg
- recursivedoubts 6y ago
  I agree with your comment. However, a good tool for controlling complexity is deciding what your system is going to do. As I said in a sibling comment, consider method overloading in java: this is a real world feature, not uncommon in other languages. There are arguments for and against it (I am against it.) The implementation of it may be amazing code, but none the less it makes the java compiler and runtime far more complicated that it would be if the feature were omitted. So, again, I agree with you, but I also agree with the articles point that choosing features carefully is an important tool in controlling complexity.
  - deleted 6y ago
    [deleted]
- ChrisMarshallNY 6y ago
  My experience, is that "complicated" vs. "complex," as you define, changes, depending on who is looking at the code. If someone has a philosophical aversion to something like abstraction, then they will label it "complicated," but I use abstraction, all the time, to insert low-cost pivot points in a design. I just did it this morning, as I was developing a design to aggregate search results from multiple servers. My abstraction will afford myself, or other future developers, the ability to add more data sources in the future, in a low-risk fashion. I also design frameworks in "layers," that often implement "philosophical realms," as opposed to practical ones. Think OSI layers. That can mean that adding a new command to the REST API, for example, may require that I implement the actual leverage in a very low-level layer, then add support in subsequent layers to pass the command through. That can add complexity, and quality problems. When I do something like that, I need to test carefully. The reason that I do that, is so, at an indeterminate point in the future, I can "swap out" entire layers, and replace them with newer tech. If I don't think that will be important, then I may want to rethink my design. That is the philosophy behind the OSI layers. They allow drastically different (and interchangeable) implementation at each layer, with clear interface points, above and below.
senko 6y ago
In this regard, software is quite similar to law - more so than to other STEM disciplines like mechanical or civil engineering or math. A body of law is adapted and extended through many years, by various groups having different priorities, and rarely does someone dive in and refactor it to be simpler (while achieving the same "business goals"). In software, at least we have an option of creating automated tests to verify correctness or internal consistency, and using stricter languages to avoid ambiguities. In light of this, we (as an industry) are actually doing quite good!
- machiaweliczny 6y ago
  Yeah, I would argue that essential work is usually similar to law (you just encode the rules to meet some objective). There's lot's of added complexity though in practice (due to distributed nature of systems, limited resources or just compatibility)
mirekrusin 6y ago
There are complex things and there are complicated things. You want to make complex things easier to understand by refining abstractions and eliminate complicated things alltogether. Constant refactoring and tabu word - intelligence - is the key here. It doesn't mean you have to reshuffle everything every week. It means spending maybe 5% - 20% of time on small things to improve code based on current knowledge. This has to be pushed by developers because semi-agile methodologies most companies use have built-in forces to move those tasks out of sprints.
recursivedoubts 6y ago
This is one vector for complexity, to be sure. Saying "no" to a feature that is unnecessary, foists a lot of complexity on a system, or has a low power to weight ratio is one of the best skills a senior developer can develop. One of my favorite real world examples is method overloading in Java. It's not a particularly useful feature (especially given alternative, less complicated features like default parameter values), interacts poorly with other language features (e.g. varargs) and ends up making all sorts of things far more complex than necessary: bytecode method invocation now needs to encode the entire type signature of a method, method resolution during compilation requires complex scoring, etc. The JVM language I worked on probably had about 10% of its total complexity dedicated to dealing with this "feature" of dubious value to end users. Another vector, more dangerous for senior developers, is thinking that abstraction will necessarily work when dealing with complexity. I have seen OSGi projects achieve negative complexity savings, while chewing up decades of senior man-years, for example.
- taeric 6y ago
  Your example is funny. Default parameters are far more complicated than method overloading. That said, I fully agree on the OSGi point. Makes me worried about a lot of the new features I hear are on the way. :(
  - recursivedoubts 6y ago
    > Default parameters are far more complicated than method overloading. I've implemented both, I disagree. Unless you are talking about default arguments in the presence of method overloading, which is insane, and which I have also implemented.
    - taeric 6y ago
      I'm taking about in use, mainly. Python, as an example, really screws it up by not giving you a way to know if you have the default or a value that equals it. Then, the more default values you pile in, the more this fun facet piles up. So, if you skip on that part of it, and you can while still getting far, it is easier. With it, though, is stupid complicated for little benefit.
wslh 6y ago
I don't think most software development should end up complex since most software development in the world is redundant. The problem is that we don't have the right high level abstractions, low level robustness, and groups focusing on this problem to build software faster. It is a matter of time, there will be a silver bullet for most software needs. There is also some conflict of interest: imagine if Microsoft gives you frameworks to build/integrate large software projects with a few "parameters"? There will be fewer developers and customers for them.
- Aeolun 6y ago
  Making software that does everything by tweaking ‘a few parameters’ is orders of magnitude more complex than the software that would do it without. Since I’m now generally part of several year software projects, we should be fine for the next few centuries or so.
- codeulike 6y ago
  Its interesting that you use the term Silver Bullet. I assume you are aware of the famous essay 'No Silver Bullet' https://en.wikipedia.org/wiki/No_Silver_Bullet https://en.wikipedia.org/wiki/No_Silver_Bullet
  - wslh 6y ago
    Yes, it was intended. I think with enough redundancy there are silver bullets in certain areas.
dustingetz 6y ago
1) You can't see software, which combines with 2) a company with bad sales but great engineering is a dead company, which results in 3) decisions trickle down from sales but salesmen are blind to the software. Proposed Solution: Programmers should learn sales. (So it's diversity at the root of all problems !)
- breck 6y ago
  Proposed Solution: make it so that you CAN "see software". this is what I work on, but also things like Sublime's Minimap are huge trends in this direction.
  - dustingetz 6y ago
    could work but we’ll need to condense 50k LOC crud spa into a flowchart without loss of essence. (that’s what i work on). what is your angle? i checked out treenotation.org , is there more? flowchart is graph
    - breck 6y ago
      It’s doesn’t work with current languages. The shape of the AST needs to be the same as the shape of the source. Once you’ve done that though the program and visualization are isomorphic. Check out the toHTMLcube demo in the Sandbox, the language designer demo, or: https://v20.ohayo.computer/?filename=ohayo.ohayo-source-code-vis.ohayo&nodeBreakSymbol=%7E&edgeSymbol=_&data=doc.title_Source_code_visualization_of_ohayo.ohayo%7Edata.inline%7E_parser_text%7E_treeLanguage_ohayo%7E_text.lineCount%7E__show.median_lines_Total_lines%7E_text.wordCount%7E__show.sum_count_Total_words%7E_treenotation.3d%7E_treenotation.outline%7E_treenotation.wordTypes%7E__html.printAs_pre%7E___text.wordCount%7E____tables.basic%7E____text.wordcloud%7E_content%7E__web.get_ohayo%252Fpackages%252Fsamples%252Fwelcome.md%7E___parser_text%7E___hidden%7E___markdown.toHtml%7E__templates.list%7E__challenge.list https://v20.ohayo.computer/?filename=ohayo.ohayo-source-code... That’s all older stuff, but hints at some of the ideas.
- reidjs 6y ago
  As a programmer currently learning sales by necessity, I agree wholeheartedly.
taeric 6y ago
My favorite point on how software end up complex, is our industry somehow thinks we are unique on this. Everything winds up complex. Just look at how many ingredients go into simple store bought cookies. Look into the entire supply chain around your flour, that lets your homemade cookies be simple.
- TheOtherHobbes 6y ago
  Unless you count obesity and diabetes, the public is not noticeably inconvenienced by the cookie supply chain, because it pretty much just works. So it's hard to find examples of Cookie Technology Failure Modes with serious consequences. There's a huge difference between "hard to use" and "prone to failure." Software is often hard to use, but it's also extremely prone to not working. This is not true of airliners [1], cars, large buildings, large ships, and commodity rocket launch platforms - as well as global supply chains of all kinds. [1] Unless there's software involved.
  - taeric 6y ago
    This is beside my point. I only picked cookies because I'm making some. :) Look into the supply chain for lumber to your house. Concrete. Electricity. Literally anything. Life is complicated. Software just fools us by letting us more readily have a blank slate sometimes. Though, even then, the complexity of the tool chain to support such easy "hello world" programs is insane. If you want to know frustration, try using any of the simple tools you can buy at a dollar store for a time. Simple can openers that will fall apart in no time. Tableware that will bend and likely go unusable in months.
  - nradov 6y ago
    When the pandemic lockdown started some of the supply chain abstractions leaked, and the US public was noticeably inconvenienced by lack of toilet paper, cleaning supplies, and (in some cases) meat. It turns out the supply chains are actually rather fragile with many failure points.
tacitusarc 6y ago
Within a problem space, there are two kinds of complexity: inherent complexity, and accidental complexity. This article is about accidental complexity. There is, as far as I can tell, and enormous amount of accidental complexity in software. Far more than there is inherent complexity. From my personal experience, this largely arises when no time has been taken to fully understand the problem space, and the first potential solution is the one used. In that case, the solution will be discovered as partially deficient in some manner, and more code will simply be tacked on to address the newfound shortcomings. I'm not referring here to later expansion of the feature set or addressing corner cases, either. I'm referring to code that was not constructed to appropriately model the desired behavior and thus instances of branching logic must be embedded within the system all over the place, or perhaps some class hierarchy is injected, and reflection is used in a attempt to make make the poor design decisions function. I don't think adding features makes software more complex, unless those features are somehow non-systemic; that is, there is no way to add them into the existing representation of available behaviors. Perhaps an example would be a set of workflows a user can navigate, and adding a new workflow simply entails the construction of that workflow and making it available via the addition to some list. That would be a systemic feature. On the other hand if the entirety of the behaviors embedded within the workflow were instead presented as commands or buttons or various options that needed to be scattered throughout the application, that would be a non-systemic addition, and introduce accidental complexity.
- pydry 6y ago
  One things I've noticed about building software is that the most appropriate contours of the problem space often only become clear with hindsight. Even if you start off with the best intentions about not putting in too many features it won't always help. This is why the second mover can also have an advantage in some areas. If they recognize the appropriate contours they can avoid the crufty features and more directly and effectively tackle the main problem.
- travmatt 6y ago
  For those interested in accidental vs inherent complexity, “Out of the Tar Pit” is a very easy to read paper that explores those definitions: https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf https://github.com/papers-we-love/papers-we-love/blob/master...
dimmke 6y ago
Taking on the responsibility of pushing back hard on poorly conceived new features is one of the important hidden skills of being an effective software developer. Programmers who just do whatever they get told like marching ants end up shooting an organization in the foot long term. You have to develop an opinion of the thing you're building/maintaining and what it should be able to do and not do. You can't count on project managers to do that. The trick to doing this effectively is to find out the problem the feature is actually trying to solve and providing a better solution. Usually the request is from end users of software and they have identified the problem (we need to do xyz) and prescribed a solution (put the ability to do xyz in a modal on this page.) But if you can look to what other software has done, do a UX review and find a way to add a feature in that solves their problem in a way that makes sense in the larger context of the software, they won't have a problem with it since it solves their problem and the codebase will take less of a hit. Unfortunately, it's a lot easier to just add the modal without complaint.
- speedgoose 6y ago
  It's also faster to just add the modal. When you are asked to do xyz ASAP because it has been sold to a customer and should have been deployed a week ago, you don't feel the need to do a UX review.
  - pc86 6y ago
    Missing a deployment by a week (or more accurately, deploying on that long of a timeframe), and only doing UX reviews when you "feel the need" both speak to larger organizational problems that probably aren't going to get solved by having a senior dev push back on a poorly thought-out feature.
    - speedgoose 6y ago
      That's true.
- alkonaut 6y ago
  > Programmers who just do whatever they get told like marching ants end up shooting an organization in the foot long term. This. This is why you want senior devs too. You want people who can stand up to poorly conceived features. The most important job is to say “no”, or “how about X instead?”. I get furious when I see senior colleagues defend horrible design decisions with “it was what was specified”. Your job as a developer is to take the spec from the product owner and suggest one that actually fits the system in terms of maintainability, performance etc. Blindly implementing a specification is a horrible idea.
karmakaze 6y ago
The far greater source of complexity is the extra that has resulted from features being analyzed and implemented sequentially. Each one may have been economical in isolation, but as a whole is not. This is why a rewrite often seems so attractive, to analyze the entire known scope and use a smaller set of mechanisms to do the same often allowing for additional changes to fit more easily. Painting yourself in a corner is either short-sightedness (technical/vertical, voluntary/oblivious), a tech-debt choice, something that didn't turn out as well as expected despite efforts for technical reasons or unexpected changes. It also comes from trying too hard, generalizing too early. So there are many reasons, and pragmatically balancing them is something that is given far too little weight.
carapace 6y ago
One the one hand it's really hard to program well. I think anyone who can solve a sudoku puzzle can learn to program, but only people with freakish skill or determination can learn to program really well. On the other hand, those folks who can program really well are often what I call "complexity junkies": programming is their sudoku, it's fun and exciting. It helps that you can get paid well to do it. So you get things like Haskell and Rust.
hotcrossbunny 6y ago
In my humble opinion, a lot of projects go quietly bad when they experience some sort of new requirement that gets underestimated in terms of its architectural impact by project management. At such time senior Devs have either already moved on, or have their eye off the ball such that new features get incorporated without the necessary architectural support. These inflection points can themselves introduce complexity but often become the gateway for all sorts of subsequent small things that explode in size. In short, don't miss architecture moments
- thefourthchime 6y ago
  The other thing that happens, and really dooms a project is when the senior people leave. Eventually you end up with a team that doesn't really understand the code. This leads them to just tack on features while changing as little as possible. This will grow into something truly unmaintainable, virtually guaranteeing no competent work will be done on the project again.
goblin89 6y ago
Software can be viewed as a function. A function is designed to be called in a certain way for certain output. If the required use of the function changes, function’s signature would need to support more arguments, and its behavior will increase in complexity accordingly. To avoid that, it is important to understand that a changed use case calls for what is effectively another function (possibly more than one). It may not be immediately feasible to rewrite and switch all callers over due to limited resources and lack of control over aforementioned callers, but conceptually it is another beast now—and existing implementation should move in that direction or be put on a deprecation schedule, rather than keep widening input and output spaces in a futile attempt to try to be everything at once. Software is very similar in this regard, except it is much more tempting (and often significantly easier) to “append” features rather than rethink the fundamentals as time goes by and users with different needs get on board. This is why, I believe, rigorously assessing[0] the scope and the intended audience of a piece of software at early stages (and constantly reassessing them afterwards) can go a long way against unchecked rise in complexity over time. As can prudently abstracting architecture pieces away into separate self-containing focused pieces, which can be recombined in new ways when context inevitably calls for what essentially is a different piece of software—instead of having to rewrite everything or horseshoe new features to support use cases that were not originally envisioned (but have to be supported for business reasons). [0] By having frank discussions and asking “why” many, many times.
KingOfCoders 6y ago
Because there is no physical, 3dimensional limit.
airstrike 6y ago
Alex Gaynor! I've missed this guy. I remember him back from his work on Django. http://pyfound.blogspot.com/2019/02/the-steady-leader-of-python-community.html http://pyfound.blogspot.com/2019/02/the-steady-leader-of-pyt...
beaconstudios 6y ago
Here's my approach. I think many feature requests fall under the X/Y problem. - view a new feature request as a new user capability - extend the model that the software implements, to encompass that capability - regardless of how the feature was implemented in the requester's head. - extend the software to match the new model. This may require refactoring as the model may have had to undergo shifts to encompass the new capability For example: I have a car. I model the car as four wheels, an engine, a chassis, and a lever. The engine drives the wheels, the wheels support the chassis, the chassis contains the engine. A lever in the chassis sets the engine in motion. It's a simple model and is capable of 1. sitting still and 2. moving forwards and backwards. This is all the capabilities we've needed so far. A user requests a new feature where the wheels are instead mecanum wheels (https://en.wikipedia.org/wiki/Mecanum_wheel https://en.wikipedia.org/wiki/Mecanum_wheel). The default industry response is to either implement the change as-requested, or reject it. I propose that the correct move instead is to ask the user WHY they want mecanum wheels. They reveal that they want the car to move in 2 dimensions, rather than one. From that understanding you can extend the model of the car to encompass the feature - you may add the mecanum wheels and a mechanism to control them, you may add a steering wheel and rack-and-pinion, you may do something completely different - totally depending on how and why the user wants 2d movement (depending on further questioning, ie "5 whys"). But you are working to the capability, not the feature. By extending the model, you can then change the software to match this new model. I think as software engineers we have a tendency to forget the model and focus only on the code. A request for mecanum wheels becomes a question of how to change the software to encompass that feature. But we must always remember the existence of the model, and the user's relationship to it.
tedunangst 6y ago
I like this essay, but I've also had success with "reject all feature requests" and ended up with happy users, not no users.
frongpik 6y ago
A software project is a lot like a house of cards where players around the table (employees) take turns to insert one or a few cards (features) until the house collapses (reorg). At any time a player can leave the table and write his score on the whiteboard - the number of cards inserted. If the house of cards collapses, players who were still in the game lose their score. In addition to that, the owner of the game (VP) can stop it any time and his score will be the total number of cards on the table. The winning strategy is to insert as much bloat as possible and leave just before the house collapses.
jeff-davis 6y ago
What about other kinds of backpressure on complexity earlier in the cycle? For instance, when you are just scoping out a feature (before implementation), you could estimate the net lines that need to be added (for example). If it seems to high, people can push back before all the work has been done. And if the estimate starts to look wrong you can go back and re-scope the project and either simplify the feature or tell everyone it's going to be much more complex than planned. The key would be to recognize exploding complexity before all the work is done. Has anyone tried something like this? Would it work?
drudu 6y ago
Software ends up complex since it attempts to model the real world which is oh so complex
amw-zero 6y ago
While true, this is also incredibly reductive. All of the preaching about making software more simple leads to nothing. Software, at the current scale of things we want to build, is inherently and unavoidably complex.
lbblack 6y ago
I think too many HNers here have an incomplete picture of what makes something complex. There is genius in simplicity. Complexity can be best understood (for me) as simultaneous interactions of many simple things under the same roof, which we then consider to be in totality an object of reality. Those simple things can be different in an infinite variety of ways, in which complexity can be derived. Personally, if I've jumped into a codebase that is messy and cluttered - unorganized... It's immediately noticable that the original project developers had a shallow and or narrow strategy for how they wanted to design their system. Start with simple and useful mechanisms, which are the building blocks for whatever problem you are trying to solve. Complexity and abstraction can then be extrapolated from a simple yet brilliant foundation. I don't know how that isn't common sense.
aryehof 6y ago
I wonder whether software ends up being complex, because the building blocks used by most programmers are simply insufficient to model complex systems. Those building blocks are procedures, subroutines, control statements and modules/sub-programs. Just because these can (eventually) be used to model complexity in code that increasingly approaches a big ball of mud, doesn’t mean they are the ultimate abstractions. We need to invent and use better abstractions.
lincpa 6y ago
The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model: everything is a pipeline, the perfect way to achieve the simplicity and unity of the software ecosystem. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow An application case: Apple M1 chip adopts Warehouse/Workshop Model Warehouse: unified memory Workshop: CPU, GPU and other cores Product: information there's also a new unified memory architecture that lets the CPU, GPU, and other cores exchange information between one another, and with unified memory, the CPU and GPU can access memory simultaneously rather than copying data between one area and another. Accessing the same pool of memory without the need for copying speeds up information exchange for faster overall performance. reference: Developer Delves Into Reasons Why Apple's M1 Chip is So Fast https://www.macrumors.com/2020/11/30/m1-chip-speed-explanation-developer/ https://www.macrumors.com/2020/11/30/m1-chip-speed-explanati...
robochat 6y ago
In my experience, all of the complexity comes from feature requests following the initial development of some code. I develop code for our internal automated data analysis systems. My boss doesn't care how it works, it just has to do exactly what he wants. Even if it destroys my abstractions. Suddenly, you have lots of if/elif/else cases and a creeping suspicion that your logic now has holes in it.
taylodl 6y ago
In my experience software complexity arises from handling errors and handling edge cases. Adding additional features means there are more errors to handle and that many more edge cases to consider.
drbojingle 6y ago
Why does software end up complex? Growing requirements and poor decisions.

188 points by kiyanwang 6y ago

This is what project maintainers or code owners are for. It's the maintainer's responsibility to reduce complexity, and because it's hard to quantify the maintenance cost of additional features, the maintainer should have the final say.

TobiasA 6y ago
I have never heard of code ownership being a positive thing. Isn't this more an issue of encouraging internal knowledge sharing and confidence in refactoring?
deleted 6y ago
[deleted]

justapassenger 6y ago

Biggest underlying reason for why software ends up complex, is because real world domain in which software operates, is also complex.

pontus 6y ago
That together with the fact that nobody gets promoted and nobody collects revenue by simplifying or refactoring code.
- arbaal 6y ago
  Funny that I'm nobody again. I work for a big german retailer. A big part of my work includes simplifying and refactoring code, so that we have lower maintenance cost and can move faster as a organisation unit. This was also a big factor in my last promotion. We also increased profit by lowering our runtime cost by some of those optimizations.
  - senko 6y ago
    If I were a betting man, I would bet that the main reason why you're allowed to do that is because you lowered runtime costs by some of those optimizations, not because of the cleaner code you can move faster.
    - Aeolun 6y ago
      You can trade on previous achievements to spend time improving your codebase (unless you have a micromanager that tracks every hour).
    - ThePadawan 6y ago
      ? not parent, but I am really surprised by that mindset. A large part of my career has been telling my boss "we should invest 5h of cleanup time here to make sure that in the future we won't have those 5h worth of obstacles here before building a new feature on top of it". In industries where requirements change on a weekly or monthly basis, agility is worth a lot.
      - senko 6y ago
        I completely agree. My bet would be the bosses usually don't. So in a lot of places you need another "excuse" (or as another poster mentioned, spend the time saved to invest more into maintainability but cover it up).
- hugey010 6y ago
  People do get promoted and paid for simplifying and refactoring code. Heck, I got promoted and am currently being paid to do just that. It often involves adding value in other ways simultaneously, but what you said is just false.
  - pontus 6y ago
    I guess it's dangerous to use words like "nobody" or "everyone" or "always" or "never" on the overly pedantic Hacker News. I would hope that the sentiment of my point came across either way. If not, let me clarify: it's generally preferred to spend engineering effort on building new features and optimizing code to improve performance etc. over e.g. rewriting code from scratch to simplify it's maintenance. I would venture to guess that an overwhelming amount of engineering effort is spent on effectively adding complexity to code.
    - nthj 6y ago
      Whoever is able to demonstrate the value of a sequence of work to the business will get their work prioritized. Product’s job is to shape and predict the value of adding new features. Customer complaints about slow reports help Customer Support justify performance optimizations. If engineers want the business to pay them for simplifying maintenance, they can identify a collection of support tickets that have a compelling root cause, and propose refactoring that system to eliminate the bugs and maintenance issues. What I usually see is us engineers calling it “tech debt” and complaining we never take time for it, which is about as effective as the bookkeeper complaining about the paperwork that debt payments create for him. Businesses love debt. It’s a useful financial tool. The business doesn’t get the issue.
- ratww 6y ago
  Not directly. But simplifying and refactoring can help you understand the code better than doing routine maintenance. This helps you solve bugs and write new features faster and with more stability, and also give better input during meetings. So, indirectly, yes: you can get promoted and collect revenue by simplifying and refactoring.
ratww 6y ago
I don't know if I agree with that. Most of the complexity and bugs I see in software are not because of the problem domain, but rather because of over-abstraction, under-abstraction and abstraction leaks, and also because of limitations and complexities introduced by the programming model or environment. (unless you consider that "supporting five operating systems and the language must be X" is part of "essential complexity") Of course, the more complex your domain is, the bigger is the program. But the non-essential complexity that exists due to the bureaucracy of languages/libraries/frameworks is a much bigger factor in adding complexity, bug and lines of code. Some examples: - Manual allocation and deallocation of memory is a good example of something that we might think as essential, since it's intertwined with our domain code, but turns out to be unnecessary (even though the replacement has downsides). The billion-dollar problem (nulls) is another one. - Supporting multiple environments/browsers/platforms. Competition is good, but the cost is steep for third-party developers. Using multiplatform frameworks partially solves but also has drawbacks: performance, limitations, bugs, leaky abstractions. If you need to be closer to metal, then different OSs have different threading models or API styles. Sometimes they don't even expose the main loop to you. You need to work around those limitations. - In most environments we still don't have a nice way of handling async operations without leaking the abstraction. The current solution is adding "isLoading" data everywhere (or creating some abstraction around both buttons and the fetching mechanism). Concurrent Mode in React is probably the best thing we have so far. - Most modern Javascript toolchains need multiple parsing steps: in the transpiler (to convert to ES5), in the bundler (to detect dependencies), in the linter, in the prettifier, and in tests. Compatibility between them is not guaranteed, and you might get conflicts which have to be solved by finding a middle ground, and that sometimes take more time than writing features. - Dogmatism is another issue. I remember in one workplace years ago there was a "ORM only" rule and most of us would work the SQL and then convert to Rails ActiveRecord (or worse: Arel). In the end it was a complete waste of time and the results were impossible to maintain. - I also think that the old Peter Norvig quote that "design patterns are missing language features" still stands. Go has proven that it's possible to have "dumb, simple code", but in other languages our best practices involve adding non-essential complexity to products. The only exception to that in my experience is SQL: if a query is too big is not due to some bureaucracy of the language, but rather due to the complexity of the domain itself.

LC_ALL 6y ago

Building features on top of other features is often zero cost. Code becomes a many layered cake consumed by the end user. In the web development stack the simplest of features like text on a screen is the achievement of decades of technological progress. Text may be localized, shaped with HarfBuzz, run through libicu's BIDI algorithm, encoded with a nontrivial encoding, wrapped in markup language, nested inside of multiple layers of network headers and corresponding metadata, sent over the wire as a series of 0s and 1s, and then painstakingly unpacked in reverse order. This is clearly complicated and clearly works. Many different actors operating quasi-independently. You can imagine the difficulty when one actor in a time crunch tries to design a similarly complicated cake stitched together with parts homemade, parts open sourced and parts paid for.

nchelluri 6y ago

I agree wholeheartedly on the Brooks' "conceptual integrity" thing. I really enjoyed his book The Design of Design. It takes effort to maintain cohesion and sound architecture, but it pays off for future development.

dstick 6y ago

Is this missing half the article? The page just stops on mobile and there’s no footer, nothing. I count 4 paragraphs in total.

nchelluri 6y ago
I believe that you're seeing the whole article; it is short.

deleted 6y ago

[deleted]

k__ 6y ago

I had the impression the main problem was missing cognitive flexibility of developers. Some people just do things like they always have. They don't bother to understand what people before did and then bend everything to their will.

throwaway201103 6y ago
Another twist on this is that some people are passive-aggresively lazy. They may understand that what they are being asked to do has longer term negative consequences, but they don't want to invest the effort in trying to change any minds. They just do what they are asked, and when things get complicated they can say "I did what you asked for; you didn't ask me to simplify anything" This is seen more in rigidly hierarchical organizations where debate and ideas from the lower ranks tend to be quashed. Think of the boss who says some variant of "you're paid to do what I ask, not to talk about it"
SulfurHexaFluri 6y ago
This downplays the difficulty of "bothering" to understand things. Most developers work on applications of insane complexity that they couldn't ever hope to understand. Almost every change is done in some amount of ignorance of the surroundings.

Blikkentrekker 6y ago

I'd say that the only reason that software seems too complex, rather than as complex as it needs to be, is because every programmer thinks he can rewrite it in a simpler way, but when he's done, it's as complex that which he has rewritten. I've seen it happen so many times, and I've done it. It's the very same principle that leads to almost every construction project running behind schedule — a man simply underestimates the complexity of nigh every task he endeavors to complete.

nchelluri 6y ago
I see your points, and I see the merit of "rewrite syndrome", and lean strongly towards automated-test backed refactoring, and all in all I disagree with your thesis. Sometimes, software patches and new features get tacked on and tacked on and the system loses all semblance of cohesion or integrity. Thinking of the system as a whole, iterating with the confidence brought by tests of some sort, one can begin to detangle all the unncessary intermixing and duplicate work and begin to make the system sensible.
Aeolun 6y ago
Depends. If you can re-evaluate the requirements when you start your rewrite, you can likely consolidate a lot of the features. If the requirements stay exactly the same, then yeah, there’s no point.
- Blikkentrekker 6y ago
  Well, when Wayland started they went in on the assumption that they could cast away quite a bit that “no one was using” or that “wasn't necessary”. And then they had to include more of what they cast away because they underestimated the number of consumers of things they personally weren't using. libinput originally actually did not have a way to disable and configure pointer acceleration I believe because the developed thought there was no reason to ever turn it off. He was not a gamer and was largely ignorant of how essential being able to disable it is for the level of accuracy required for video games.
  - 6y ago

AnonC 6y ago

> This means that supporters can always point to concrete benefits to their specific use cases, while detractors claim far more abstract drawbacks. I don’t agree with this. If you have a salesperson/accountant who grew into being CEO and is just a bean counter who doesn’t (want to) understand the real costs of training new developers on increased complexity in a system or being able to maintain a complex system over decades, this could possibly be true. But in that case, your engineering manager isn’t a real engineering manager either, and is just another bean counter in disguise. I get that senior management may not always (want to) understand the nuances in building, maintaining and supporting a complex system, but it’s not abstract. It costs real money that all these types can feel the pinch of. The real reason why software is complex or can get complex is because the underlying domain and its requirements and constraints are complex, combined with layers of complexity added on the technical side to enable certain things (perhaps easy configurability, scalability, reliability, etc.). There are self-inflicted wounds too, where complexity is prematurely added. But that’s not the full story in all cases.

tonetheman 6y ago

Software (like the universe) is entropy over time.

lhpz 6y ago
Let me expand on this thermodynamical metaphor. Open systems that may exchange matter and/or energy with their environment, can maintain or even decrease their own entropy. This is how living organisms exist, without violating the second law of thermodynamics. If you see a software system as a living organism, then the natural entropy increase (new feature requests?) may be contained if enough energy (developer time?) is invested. Of course this kind of investment may contravene some other commercial software law, like being profitable.

khaledh 6y ago

Complexity is a fact of the world we live in. The human body is extremely complex, but all organs and systems in the human body work in harmony and with astounding efficiency. We need to stop fighting complexity in software and embrace it instead. The goal shouldn't be to pursue simplicity at the cost of ignoring complexity of the domain being modelled, but to acknowledge that we're modelling a complex system, and design it such that subsystems work together in harmony. This article in particular touches on an important point which resonates with me. It's what I call "displacement oriented design". I view a code base as a puzzle that grows in size over time; you can't grow a puzzle just by adding to it and leaving all other pieces intact. As you expand the size of the puzzle you need "displace" some of the existing pieces, reshape them, and fit them with the new pieces.

nchelluri 6y ago
I agree with some of your points and disagree with others. The human body is exceptionally awesome, but also so very fragile. I think we can do better with well-thought-out design, and a part of that that I embrace is the value of simplicity. My thoughts can be best said with a quote (especially the part about a complexity budget): > I used to tolerate and expect complexity. Working on Go the past 10 years has changed my perspective, though. I now value simplicity above almost all else and tolerate complexity only when it's well isolated, well documented, well tested, and necessary to make things simpler overall at other layers for most people. For example, the Go runtime is relatively complex internally but it permits simple APIs and programming models for users who then don't need to worry about memory management, thread management, blocking, the color of their functions, etc. A small number of people need to understand the runtime's complexity, but millions of people can read & write simple Go code as a result. More importantly, Go users then have that much more complexity budget to work with to build their actual application. I would've never built Perkeep had I needed to fight both its internal complexity and the complexity imposed on me by other contender languages/environments at the time. > > All that is to say, simplicity is not only refreshing, but it also enables. Go made me feel productive in a way I hadn't felt in many years where everything just felt like it was getting more complex. Ever since finding Go, I've been regularly hunting for other technologies that provide simplicity as a feature. https://bradfitz.com/2020/01/30/joining-tailscale https://bradfitz.com/2020/01/30/joining-tailscale

jameskilton 6y ago

I feel that a lot of people misunderstand "complexity" vs "complicated". There's nothing wrong with complex. It's the nature of life that things are complex. Complicated though is almost always a negative. Complex code is fine, it's probably solving a real problem. Complicated code is not, it's just hard to work with.

breck 6y ago
Rich Hickey had a great talk on this that introduced me to the word "complecting" (https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...) Relatedly, I have a simple new maths for counting complexity, so that you can compare two "complex" solutions and pick the less "complicated" one: https://github.com/treenotation/research/blob/master/papers/paper3/countingComplexity.pdf https://github.com/treenotation/research/blob/master/papers/... SVG form: https://treenotation.org/demos/complexityCanBeCounted.svg https://treenotation.org/demos/complexityCanBeCounted.svg
recursivedoubts 6y ago
I agree with your comment. However, a good tool for controlling complexity is deciding what your system is going to do. As I said in a sibling comment, consider method overloading in java: this is a real world feature, not uncommon in other languages. There are arguments for and against it (I am against it.) The implementation of it may be amazing code, but none the less it makes the java compiler and runtime far more complicated that it would be if the feature were omitted. So, again, I agree with you, but I also agree with the articles point that choosing features carefully is an important tool in controlling complexity.
- deleted 6y ago
  [deleted]
ChrisMarshallNY 6y ago
My experience, is that "complicated" vs. "complex," as you define, changes, depending on who is looking at the code. If someone has a philosophical aversion to something like abstraction, then they will label it "complicated," but I use abstraction, all the time, to insert low-cost pivot points in a design. I just did it this morning, as I was developing a design to aggregate search results from multiple servers. My abstraction will afford myself, or other future developers, the ability to add more data sources in the future, in a low-risk fashion. I also design frameworks in "layers," that often implement "philosophical realms," as opposed to practical ones. Think OSI layers. That can mean that adding a new command to the REST API, for example, may require that I implement the actual leverage in a very low-level layer, then add support in subsequent layers to pass the command through. That can add complexity, and quality problems. When I do something like that, I need to test carefully. The reason that I do that, is so, at an indeterminate point in the future, I can "swap out" entire layers, and replace them with newer tech. If I don't think that will be important, then I may want to rethink my design. That is the philosophy behind the OSI layers. They allow drastically different (and interchangeable) implementation at each layer, with clear interface points, above and below.

senko 6y ago

In this regard, software is quite similar to law - more so than to other STEM disciplines like mechanical or civil engineering or math. A body of law is adapted and extended through many years, by various groups having different priorities, and rarely does someone dive in and refactor it to be simpler (while achieving the same "business goals"). In software, at least we have an option of creating automated tests to verify correctness or internal consistency, and using stricter languages to avoid ambiguities. In light of this, we (as an industry) are actually doing quite good!

machiaweliczny 6y ago
Yeah, I would argue that essential work is usually similar to law (you just encode the rules to meet some objective). There's lot's of added complexity though in practice (due to distributed nature of systems, limited resources or just compatibility)

mirekrusin 6y ago

There are complex things and there are complicated things. You want to make complex things easier to understand by refining abstractions and eliminate complicated things alltogether. Constant refactoring and tabu word - intelligence - is the key here. It doesn't mean you have to reshuffle everything every week. It means spending maybe 5% - 20% of time on small things to improve code based on current knowledge. This has to be pushed by developers because semi-agile methodologies most companies use have built-in forces to move those tasks out of sprints.

recursivedoubts 6y ago

This is one vector for complexity, to be sure. Saying "no" to a feature that is unnecessary, foists a lot of complexity on a system, or has a low power to weight ratio is one of the best skills a senior developer can develop. One of my favorite real world examples is method overloading in Java. It's not a particularly useful feature (especially given alternative, less complicated features like default parameter values), interacts poorly with other language features (e.g. varargs) and ends up making all sorts of things far more complex than necessary: bytecode method invocation now needs to encode the entire type signature of a method, method resolution during compilation requires complex scoring, etc. The JVM language I worked on probably had about 10% of its total complexity dedicated to dealing with this "feature" of dubious value to end users. Another vector, more dangerous for senior developers, is thinking that abstraction will necessarily work when dealing with complexity. I have seen OSGi projects achieve negative complexity savings, while chewing up decades of senior man-years, for example.

taeric 6y ago
Your example is funny. Default parameters are far more complicated than method overloading. That said, I fully agree on the OSGi point. Makes me worried about a lot of the new features I hear are on the way. :(
- recursivedoubts 6y ago
  > Default parameters are far more complicated than method overloading. I've implemented both, I disagree. Unless you are talking about default arguments in the presence of method overloading, which is insane, and which I have also implemented.
  - taeric 6y ago
    I'm taking about in use, mainly. Python, as an example, really screws it up by not giving you a way to know if you have the default or a value that equals it. Then, the more default values you pile in, the more this fun facet piles up. So, if you skip on that part of it, and you can while still getting far, it is easier. With it, though, is stupid complicated for little benefit.

wslh 6y ago

I don't think most software development should end up complex since most software development in the world is redundant. The problem is that we don't have the right high level abstractions, low level robustness, and groups focusing on this problem to build software faster. It is a matter of time, there will be a silver bullet for most software needs. There is also some conflict of interest: imagine if Microsoft gives you frameworks to build/integrate large software projects with a few "parameters"? There will be fewer developers and customers for them.

Aeolun 6y ago
Making software that does everything by tweaking ‘a few parameters’ is orders of magnitude more complex than the software that would do it without. Since I’m now generally part of several year software projects, we should be fine for the next few centuries or so.
codeulike 6y ago
Its interesting that you use the term Silver Bullet. I assume you are aware of the famous essay 'No Silver Bullet' https://en.wikipedia.org/wiki/No_Silver_Bullet https://en.wikipedia.org/wiki/No_Silver_Bullet
- wslh 6y ago
  Yes, it was intended. I think with enough redundancy there are silver bullets in certain areas.

dustingetz 6y ago

1) You can't see software, which combines with 2) a company with bad sales but great engineering is a dead company, which results in 3) decisions trickle down from sales but salesmen are blind to the software. Proposed Solution: Programmers should learn sales. (So it's diversity at the root of all problems !)

breck 6y ago
Proposed Solution: make it so that you CAN "see software". this is what I work on, but also things like Sublime's Minimap are huge trends in this direction.
- dustingetz 6y ago
  could work but we’ll need to condense 50k LOC crud spa into a flowchart without loss of essence. (that’s what i work on). what is your angle? i checked out treenotation.org , is there more? flowchart is graph
  - breck 6y ago
    It’s doesn’t work with current languages. The shape of the AST needs to be the same as the shape of the source. Once you’ve done that though the program and visualization are isomorphic. Check out the toHTMLcube demo in the Sandbox, the language designer demo, or: https://v20.ohayo.computer/?filename=ohayo.ohayo-source-code-vis.ohayo&nodeBreakSymbol=%7E&edgeSymbol=_&data=doc.title_Source_code_visualization_of_ohayo.ohayo%7Edata.inline%7E_parser_text%7E_treeLanguage_ohayo%7E_text.lineCount%7E__show.median_lines_Total_lines%7E_text.wordCount%7E__show.sum_count_Total_words%7E_treenotation.3d%7E_treenotation.outline%7E_treenotation.wordTypes%7E__html.printAs_pre%7E___text.wordCount%7E____tables.basic%7E____text.wordcloud%7E_content%7E__web.get_ohayo%252Fpackages%252Fsamples%252Fwelcome.md%7E___parser_text%7E___hidden%7E___markdown.toHtml%7E__templates.list%7E__challenge.list https://v20.ohayo.computer/?filename=ohayo.ohayo-source-code... That’s all older stuff, but hints at some of the ideas.
reidjs 6y ago
As a programmer currently learning sales by necessity, I agree wholeheartedly.

taeric 6y ago

My favorite point on how software end up complex, is our industry somehow thinks we are unique on this. Everything winds up complex. Just look at how many ingredients go into simple store bought cookies. Look into the entire supply chain around your flour, that lets your homemade cookies be simple.

TheOtherHobbes 6y ago
Unless you count obesity and diabetes, the public is not noticeably inconvenienced by the cookie supply chain, because it pretty much just works. So it's hard to find examples of Cookie Technology Failure Modes with serious consequences. There's a huge difference between "hard to use" and "prone to failure." Software is often hard to use, but it's also extremely prone to not working. This is not true of airliners [1], cars, large buildings, large ships, and commodity rocket launch platforms - as well as global supply chains of all kinds. [1] Unless there's software involved.
- taeric 6y ago
  This is beside my point. I only picked cookies because I'm making some. :) Look into the supply chain for lumber to your house. Concrete. Electricity. Literally anything. Life is complicated. Software just fools us by letting us more readily have a blank slate sometimes. Though, even then, the complexity of the tool chain to support such easy "hello world" programs is insane. If you want to know frustration, try using any of the simple tools you can buy at a dollar store for a time. Simple can openers that will fall apart in no time. Tableware that will bend and likely go unusable in months.
- nradov 6y ago
  When the pandemic lockdown started some of the supply chain abstractions leaked, and the US public was noticeably inconvenienced by lack of toilet paper, cleaning supplies, and (in some cases) meat. It turns out the supply chains are actually rather fragile with many failure points.

tacitusarc 6y ago

Within a problem space, there are two kinds of complexity: inherent complexity, and accidental complexity. This article is about accidental complexity. There is, as far as I can tell, and enormous amount of accidental complexity in software. Far more than there is inherent complexity. From my personal experience, this largely arises when no time has been taken to fully understand the problem space, and the first potential solution is the one used. In that case, the solution will be discovered as partially deficient in some manner, and more code will simply be tacked on to address the newfound shortcomings. I'm not referring here to later expansion of the feature set or addressing corner cases, either. I'm referring to code that was not constructed to appropriately model the desired behavior and thus instances of branching logic must be embedded within the system all over the place, or perhaps some class hierarchy is injected, and reflection is used in a attempt to make make the poor design decisions function. I don't think adding features makes software more complex, unless those features are somehow non-systemic; that is, there is no way to add them into the existing representation of available behaviors. Perhaps an example would be a set of workflows a user can navigate, and adding a new workflow simply entails the construction of that workflow and making it available via the addition to some list. That would be a systemic feature. On the other hand if the entirety of the behaviors embedded within the workflow were instead presented as commands or buttons or various options that needed to be scattered throughout the application, that would be a non-systemic addition, and introduce accidental complexity.

pydry 6y ago
One things I've noticed about building software is that the most appropriate contours of the problem space often only become clear with hindsight. Even if you start off with the best intentions about not putting in too many features it won't always help. This is why the second mover can also have an advantage in some areas. If they recognize the appropriate contours they can avoid the crufty features and more directly and effectively tackle the main problem.
travmatt 6y ago
For those interested in accidental vs inherent complexity, “Out of the Tar Pit” is a very easy to read paper that explores those definitions: https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf https://github.com/papers-we-love/papers-we-love/blob/master...

dimmke 6y ago

Taking on the responsibility of pushing back hard on poorly conceived new features is one of the important hidden skills of being an effective software developer. Programmers who just do whatever they get told like marching ants end up shooting an organization in the foot long term. You have to develop an opinion of the thing you're building/maintaining and what it should be able to do and not do. You can't count on project managers to do that. The trick to doing this effectively is to find out the problem the feature is actually trying to solve and providing a better solution. Usually the request is from end users of software and they have identified the problem (we need to do xyz) and prescribed a solution (put the ability to do xyz in a modal on this page.) But if you can look to what other software has done, do a UX review and find a way to add a feature in that solves their problem in a way that makes sense in the larger context of the software, they won't have a problem with it since it solves their problem and the codebase will take less of a hit. Unfortunately, it's a lot easier to just add the modal without complaint.

speedgoose 6y ago
It's also faster to just add the modal. When you are asked to do xyz ASAP because it has been sold to a customer and should have been deployed a week ago, you don't feel the need to do a UX review.
- pc86 6y ago
  Missing a deployment by a week (or more accurately, deploying on that long of a timeframe), and only doing UX reviews when you "feel the need" both speak to larger organizational problems that probably aren't going to get solved by having a senior dev push back on a poorly thought-out feature.
  - speedgoose 6y ago
    That's true.
alkonaut 6y ago
> Programmers who just do whatever they get told like marching ants end up shooting an organization in the foot long term. This. This is why you want senior devs too. You want people who can stand up to poorly conceived features. The most important job is to say “no”, or “how about X instead?”. I get furious when I see senior colleagues defend horrible design decisions with “it was what was specified”. Your job as a developer is to take the spec from the product owner and suggest one that actually fits the system in terms of maintainability, performance etc. Blindly implementing a specification is a horrible idea.

karmakaze 6y ago

The far greater source of complexity is the extra that has resulted from features being analyzed and implemented sequentially. Each one may have been economical in isolation, but as a whole is not. This is why a rewrite often seems so attractive, to analyze the entire known scope and use a smaller set of mechanisms to do the same often allowing for additional changes to fit more easily. Painting yourself in a corner is either short-sightedness (technical/vertical, voluntary/oblivious), a tech-debt choice, something that didn't turn out as well as expected despite efforts for technical reasons or unexpected changes. It also comes from trying too hard, generalizing too early. So there are many reasons, and pragmatically balancing them is something that is given far too little weight.

carapace 6y ago

One the one hand it's really hard to program well. I think anyone who can solve a sudoku puzzle can learn to program, but only people with freakish skill or determination can learn to program really well. On the other hand, those folks who can program really well are often what I call "complexity junkies": programming is their sudoku, it's fun and exciting. It helps that you can get paid well to do it. So you get things like Haskell and Rust.

hotcrossbunny 6y ago

In my humble opinion, a lot of projects go quietly bad when they experience some sort of new requirement that gets underestimated in terms of its architectural impact by project management. At such time senior Devs have either already moved on, or have their eye off the ball such that new features get incorporated without the necessary architectural support. These inflection points can themselves introduce complexity but often become the gateway for all sorts of subsequent small things that explode in size. In short, don't miss architecture moments

thefourthchime 6y ago
The other thing that happens, and really dooms a project is when the senior people leave. Eventually you end up with a team that doesn't really understand the code. This leads them to just tack on features while changing as little as possible. This will grow into something truly unmaintainable, virtually guaranteeing no competent work will be done on the project again.

goblin89 6y ago

Software can be viewed as a function. A function is designed to be called in a certain way for certain output. If the required use of the function changes, function’s signature would need to support more arguments, and its behavior will increase in complexity accordingly. To avoid that, it is important to understand that a changed use case calls for what is effectively another function (possibly more than one). It may not be immediately feasible to rewrite and switch all callers over due to limited resources and lack of control over aforementioned callers, but conceptually it is another beast now—and existing implementation should move in that direction or be put on a deprecation schedule, rather than keep widening input and output spaces in a futile attempt to try to be everything at once. Software is very similar in this regard, except it is much more tempting (and often significantly easier) to “append” features rather than rethink the fundamentals as time goes by and users with different needs get on board. This is why, I believe, rigorously assessing[0] the scope and the intended audience of a piece of software at early stages (and constantly reassessing them afterwards) can go a long way against unchecked rise in complexity over time. As can prudently abstracting architecture pieces away into separate self-containing focused pieces, which can be recombined in new ways when context inevitably calls for what essentially is a different piece of software—instead of having to rewrite everything or horseshoe new features to support use cases that were not originally envisioned (but have to be supported for business reasons). [0] By having frank discussions and asking “why” many, many times.

KingOfCoders 6y ago

Because there is no physical, 3dimensional limit.

airstrike 6y ago

Alex Gaynor! I've missed this guy. I remember him back from his work on Django. http://pyfound.blogspot.com/2019/02/the-steady-leader-of-python-community.html http://pyfound.blogspot.com/2019/02/the-steady-leader-of-pyt...

beaconstudios 6y ago

Here's my approach. I think many feature requests fall under the X/Y problem. - view a new feature request as a new user capability - extend the model that the software implements, to encompass that capability - regardless of how the feature was implemented in the requester's head. - extend the software to match the new model. This may require refactoring as the model may have had to undergo shifts to encompass the new capability For example: I have a car. I model the car as four wheels, an engine, a chassis, and a lever. The engine drives the wheels, the wheels support the chassis, the chassis contains the engine. A lever in the chassis sets the engine in motion. It's a simple model and is capable of 1. sitting still and 2. moving forwards and backwards. This is all the capabilities we've needed so far. A user requests a new feature where the wheels are instead mecanum wheels (https://en.wikipedia.org/wiki/Mecanum_wheel https://en.wikipedia.org/wiki/Mecanum_wheel). The default industry response is to either implement the change as-requested, or reject it. I propose that the correct move instead is to ask the user WHY they want mecanum wheels. They reveal that they want the car to move in 2 dimensions, rather than one. From that understanding you can extend the model of the car to encompass the feature - you may add the mecanum wheels and a mechanism to control them, you may add a steering wheel and rack-and-pinion, you may do something completely different - totally depending on how and why the user wants 2d movement (depending on further questioning, ie "5 whys"). But you are working to the capability, not the feature. By extending the model, you can then change the software to match this new model. I think as software engineers we have a tendency to forget the model and focus only on the code. A request for mecanum wheels becomes a question of how to change the software to encompass that feature. But we must always remember the existence of the model, and the user's relationship to it.

tedunangst 6y ago

I like this essay, but I've also had success with "reject all feature requests" and ended up with happy users, not no users.

frongpik 6y ago

A software project is a lot like a house of cards where players around the table (employees) take turns to insert one or a few cards (features) until the house collapses (reorg). At any time a player can leave the table and write his score on the whiteboard - the number of cards inserted. If the house of cards collapses, players who were still in the game lose their score. In addition to that, the owner of the game (VP) can stop it any time and his score will be the total number of cards on the table. The winning strategy is to insert as much bloat as possible and leave just before the house collapses.

jeff-davis 6y ago

What about other kinds of backpressure on complexity earlier in the cycle? For instance, when you are just scoping out a feature (before implementation), you could estimate the net lines that need to be added (for example). If it seems to high, people can push back before all the work has been done. And if the estimate starts to look wrong you can go back and re-scope the project and either simplify the feature or tell everyone it's going to be much more complex than planned. The key would be to recognize exploding complexity before all the work is done. Has anyone tried something like this? Would it work?

drudu 6y ago

Software ends up complex since it attempts to model the real world which is oh so complex

amw-zero 6y ago

While true, this is also incredibly reductive. All of the preaching about making software more simple leads to nothing. Software, at the current scale of things we want to build, is inherently and unavoidably complex.

lbblack 6y ago

I think too many HNers here have an incomplete picture of what makes something complex. There is genius in simplicity. Complexity can be best understood (for me) as simultaneous interactions of many simple things under the same roof, which we then consider to be in totality an object of reality. Those simple things can be different in an infinite variety of ways, in which complexity can be derived. Personally, if I've jumped into a codebase that is messy and cluttered - unorganized... It's immediately noticable that the original project developers had a shallow and or narrow strategy for how they wanted to design their system. Start with simple and useful mechanisms, which are the building blocks for whatever problem you are trying to solve. Complexity and abstraction can then be extrapolated from a simple yet brilliant foundation. I don't know how that isn't common sense.

aryehof 6y ago

I wonder whether software ends up being complex, because the building blocks used by most programmers are simply insufficient to model complex systems. Those building blocks are procedures, subroutines, control statements and modules/sub-programs. Just because these can (eventually) be used to model complexity in code that increasingly approaches a big ball of mud, doesn’t mean they are the ultimate abstractions. We need to invent and use better abstractions.

lincpa 6y ago

The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Warehouse/Workshop Model: everything is a pipeline, the perfect way to achieve the simplicity and unity of the software ecosystem. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow An application case: Apple M1 chip adopts Warehouse/Workshop Model Warehouse: unified memory Workshop: CPU, GPU and other cores Product: information there's also a new unified memory architecture that lets the CPU, GPU, and other cores exchange information between one another, and with unified memory, the CPU and GPU can access memory simultaneously rather than copying data between one area and another. Accessing the same pool of memory without the need for copying speeds up information exchange for faster overall performance. reference: Developer Delves Into Reasons Why Apple's M1 Chip is So Fast https://www.macrumors.com/2020/11/30/m1-chip-speed-explanation-developer/ https://www.macrumors.com/2020/11/30/m1-chip-speed-explanati...

robochat 6y ago

In my experience, all of the complexity comes from feature requests following the initial development of some code. I develop code for our internal automated data analysis systems. My boss doesn't care how it works, it just has to do exactly what he wants. Even if it destroys my abstractions. Suddenly, you have lots of if/elif/else cases and a creeping suspicion that your logic now has holes in it.

taylodl 6y ago

In my experience software complexity arises from handling errors and handling edge cases. Adding additional features means there are more errors to handle and that many more edge cases to consider.

drbojingle 6y ago

Why does software end up complex? Growing requirements and poor decisions.