20 ms·

Simple Systems Have Less Downtime (2020)

303 points by gk1 5y ago

brobdingnagians 5y ago
So true. I am working with a system currently with tons of "moving parts", once it's up it's neat, but there's always at least 3 things broken every time I install it. The person who created it is the de facto expert, and everyone else just relies on them to fix it or extend the more tricky bits. It does leave one wonderful if the niftiness is worth the pain, or if there is a way of getting the good bits without the complexity.
AllegedAlec 5y ago
I disagree with the notion that a container ship is a simple system because 13 people can man it. It just means that the complexity is hidden in deeper layers.
- motoboi 5y ago
  Yes! In the future, a whole car factory will have only one employee. His job: feed the dog. The dog’s: keep humans away from the machines. Just as a ship with dozens of people got replaced by 13: the complexity was encapsulated in simpler interfaces. Diesel motors: just change this lever. Up is fast, down is slow. But it took a century to refine this absurdly complex device until this simple interface became possible. I think the point here is: don’t build your own diesel motors. It’s unlikely you’ll beat a good model and your interface to it will probably be worst. Remember: people take years learning from their own mistakes to hide the complexity under simpler interfaces layers. Unless your company IS building diesel motors.
  - AllegedAlec 5y ago
    So what happens to your simple encapsulated system when the diesel motor breaks down, or one of your hydraulic lines breaks? Does your simple encapsulated system them also say "we have no downtime, because we have a simple system"?
    - cl3misch 5y ago
      I'm pretty sure that is the exact point the parent post is trying to make!
    - motoboi 5y ago
      You leave port with 10 replacements for every component of the encapsulated device. Or just a full replacement for it. Hidden complexity in the case of the diesel motor was made possible also because of know failure modes (how many years until the low hanging fruits all got solved, how many decades until the not so easy got too?) and standardization of its components (how many centuries just to understand and standardize the universal joint https://en.wikipedia.org/wiki/Universal_joint https://en.wikipedia.org/wiki/Universal_joint?).
- naasking 5y ago
  But the question is, is the 13 manned ship still simpler than one that requires people doing all of those jobs? ie. is maintaining the machinery less complicated than maintaining the equivalent number of people that would be required to do the same jobs? Arguably yes.
  - AllegedAlec 5y ago
    I'm doubtful. You might be able to get to harbor more easily if something breaks down along the way, but actually getting the ship fixed afterwards might take longer, because the apparent simplicity is built using highly complex systems. You see the same thing in modern cars, which are often a real pain to fix, because it's all computers and complex parts, rather than one axle going from the steering wheel directly to the wheels.
    - naasking 5y ago
      > because the apparent simplicity is built using highly complex systems. More complex than humans, with wives, kids, need for downtime, hazard pay, insurance requirements, mental and physical limitations, and so on? A lot of the complexity of humans is invisible to us because we're just used to it.
jacquesm 5y ago
When applying this to software and you have a complex system there is a lot to learn from Joe Armstrong's excellent paper: https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf As well as from the way the Erlang language and runtime are organized.
threeseed 5y ago
People don't go out of their way to build complex systems. What happens is that it starts simple and becomes complex as more features are added often with limitations attached e.g. time, money etc. So yes this guy migrated from Marketo to Hubspot and it was simple. But the idea it will simply remain that way is laughable.
- stingraycharles 5y ago
  While what you’re saying is, of course, fair, the other side is also true: people often choose to complex a technology “because it’s future-proof!”, while they need just a simple system. There are tons of examples: CQRS over simple databases, Hadoop while a single server suffices, or even people choosing JIRA over a simple trello board. As such, the obvious answer is “it depends”, and making the right trade-off is rarely properly captured in a single rule of thumb such as “always choose simple”.
  - embik 5y ago
    > a single server suffices A genuine question that I wonder about: How does a single server protect you against someone in your DC pulling a plug by mistake or your cloud provider being unreliable? If it’s a database it seems like a huge potential for going out of business. I guess if you are running single-server databases you better have a good backup strategy anyway, but is that a risk people and businesses just accept?
    - VBprogrammer 5y ago
      I think what you are missing is that simple databases in a well run data-centre are surprisingly reliable. The tradeoffs involved in avoiding some of the simple "the cleaner unplugged the DB server" introduce complexities which may themselves cause outages which are much harder to solve.
    - citrin_ru 5y ago
      If you have a single server pulling of a plug (or even any HW failure which cannot be fixed quickly) is a very rare even (unless you use cheap desktop hardware instead of a decent server with ECC RAM, hot-swap HDD, redundant PSU e. t. c.). If instead of using this single server (with manual fail-over in case it would fail) you'll start to create complex HA cluster from multiple servers not having enough people to properly design, test and maintain it you can end up having a less reliable system.
mschuster91 5y ago
To add a point: "simple" can also mean "standardized". Self-hosted or cloud-hosted Kubernetes is a highly complex system. But using it to power your internal tooling (e.g. Jira, Confluence, GitLab, Jenkins, whatever) over classic, manually-installed-on-VMs deployment has the advantage that you can drop in anyone with experience in Kubernetes to manage your workloads.
- cryptica 5y ago
  Kubernetes is a complex solution but it solves a very complex problem. It could have been made slightly simpler (and slightly less flexible too) but it's a really good solution IMO. The structure is mostly straight forward and intuitive (at least to me as a developer).
- sdevonoes 5y ago
  I would go one level up and ask "do we actually need Jira, Confluence, GitLab, Jenkins, whatever on premise?". The question then is not "K8s or manual way", but "Do we actually need K8s?"
  - mschuster91 5y ago
    Many companies in the EU actually blanket ban any storage of their sensitive information in the cloud, especially the Atlassian cloud after the Australian espionage law change.
    - sdevonoes 5y ago
      That's a good point! Then we raise the question one level up: "does the current law regarding storage of sensitive information make sense?" It's a never ending story for those who like to delve in these sort of things.
      - mschuster91 5y ago
        For all what it's worth: yes. The larger the cloud, the larger the motivation for enemies to attack it. And in this case, the enemy is China (and, for European companies, also the US/FVEY alliance) - all of which have already been caught multiple times doing industrial espionage. Core company secrets belong on premises of the company owning the secret, not in a cloud where it is fair game for security services and criminals of all kinds and nations.
WolfOliver 5y ago
Would somebody agree that a MySQL/PostgreSQL cluster has more downtime than a replicated cluster? How often do high availability mechanism cause downtime itself?
Sebb767 5y ago
I think a better point would be "simple systems are quicker to fix" or, more concise, "avoid unnecessary complexity". In some way, complexity allows flaws to get in, but that does not mean it's unnecessary. Take, for example, website hosting. You can put your full page on a single server, but then you're prone to the machine going down, the disk having errors etc.. A more complex system, like Kubernetes, will be far harder to troubleshoot, but it will not go down just because one machine died. Or, as an engineering example, take planes: They're a lot more complex than ships. Fixing them when something is wrong can take long. But that complexity is needed so what when something fails, you get to the airport instead of going through a rapid unscheduled disassembly.
- trasz 5y ago
  Until recently even the huge airliners had physical strings going from the rudder all the way to the wing and tail control surfaces. Precisely because more complex systems were less reliable. Also, even with fly by wire there’s sometimes (or often, or always, no idea) a backup, eg a direct electric / hydraulic link.
  - Pokepokalypse 5y ago
    sounds like the 737MAX. . .
namelosw 5y ago
The Author generalized "fewer features lead to less downtime" to "simple systems have less downtime". A simple system such as a hand-written web server is very likely to crash. It's very likely cannot serve many users if we don't make it serve requests concurrently. What's worse, one tiny exception in one request would bring the whole system down. It stops serving any request to any user. A relatively complex system with a self-recovery mechanism added goes a long way compared to the former. The Erlang implementation is much more complex than "an Erlang without processes". Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes. The real world is chaotic. If a system needs to strive for less downtime, it needs to have some features that mimic biological creatures, instead of being idealistic simple.
- citrin_ru 5y ago
  > A simple system such as a hand-written web server is very likely to crash. > Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes There is a point in between 100LoC hand-written web server and K8s cluster. E. g. you need to serve static files and load is small to moderate (say <=1Gbps) you can do this in two ways: 1. Create a K8s cluster where HTTP traffic will be dynamically proxied to a container running on some set of nodes reading files from GlusterFS cluster. 2. Install nginx on a pair of physical servers (with disk configuration depending on the load: from HDD mirror to SSD RAID 10) and something like CARP for HA IPs shared between them. 2nd option is much simple and in most cases will be more reliable: in theory K8s option can have higher availability but it requires much more effort to not screw up something and much harder to troubleshoot when something goes wrong.
  - ocdtrekkie 5y ago
    My single Intel NUC in my basement on residential cable serves cloud services at higher reliability than Office 365. Sure, it doesn't do it for millions of people, but it's drastically less failure-prone, despite the lack of resilient design. What boggles the mind is people are often sold the cloud even though their org isn't serving millions of users, and could just as easily operate as a box in a closet.
lmilcin 5y ago
It should really say "overcomplicated system tend to break more frequently". It is not simplicity that makes for less downtime, it is unnecessary complication that does the opposite. I spend time to complicate my applications a little bit to make sure there is no downtime, something pretty important when one of the largest banks on Earth will stop along with your application. The simplest solutions would typically not be able to ensure no downtime operation. I need code so that I can do rolling upgrades and I need code so that my application can partition work and rebalance it reliably as cluster map changes. The problem starts when you start overdoing it. Maybe you are expecting too much in terms of guarantees. Or you want a simple guarantee but then you duct tape to it a huge and complicated clustering solution. Now you have a lot of problems. Your team doesn't know how it works. Your team doesn't know how it fails. It is not easy to tell if you are obviously integrated the right way. And so on. The goal, as usually, should be to "keep it simple, but not simpler than is necessary". For example, the approach we have chosen was to get by with as little guarantees as possible implemented as simply as possible. We decided on immutable data. We decided objects being saved as documents each with entire state of the object after each change. This costs a lot in space and processing needs, but you know what? It is fine. I work for a bank after all. The one think that costs more than space and processing power is downtime, and that's what we are trying to focus on. We know how to deal with duplicate data. So another rule of thumb: try to find compromises in your application, use them to replace hard problems with easier problems. Making an application super reliable is hard problem. Adding more storage space and memory is (relatively) easy one. If you can solve hard problem by replacing it with an easier one, you are winning.
- willis936 5y ago
  Indeed. Ships contain redundancy. Should this complication be removed to reduce downtime? Over-engineering does not mean "making the thing worse". It does if your engineers don't know what they're doing. If they do then the added complexity increases reliability.
- chousuke 5y ago
  Regarding "no downtime", there are very few applications where that's actually a good goal to have at all. Quite often, simple systems can provide you high availability where your downtime is a few seconds or minutes at most (during larger maintenance operations), and in many cases you can hide those blips by simply retrying (with proper backoff). There aren't many systems where you actually need to guarantee "no downtime". What I think is important most of the time instead of zero downtime is predictable behaviour when failures occur, so that your system doesn't end up in an undefined state where you don't know if you can recover without data loss.
_wldu 5y ago
I'm an architect and I keep things simple too. The problem I have is people around me expect more complex/intricate solutions. They think simple is not sophisticated and savvy. They think the competition is ahead of our 'old, simple solutions'. I have to continually justify and explain that KISS is always the right approach.
- tux3 5y ago
  Well, it's a trade-off, isn't it? Complexity can result from features which pay for themselves. Complexity is not automatically always wrong. At scale, complexity solves problems that you might not otherwise realize exist. Some of these problems are intrinsically complex, they can't be solved with a simpler solution. Not without "simple" organically growing into a much worse mess than the dominant complex solution that was designed by acknowledging complexity from the start. But what I want to respond to is your trying to explain that "KISS is always the right approach". Rules of thumb like these are cheap. They cause people to apply the same patterns regardless of the situation. Dispensing folk wisdom indiscriminately only serves to stop all thought and analysis. Zealous rule-based engineering is the deathrattle of good systems design, and it makes me ill at ease.
  - AlgorithmicTime 5y ago
    Complexity results in better productivity until it suddenly doesn't.
  - aserafini 5y ago
    Right, but because the ‘default’ inclination is towards ever increasing complexity, I still believe the mental exercise of attempting to simplify is almost always worthwhile. Meaning to say, I find KISS almost universally applicable, there are not many situations where you wouldn’t want to try and simplify (if you can).
  - tintt 5y ago
    People don’t always agree on what simplicity is. To some, older proven technology is simpler, even if it requires more time, hustle and code to use. To the other, the same problem solved with a newer intricate tech, but solved quicker and leaner, is a simpler solved problem.
094459 5y ago
In my experience complexity is more a factor of poor governance, management, focus, and purpose than the actual technology itself.
xet7 5y ago
Yes, let's replace these computers and smartphones with pen, paper and dumbphones.
- anyfoo 5y ago
  If you were to do that, those "systems" would definitely have much less downtime. It might not be as efficient or convenient, but that wasn't the question. If the efficiency of convenience of two competing system is sufficient, the less complex one will have fewer parts to fail, and cost less downtime. So for example for a "who touched this cabinet last" sheet, pen and paper would indeed often be better than smartphones.
zgniatacz 5y ago
“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” – Edsger W. Dijkstra
- kaba0 5y ago
  I think every talk on complexity is meaningless, without taking into account the difference between accidental and essential complexity. There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) is of course bad, and “clean” code or whatever should try to minimize its amount, but it is harmful in my view if we keep believing that it could be made simpler. One really bad example is this list: http://harmful.cat-v.org/software/ http://harmful.cat-v.org/software/
  - trasz 5y ago
    Would turning that font rendering into a web service an accidental complexity, or an essential one?
    - tomrod 5y ago
      Accidental, of course. There is never a need to introduce latency in text delivery, simply a long line of unmitigated wants.
      - trasz 5y ago
        Thai reduces a number of companies, eg Slack, to an “accidental complexity” then.
        mcphage 5y ago
        Are you saying that you think Slack's primary challenge is font rendering?
Puts 5y ago
When people face a new problem they intuitively try to find something they can add as a solution, not thinking about what they can remove to prevent the problem from occurring in the first place. This becomes apparent in the way marketing works. You sell products by promising they will solve a problem. You don't sell things by telling people what they really need to hear, that they just need less of everything. Simplicity is unfortunately not a product you can sell.
- SMAAART 5y ago
  Corporate America rewards higher budgets and adding new anything, in all practicality Corporate America rewards complexity. There's no glory in simplicity.
  - diognesofsinope 5y ago
    Because then you can employ more of your lawyer buddies to consult on the complexity.
- ethbr0 5y ago
  Part of the problem is counterparties. Usually, large parts of "the system" are owned by someone not-you. This differentiates who can do the work (or more typically, halt the work) in each case. Add something - you Remove something - you + other owners + all other users Which explains why system migrations only succeed when championed by a VP+.
- diognesofsinope 5y ago
  The other side of the coin is that complexity creates jobs because you need individual who guide you through it. Lawyers and bureaucrats careers are based on this complexity of criminal justice systems and government administration, respectively.
- jiggawatts 5y ago
  Did you just watch this tour of the Starbase with Elon Musk? He spends about 5 minutes talking about how removing things is the most essential part of a good design: https://youtu.be/t705r8ICkRw?t=833 https://youtu.be/t705r8ICkRw?t=833
Buttons840 5y ago
I'm going to make a controversial claim for the sake of argument. We can and will debate what programs are "simpler" all day, it's all subjective... except for one metric: line count. Line count is objective. Take pause before rejecting any change that uses fewer lines of code. It takes an awfully good abstraction to beat simply having less code.
- joekrill 5y ago
  It seems like it should be objective, but it isn't. Because you get people writing ridiculous "clever" one-liners that are difficult to parse and understand. That clever one-liner could be re-written to do the exact same thing and be much more clear and readable, but it would take, say, 8 lines instead of 1. Then there's the subjectiveness of what you actually consider a "line".
  - Buttons840 5y ago
    Fair. "Less code" is probably a better metric, but still has the same problem you describe. Still, some argue that an 8 line for-loop is better than a 1 line map or something, but a map is more constrained in what it can do than a free style for-loop. And 8 vs 1 lines in some corner of the program is less important than whether or not we make it a coding standard to write and use AbstractBeanThingyamabobs for all our stuff, etc.
  - Pokepokalypse 5y ago
    and then there's "syntactic sugar"
- Jtsummers 5y ago
  Kolmogorov complexity is close to what you're getting at, but is about total string length of the program which is more useful than line count when line count can be gamed. That is, a shorter program is "simpler" than a longer program. In quotes because it's not necessarily true (see code golfing, the language itself may become very complex to permit such a short program and the requisite knowledge and competency then increases the total complexity to achieve, or even understand, the simpler result). A similar technique that doesn't discount meaningful variable names might use syntactic token count. That way: printf("Hello, World!\n"); and: Put("Hello, World!\n"); Can be treated as the same complexity (the difference is 3 characters, and they are otherwise equivalent). There are also analysis methods (names escaping me, and Google fu is weak today) that look at loops, procedure calls, dependency graphs, and other things to attempt to discern (should be treated as guidelines and not rules) the complexity of a program using objective metrics. https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity
robertwt7 5y ago
Yes, that’s right. But then how long can you achieve simplicity after your startup grows? That’s very hard to implement
adevx 5y ago
I fell for the everything must be a microservice / distributed across as many servers as possible trap. Even though I've read so many warnings about it here on HA and knew upfront I might have to rewind everything. The setup : - Distributed file system using GlusterFS - DNS load balancing using Amazon Route 53 - PostgreSQL HA clusters using Patroni - A WireGuard mesh topology between all instances. Even though it was much fun setting this all up, the ballooning complexity of all this outpaced the benefits and it felt more fragile than where I came from. I decided to just scale vertically and keep it simple. Most of my refactoring these days is coming up with more simple, more manageable solutions.
- api 5y ago
  > Even though it was so much fun That’s the root of quite a bit of evil in software. A closely related motive is a desire to show off. Play is good for learning but not for production systems. You’ll regret it later when you are up at 4am on a Sunday morning troubleshooting some Byzantine stack. The antidote is the realization that simplicity is harder than complexity. Simple but highly effective systems are the ones that should inspire awe and admiration. Complexity is a sign of an immature design or a lack of high level conceptual thinking. A system should be only complex enough to capture essential complexity (problem domain requirements) and no more. One more thing… there is a ton of submarine marketing in our industry that encourages complexity because it’s profitable for vendors and cloud providers. The design you outline leads to more cloud resource consumption, more lock in to cloud platforms, and eventually a need for lots of service mesh, config management, and orchestration products that if the project grows will eventually start costing money.
  - dominotw 5y ago
    > That’s the root of quite a bit of evil in software. A closely related motive is a desire to show off. Yes but its also driven by keeping oneself employable. " I built a simple system to do X" is not a winning conversation in an interview. GP's complex system is what ppl say they built.
    - 5y ago
tacotacotacos 5y ago
This reads like a sales pitch. And the idea of letting a sales team sort and organize leads is impractical. I absolutely agree with the premise - but some of the anecdotes are laughable.
tintt 5y ago
That’s why I like Hacker News front end
jcq3 5y ago
Arguments are convincing
dredmorbius 5y ago
One of my hobbies is tracking down the origin of concepts and phrases. One such phrase is "complexity is the enemy", which I'd first encountered through the Jargon File, though it appears elsewhere, e.g., http://www.neugierig.org/software/blog/2011/04/complexity.html http://www.neugierig.org/software/blog/2011/04/complexity.ht... Tracing that trough Google's Ngram Viewer, I found the extended form, "complexity is the enemy of reliability" in a short item in a 1959 issue of The Economist (Jan 18, 1958, volume 186 https://books.google.com/books?id=aDsiAQAAMAAJ&q=%22complexity+is+the+enemy%22&dq=%22complexity+is+the+enemy%22&hl=en&sa=X&ei=_tZgU9ncAYGQyATBp4GYDg&ved=0CDUQ6AEwAQ https://books.google.com/books?id=aDsiAQAAMAAJ&q=%22complexi...). That in turn was referencing the UK's Annual Report of the Chief Inspector of Factories for the Year 1956, which had been published in late 1958. Among the factors assessed was factory and equipment reliability, which scaled precisely inversely with the number of components in equipment. https://www.worldcat.org/title/annual-report-of-hm-chief-inspector-of-factories/oclc/50350705?referer=di&ht=edition https://www.worldcat.org/title/annual-report-of-hm-chief-ins... Software isn't quite the same as mechanical equipment where individual components may wear, but each software component does exist and interact in relation to others. And eventually that complexity comes home to bite. A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. But extraneous components are a problem. (I cannot find either reference online, presently, though I've copies of both documents. I'd tracked the Economist issue through Google Books, and received a copy of the report through a friend. I'm told that the UK generally has excellent availability of government publications, though none of https://www.gov.uk https://www.gov.uk, the National Archives, nor the British Library seem to turn up the Factories report presently.)
- naasking 5y ago
  > A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. Perhaps "somewhat", but not entirely or arguably even mostly: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf
  - dredmorbius 5y ago
    Simplicity is simple to define for simple problems, but complex to define for complex ones.
ubermonkey 5y ago
A pattern I see with some companies -- usually, but not always, companies with little turnover, which leads to internal stagnation -- is the foundational belief that THEY ARE VERY SPECIAL and need things done a CERTAIN WEIRD WAY that is not supported by any commercial offering. Often, this leads to homegrown solutions that are never at feature-parity with COTS options, but have the benefit of costing more. If they buy something instead, they festoon it with a bunch of customization and external, parallel tools that must be kept in sync at the data and configuration level for everything to "work". What they NEVER do is re-examine their assumptions about how they must work, or interrogate why they can't go the same route as the bulk of the market on these points and thus have a simpler, more maintainable, less frustrating system in place. It's not quite this bad, but it's ALMOST like a company insisting that "well, here, for historical reasons, 2 and 2 are 5, not 4, so we have to do . . . . " It's exhausting, but OTOH, well, to the extend that my employer is involved, we tell them our recommendations but ultimately bill by the hour.
linsomniac 5y ago
"Complexity is the enemy of availability."
deleted 5y ago
[deleted]
Amin699 5y ago
Now imagine a sales process in which the sales team is simply notified of each new sales lead along with pertinent details, letting them decide whether or not to follow up with the lead. If the Salesforce notification step fails, it’s easy to come up with a hundred other ways of getting that information to the sales team: Reports, Slack notifications, list exports, manual observation, or using Zapier to send an alert through virtually any medium. The downtime would last a few minutes, at most.
rsync 5y ago
I gave a talk at CU Boulder[1] in 2019 on complexity in cloud infrastructure and the design of rsync.net. What I wrote, and then said several times was that: "Simple systems fail in boring ways. Complex systems fail in fascinating ways." I gave examples from Chernobyl, Air France Flight 447, etc. as examples of such. It's a topic that fascinates me and I had hoped to give several more such talks - but alas, the pandemic. In my opinion, the best discussion of this topic is the book _Normal Accidents_ by Charles Perrow[2]. [1] https://www.colorado.edu/libraries/2019/11/08/cyberinfrastructure-expert-speak-norlin-library https://www.colorado.edu/libraries/2019/11/08/cyberinfrastru... [2] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
- neolog 5y ago
  I'd love to see/read that talk if you happen to youtube it or write it up.
  - rsync 5y ago
    No, it was neither recorded nor transcribed. Some of my observations on complexity were touched upon in this interview, however: https://console.dev/interviews/rsync-john-kozubik/ https://console.dev/interviews/rsync-john-kozubik/ Specifically: "rsync.net has no firewalls and no routers. In each location we connect to our IP provider with a dumb, unmanaged switch. This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security." ... and I recommend these console.dev interviews in general - they're usually very interesting.
javajosh 5y ago
Sure, but you need a path there. How do you get from complex to simple? That's really hard. It's relatively easier to keep things simple over time. By default Money does not know or care about conscientious design. It wants functionality and it wants it now, future be damned. This is not irrational. Usually there is no future. The problem start with your unlikely success, and yes now you have money to throw at it, but ironically the people that are best suited to wrestle complexity to the ground are not super motivated by money. They are motivated by a peculiar kind of beauty that is, by hypothesis, missing in the successful, complex system, and so they will not want to contribute, money or no.
im_down_w_otp 5y ago
I'm very confused by the OP. There's legitimately nothing "simple" about the massive container ship system described in the OP. It's a complicated semi-autonomous cyber-physical system with multiple complex failure modes possible in the software, electrical, hydraulic, and/or mechanical domains. The reason said system has both the superficial appearance of simplicity to its operators and the high-probability of low-downtime is because its very complex systems and subsystems were fastidiously designed and built using tools, processes, and practices which helped manage and understand all that complexity. Not because it is "simple".
ChrisMarshallNY 5y ago
This is sort of like the old "Water is wet; news at 11." thing. But simple is not easy. In my experience, it often comes after complexity. My general approach to simplification is to first get it working; even in a complicated fashion, then start removing stuff, until it stops working. If I can't get it working on the latest simplicity, I am forced to add the last thing back. But I can usually get it going, which may sometimes require a rearchitecting. One of the happiest times for me, when I'm writing code, is when I get to toss out a whole bunch of painstaking work that I did. I'm going through that now. I'm working on an app that has necessary complexity, and I'm finding ways to toss out code all over the place.
- lostcolony 5y ago
  "First, make it correct. Then, make it beautiful. Then, if you need to, make it performant. Because 9 times out of 10, making it beautiful also makes it performant enough." - Joe Armstrong
henning 5y ago
Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. You get promoted by pulling heroics to build some absurdly complex cloud Kubernetes bullshit, and then pulling more heroics to fix the endless stream of production issues that will result from your overengineered nightmare.
- Jtsummers 5y ago
  http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight. EDIT: One past discussion here (2015): https://news.ycombinator.com/item?id=8940820 https://news.ycombinator.com/item?id=8940820
  - Aqueous 5y ago
    It is very bad. I spent the last year or so rebuilding a system with a fundamental architecture error that it would likely be fighting against for decades. My reward? The refactor was so difficult to do that I ended up getting dinged on timeline. And promotions got delayed The incentive structures at big companies are out of whack. Instead of the original developer, who was promoted to staff engineer, gettting dinged for making the mistake, I got dinged for not fixing it fast enough, and my career slowed down as a result. The refactor was incredibly difficult and launched without issue and I am probably back on track for a promotion, but I have started to make noise about the incentives being askew
    - Buttons840 5y ago
      > I have started to make noise about the incentives being askew Choosing the hard path again I see.
      - Aqueous 5y ago
        Haha, yep. I just think it's fundamentally unjust for the developers doing the (easiest) green field development to get promoted while making significant architecture mistakes, then punishing later developers for pointing out those mistakes and getting stuck with fixing them. Until someone makes noise, this is just going to keep happening.
France_is_bacon 5y ago
“I didn't have time to write a short letter, so I wrote a long one instead.” ― Mark Twain
heartunderbread 5y ago
Classic case of https://en.wikipedia.org/wiki/Occam's_razor https://en.wikipedia.org/wiki/Occam's_razor
ChrisArchitect 5y ago
previous discussion from a year ago when first submitted: https://news.ycombinator.com/item?id=22471355 https://news.ycombinator.com/item?id=22471355
mro_name 5y ago
or put the way round: smart systems have more downtime. In short: smart is stupid.

303 points by gk1 5y ago

So true. I am working with a system currently with tons of "moving parts", once it's up it's neat, but there's always at least 3 things broken every time I install it. The person who created it is the de facto expert, and everyone else just relies on them to fix it or extend the more tricky bits. It does leave one wonderful if the niftiness is worth the pain, or if there is a way of getting the good bits without the complexity.

AllegedAlec 5y ago

I disagree with the notion that a container ship is a simple system because 13 people can man it. It just means that the complexity is hidden in deeper layers.

motoboi 5y ago
Yes! In the future, a whole car factory will have only one employee. His job: feed the dog. The dog’s: keep humans away from the machines. Just as a ship with dozens of people got replaced by 13: the complexity was encapsulated in simpler interfaces. Diesel motors: just change this lever. Up is fast, down is slow. But it took a century to refine this absurdly complex device until this simple interface became possible. I think the point here is: don’t build your own diesel motors. It’s unlikely you’ll beat a good model and your interface to it will probably be worst. Remember: people take years learning from their own mistakes to hide the complexity under simpler interfaces layers. Unless your company IS building diesel motors.
- AllegedAlec 5y ago
  So what happens to your simple encapsulated system when the diesel motor breaks down, or one of your hydraulic lines breaks? Does your simple encapsulated system them also say "we have no downtime, because we have a simple system"?
  - cl3misch 5y ago
    I'm pretty sure that is the exact point the parent post is trying to make!
  - motoboi 5y ago
    You leave port with 10 replacements for every component of the encapsulated device. Or just a full replacement for it. Hidden complexity in the case of the diesel motor was made possible also because of know failure modes (how many years until the low hanging fruits all got solved, how many decades until the not so easy got too?) and standardization of its components (how many centuries just to understand and standardize the universal joint https://en.wikipedia.org/wiki/Universal_joint https://en.wikipedia.org/wiki/Universal_joint?).
naasking 5y ago
But the question is, is the 13 manned ship still simpler than one that requires people doing all of those jobs? ie. is maintaining the machinery less complicated than maintaining the equivalent number of people that would be required to do the same jobs? Arguably yes.
- AllegedAlec 5y ago
  I'm doubtful. You might be able to get to harbor more easily if something breaks down along the way, but actually getting the ship fixed afterwards might take longer, because the apparent simplicity is built using highly complex systems. You see the same thing in modern cars, which are often a real pain to fix, because it's all computers and complex parts, rather than one axle going from the steering wheel directly to the wheels.
  - naasking 5y ago
    > because the apparent simplicity is built using highly complex systems. More complex than humans, with wives, kids, need for downtime, hazard pay, insurance requirements, mental and physical limitations, and so on? A lot of the complexity of humans is invisible to us because we're just used to it.

jacquesm 5y ago

When applying this to software and you have a complex system there is a lot to learn from Joe Armstrong's excellent paper: https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf As well as from the way the Erlang language and runtime are organized.

threeseed 5y ago

People don't go out of their way to build complex systems. What happens is that it starts simple and becomes complex as more features are added often with limitations attached e.g. time, money etc. So yes this guy migrated from Marketo to Hubspot and it was simple. But the idea it will simply remain that way is laughable.

stingraycharles 5y ago
While what you’re saying is, of course, fair, the other side is also true: people often choose to complex a technology “because it’s future-proof!”, while they need just a simple system. There are tons of examples: CQRS over simple databases, Hadoop while a single server suffices, or even people choosing JIRA over a simple trello board. As such, the obvious answer is “it depends”, and making the right trade-off is rarely properly captured in a single rule of thumb such as “always choose simple”.
- embik 5y ago
  > a single server suffices A genuine question that I wonder about: How does a single server protect you against someone in your DC pulling a plug by mistake or your cloud provider being unreliable? If it’s a database it seems like a huge potential for going out of business. I guess if you are running single-server databases you better have a good backup strategy anyway, but is that a risk people and businesses just accept?
  - VBprogrammer 5y ago
    I think what you are missing is that simple databases in a well run data-centre are surprisingly reliable. The tradeoffs involved in avoiding some of the simple "the cleaner unplugged the DB server" introduce complexities which may themselves cause outages which are much harder to solve.
  - citrin_ru 5y ago
    If you have a single server pulling of a plug (or even any HW failure which cannot be fixed quickly) is a very rare even (unless you use cheap desktop hardware instead of a decent server with ECC RAM, hot-swap HDD, redundant PSU e. t. c.). If instead of using this single server (with manual fail-over in case it would fail) you'll start to create complex HA cluster from multiple servers not having enough people to properly design, test and maintain it you can end up having a less reliable system.

mschuster91 5y ago

To add a point: "simple" can also mean "standardized". Self-hosted or cloud-hosted Kubernetes is a highly complex system. But using it to power your internal tooling (e.g. Jira, Confluence, GitLab, Jenkins, whatever) over classic, manually-installed-on-VMs deployment has the advantage that you can drop in anyone with experience in Kubernetes to manage your workloads.

cryptica 5y ago
Kubernetes is a complex solution but it solves a very complex problem. It could have been made slightly simpler (and slightly less flexible too) but it's a really good solution IMO. The structure is mostly straight forward and intuitive (at least to me as a developer).
sdevonoes 5y ago
I would go one level up and ask "do we actually need Jira, Confluence, GitLab, Jenkins, whatever on premise?". The question then is not "K8s or manual way", but "Do we actually need K8s?"
- mschuster91 5y ago
  Many companies in the EU actually blanket ban any storage of their sensitive information in the cloud, especially the Atlassian cloud after the Australian espionage law change.
  - sdevonoes 5y ago
    That's a good point! Then we raise the question one level up: "does the current law regarding storage of sensitive information make sense?" It's a never ending story for those who like to delve in these sort of things.
    - mschuster91 5y ago
      For all what it's worth: yes. The larger the cloud, the larger the motivation for enemies to attack it. And in this case, the enemy is China (and, for European companies, also the US/FVEY alliance) - all of which have already been caught multiple times doing industrial espionage. Core company secrets belong on premises of the company owning the secret, not in a cloud where it is fair game for security services and criminals of all kinds and nations.

WolfOliver 5y ago

Would somebody agree that a MySQL/PostgreSQL cluster has more downtime than a replicated cluster? How often do high availability mechanism cause downtime itself?

Sebb767 5y ago

I think a better point would be "simple systems are quicker to fix" or, more concise, "avoid unnecessary complexity". In some way, complexity allows flaws to get in, but that does not mean it's unnecessary. Take, for example, website hosting. You can put your full page on a single server, but then you're prone to the machine going down, the disk having errors etc.. A more complex system, like Kubernetes, will be far harder to troubleshoot, but it will not go down just because one machine died. Or, as an engineering example, take planes: They're a lot more complex than ships. Fixing them when something is wrong can take long. But that complexity is needed so what when something fails, you get to the airport instead of going through a rapid unscheduled disassembly.

trasz 5y ago
Until recently even the huge airliners had physical strings going from the rudder all the way to the wing and tail control surfaces. Precisely because more complex systems were less reliable. Also, even with fly by wire there’s sometimes (or often, or always, no idea) a backup, eg a direct electric / hydraulic link.
- Pokepokalypse 5y ago
  sounds like the 737MAX. . .

namelosw 5y ago

The Author generalized "fewer features lead to less downtime" to "simple systems have less downtime". A simple system such as a hand-written web server is very likely to crash. It's very likely cannot serve many users if we don't make it serve requests concurrently. What's worse, one tiny exception in one request would bring the whole system down. It stops serving any request to any user. A relatively complex system with a self-recovery mechanism added goes a long way compared to the former. The Erlang implementation is much more complex than "an Erlang without processes". Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes. The real world is chaotic. If a system needs to strive for less downtime, it needs to have some features that mimic biological creatures, instead of being idealistic simple.

citrin_ru 5y ago
> A simple system such as a hand-written web server is very likely to crash. > Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes There is a point in between 100LoC hand-written web server and K8s cluster. E. g. you need to serve static files and load is small to moderate (say <=1Gbps) you can do this in two ways: 1. Create a K8s cluster where HTTP traffic will be dynamically proxied to a container running on some set of nodes reading files from GlusterFS cluster. 2. Install nginx on a pair of physical servers (with disk configuration depending on the load: from HDD mirror to SSD RAID 10) and something like CARP for HA IPs shared between them. 2nd option is much simple and in most cases will be more reliable: in theory K8s option can have higher availability but it requires much more effort to not screw up something and much harder to troubleshoot when something goes wrong.
- ocdtrekkie 5y ago
  My single Intel NUC in my basement on residential cable serves cloud services at higher reliability than Office 365. Sure, it doesn't do it for millions of people, but it's drastically less failure-prone, despite the lack of resilient design. What boggles the mind is people are often sold the cloud even though their org isn't serving millions of users, and could just as easily operate as a box in a closet.

lmilcin 5y ago

It should really say "overcomplicated system tend to break more frequently". It is not simplicity that makes for less downtime, it is unnecessary complication that does the opposite. I spend time to complicate my applications a little bit to make sure there is no downtime, something pretty important when one of the largest banks on Earth will stop along with your application. The simplest solutions would typically not be able to ensure no downtime operation. I need code so that I can do rolling upgrades and I need code so that my application can partition work and rebalance it reliably as cluster map changes. The problem starts when you start overdoing it. Maybe you are expecting too much in terms of guarantees. Or you want a simple guarantee but then you duct tape to it a huge and complicated clustering solution. Now you have a lot of problems. Your team doesn't know how it works. Your team doesn't know how it fails. It is not easy to tell if you are obviously integrated the right way. And so on. The goal, as usually, should be to "keep it simple, but not simpler than is necessary". For example, the approach we have chosen was to get by with as little guarantees as possible implemented as simply as possible. We decided on immutable data. We decided objects being saved as documents each with entire state of the object after each change. This costs a lot in space and processing needs, but you know what? It is fine. I work for a bank after all. The one think that costs more than space and processing power is downtime, and that's what we are trying to focus on. We know how to deal with duplicate data. So another rule of thumb: try to find compromises in your application, use them to replace hard problems with easier problems. Making an application super reliable is hard problem. Adding more storage space and memory is (relatively) easy one. If you can solve hard problem by replacing it with an easier one, you are winning.

willis936 5y ago
Indeed. Ships contain redundancy. Should this complication be removed to reduce downtime? Over-engineering does not mean "making the thing worse". It does if your engineers don't know what they're doing. If they do then the added complexity increases reliability.
chousuke 5y ago
Regarding "no downtime", there are very few applications where that's actually a good goal to have at all. Quite often, simple systems can provide you high availability where your downtime is a few seconds or minutes at most (during larger maintenance operations), and in many cases you can hide those blips by simply retrying (with proper backoff). There aren't many systems where you actually need to guarantee "no downtime". What I think is important most of the time instead of zero downtime is predictable behaviour when failures occur, so that your system doesn't end up in an undefined state where you don't know if you can recover without data loss.

_wldu 5y ago

I'm an architect and I keep things simple too. The problem I have is people around me expect more complex/intricate solutions. They think simple is not sophisticated and savvy. They think the competition is ahead of our 'old, simple solutions'. I have to continually justify and explain that KISS is always the right approach.

tux3 5y ago
Well, it's a trade-off, isn't it? Complexity can result from features which pay for themselves. Complexity is not automatically always wrong. At scale, complexity solves problems that you might not otherwise realize exist. Some of these problems are intrinsically complex, they can't be solved with a simpler solution. Not without "simple" organically growing into a much worse mess than the dominant complex solution that was designed by acknowledging complexity from the start. But what I want to respond to is your trying to explain that "KISS is always the right approach". Rules of thumb like these are cheap. They cause people to apply the same patterns regardless of the situation. Dispensing folk wisdom indiscriminately only serves to stop all thought and analysis. Zealous rule-based engineering is the deathrattle of good systems design, and it makes me ill at ease.
- AlgorithmicTime 5y ago
  Complexity results in better productivity until it suddenly doesn't.
- aserafini 5y ago
  Right, but because the ‘default’ inclination is towards ever increasing complexity, I still believe the mental exercise of attempting to simplify is almost always worthwhile. Meaning to say, I find KISS almost universally applicable, there are not many situations where you wouldn’t want to try and simplify (if you can).
- tintt 5y ago
  People don’t always agree on what simplicity is. To some, older proven technology is simpler, even if it requires more time, hustle and code to use. To the other, the same problem solved with a newer intricate tech, but solved quicker and leaner, is a simpler solved problem.

094459 5y ago

In my experience complexity is more a factor of poor governance, management, focus, and purpose than the actual technology itself.

xet7 5y ago

Yes, let's replace these computers and smartphones with pen, paper and dumbphones.

anyfoo 5y ago
If you were to do that, those "systems" would definitely have much less downtime. It might not be as efficient or convenient, but that wasn't the question. If the efficiency of convenience of two competing system is sufficient, the less complex one will have fewer parts to fail, and cost less downtime. So for example for a "who touched this cabinet last" sheet, pen and paper would indeed often be better than smartphones.

zgniatacz 5y ago

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” – Edsger W. Dijkstra

kaba0 5y ago
I think every talk on complexity is meaningless, without taking into account the difference between accidental and essential complexity. There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) is of course bad, and “clean” code or whatever should try to minimize its amount, but it is harmful in my view if we keep believing that it could be made simpler. One really bad example is this list: http://harmful.cat-v.org/software/ http://harmful.cat-v.org/software/
- trasz 5y ago
  Would turning that font rendering into a web service an accidental complexity, or an essential one?
  - tomrod 5y ago
    Accidental, of course. There is never a need to introduce latency in text delivery, simply a long line of unmitigated wants.
    - trasz 5y ago
      Thai reduces a number of companies, eg Slack, to an “accidental complexity” then.
      - mcphage 5y ago
        Are you saying that you think Slack's primary challenge is font rendering?

Puts 5y ago

When people face a new problem they intuitively try to find something they can add as a solution, not thinking about what they can remove to prevent the problem from occurring in the first place. This becomes apparent in the way marketing works. You sell products by promising they will solve a problem. You don't sell things by telling people what they really need to hear, that they just need less of everything. Simplicity is unfortunately not a product you can sell.

SMAAART 5y ago
Corporate America rewards higher budgets and adding new anything, in all practicality Corporate America rewards complexity. There's no glory in simplicity.
- diognesofsinope 5y ago
  Because then you can employ more of your lawyer buddies to consult on the complexity.
ethbr0 5y ago
Part of the problem is counterparties. Usually, large parts of "the system" are owned by someone not-you. This differentiates who can do the work (or more typically, halt the work) in each case. Add something - you Remove something - you + other owners + all other users Which explains why system migrations only succeed when championed by a VP+.
diognesofsinope 5y ago
The other side of the coin is that complexity creates jobs because you need individual who guide you through it. Lawyers and bureaucrats careers are based on this complexity of criminal justice systems and government administration, respectively.
jiggawatts 5y ago
Did you just watch this tour of the Starbase with Elon Musk? He spends about 5 minutes talking about how removing things is the most essential part of a good design: https://youtu.be/t705r8ICkRw?t=833 https://youtu.be/t705r8ICkRw?t=833

Buttons840 5y ago

I'm going to make a controversial claim for the sake of argument. We can and will debate what programs are "simpler" all day, it's all subjective... except for one metric: line count. Line count is objective. Take pause before rejecting any change that uses fewer lines of code. It takes an awfully good abstraction to beat simply having less code.

joekrill 5y ago
It seems like it should be objective, but it isn't. Because you get people writing ridiculous "clever" one-liners that are difficult to parse and understand. That clever one-liner could be re-written to do the exact same thing and be much more clear and readable, but it would take, say, 8 lines instead of 1. Then there's the subjectiveness of what you actually consider a "line".
- Buttons840 5y ago
  Fair. "Less code" is probably a better metric, but still has the same problem you describe. Still, some argue that an 8 line for-loop is better than a 1 line map or something, but a map is more constrained in what it can do than a free style for-loop. And 8 vs 1 lines in some corner of the program is less important than whether or not we make it a coding standard to write and use AbstractBeanThingyamabobs for all our stuff, etc.
- Pokepokalypse 5y ago
  and then there's "syntactic sugar"
Jtsummers 5y ago
Kolmogorov complexity is close to what you're getting at, but is about total string length of the program which is more useful than line count when line count can be gamed. That is, a shorter program is "simpler" than a longer program. In quotes because it's not necessarily true (see code golfing, the language itself may become very complex to permit such a short program and the requisite knowledge and competency then increases the total complexity to achieve, or even understand, the simpler result). A similar technique that doesn't discount meaningful variable names might use syntactic token count. That way: printf("Hello, World!\n"); and: Put("Hello, World!\n"); Can be treated as the same complexity (the difference is 3 characters, and they are otherwise equivalent). There are also analysis methods (names escaping me, and Google fu is weak today) that look at loops, procedure calls, dependency graphs, and other things to attempt to discern (should be treated as guidelines and not rules) the complexity of a program using objective metrics. https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity

robertwt7 5y ago

Yes, that’s right. But then how long can you achieve simplicity after your startup grows? That’s very hard to implement

adevx 5y ago

I fell for the everything must be a microservice / distributed across as many servers as possible trap. Even though I've read so many warnings about it here on HA and knew upfront I might have to rewind everything. The setup : - Distributed file system using GlusterFS - DNS load balancing using Amazon Route 53 - PostgreSQL HA clusters using Patroni - A WireGuard mesh topology between all instances. Even though it was much fun setting this all up, the ballooning complexity of all this outpaced the benefits and it felt more fragile than where I came from. I decided to just scale vertically and keep it simple. Most of my refactoring these days is coming up with more simple, more manageable solutions.

api 5y ago
> Even though it was so much fun That’s the root of quite a bit of evil in software. A closely related motive is a desire to show off. Play is good for learning but not for production systems. You’ll regret it later when you are up at 4am on a Sunday morning troubleshooting some Byzantine stack. The antidote is the realization that simplicity is harder than complexity. Simple but highly effective systems are the ones that should inspire awe and admiration. Complexity is a sign of an immature design or a lack of high level conceptual thinking. A system should be only complex enough to capture essential complexity (problem domain requirements) and no more. One more thing… there is a ton of submarine marketing in our industry that encourages complexity because it’s profitable for vendors and cloud providers. The design you outline leads to more cloud resource consumption, more lock in to cloud platforms, and eventually a need for lots of service mesh, config management, and orchestration products that if the project grows will eventually start costing money.
- dominotw 5y ago
  > That’s the root of quite a bit of evil in software. A closely related motive is a desire to show off. Yes but its also driven by keeping oneself employable. " I built a simple system to do X" is not a winning conversation in an interview. GP's complex system is what ppl say they built.
  - 5y ago

tacotacotacos 5y ago

This reads like a sales pitch. And the idea of letting a sales team sort and organize leads is impractical. I absolutely agree with the premise - but some of the anecdotes are laughable.

tintt 5y ago

That’s why I like Hacker News front end

jcq3 5y ago

Arguments are convincing

dredmorbius 5y ago

One of my hobbies is tracking down the origin of concepts and phrases. One such phrase is "complexity is the enemy", which I'd first encountered through the Jargon File, though it appears elsewhere, e.g., http://www.neugierig.org/software/blog/2011/04/complexity.html http://www.neugierig.org/software/blog/2011/04/complexity.ht... Tracing that trough Google's Ngram Viewer, I found the extended form, "complexity is the enemy of reliability" in a short item in a 1959 issue of The Economist (Jan 18, 1958, volume 186 https://books.google.com/books?id=aDsiAQAAMAAJ&q=%22complexity+is+the+enemy%22&dq=%22complexity+is+the+enemy%22&hl=en&sa=X&ei=_tZgU9ncAYGQyATBp4GYDg&ved=0CDUQ6AEwAQ https://books.google.com/books?id=aDsiAQAAMAAJ&q=%22complexi...). That in turn was referencing the UK's Annual Report of the Chief Inspector of Factories for the Year 1956, which had been published in late 1958. Among the factors assessed was factory and equipment reliability, which scaled precisely inversely with the number of components in equipment. https://www.worldcat.org/title/annual-report-of-hm-chief-inspector-of-factories/oclc/50350705?referer=di&ht=edition https://www.worldcat.org/title/annual-report-of-hm-chief-ins... Software isn't quite the same as mechanical equipment where individual components may wear, but each software component does exist and interact in relation to others. And eventually that complexity comes home to bite. A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. But extraneous components are a problem. (I cannot find either reference online, presently, though I've copies of both documents. I'd tracked the Economist issue through Google Books, and received a copy of the report through a friend. I'm told that the UK generally has excellent availability of government publications, though none of https://www.gov.uk https://www.gov.uk, the National Archives, nor the British Library seem to turn up the Factories report presently.)

naasking 5y ago
> A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. Perhaps "somewhat", but not entirely or arguably even mostly: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf
- dredmorbius 5y ago
  Simplicity is simple to define for simple problems, but complex to define for complex ones.

ubermonkey 5y ago

A pattern I see with some companies -- usually, but not always, companies with little turnover, which leads to internal stagnation -- is the foundational belief that THEY ARE VERY SPECIAL and need things done a CERTAIN WEIRD WAY that is not supported by any commercial offering. Often, this leads to homegrown solutions that are never at feature-parity with COTS options, but have the benefit of costing more. If they buy something instead, they festoon it with a bunch of customization and external, parallel tools that must be kept in sync at the data and configuration level for everything to "work". What they NEVER do is re-examine their assumptions about how they must work, or interrogate why they can't go the same route as the bulk of the market on these points and thus have a simpler, more maintainable, less frustrating system in place. It's not quite this bad, but it's ALMOST like a company insisting that "well, here, for historical reasons, 2 and 2 are 5, not 4, so we have to do . . . . " It's exhausting, but OTOH, well, to the extend that my employer is involved, we tell them our recommendations but ultimately bill by the hour.

linsomniac 5y ago

"Complexity is the enemy of availability."

deleted 5y ago

[deleted]

Amin699 5y ago

Now imagine a sales process in which the sales team is simply notified of each new sales lead along with pertinent details, letting them decide whether or not to follow up with the lead. If the Salesforce notification step fails, it’s easy to come up with a hundred other ways of getting that information to the sales team: Reports, Slack notifications, list exports, manual observation, or using Zapier to send an alert through virtually any medium. The downtime would last a few minutes, at most.

rsync 5y ago

I gave a talk at CU Boulder[1] in 2019 on complexity in cloud infrastructure and the design of rsync.net. What I wrote, and then said several times was that: "Simple systems fail in boring ways. Complex systems fail in fascinating ways." I gave examples from Chernobyl, Air France Flight 447, etc. as examples of such. It's a topic that fascinates me and I had hoped to give several more such talks - but alas, the pandemic. In my opinion, the best discussion of this topic is the book _Normal Accidents_ by Charles Perrow[2]. [1] https://www.colorado.edu/libraries/2019/11/08/cyberinfrastructure-expert-speak-norlin-library https://www.colorado.edu/libraries/2019/11/08/cyberinfrastru... [2] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents

neolog 5y ago
I'd love to see/read that talk if you happen to youtube it or write it up.
- rsync 5y ago
  No, it was neither recorded nor transcribed. Some of my observations on complexity were touched upon in this interview, however: https://console.dev/interviews/rsync-john-kozubik/ https://console.dev/interviews/rsync-john-kozubik/ Specifically: "rsync.net has no firewalls and no routers. In each location we connect to our IP provider with a dumb, unmanaged switch. This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security." ... and I recommend these console.dev interviews in general - they're usually very interesting.

javajosh 5y ago

Sure, but you need a path there. How do you get from complex to simple? That's really hard. It's relatively easier to keep things simple over time. By default Money does not know or care about conscientious design. It wants functionality and it wants it now, future be damned. This is not irrational. Usually there is no future. The problem start with your unlikely success, and yes now you have money to throw at it, but ironically the people that are best suited to wrestle complexity to the ground are not super motivated by money. They are motivated by a peculiar kind of beauty that is, by hypothesis, missing in the successful, complex system, and so they will not want to contribute, money or no.

im_down_w_otp 5y ago

I'm very confused by the OP. There's legitimately nothing "simple" about the massive container ship system described in the OP. It's a complicated semi-autonomous cyber-physical system with multiple complex failure modes possible in the software, electrical, hydraulic, and/or mechanical domains. The reason said system has both the superficial appearance of simplicity to its operators and the high-probability of low-downtime is because its very complex systems and subsystems were fastidiously designed and built using tools, processes, and practices which helped manage and understand all that complexity. Not because it is "simple".

ChrisMarshallNY 5y ago

This is sort of like the old "Water is wet; news at 11." thing. But simple is not easy. In my experience, it often comes after complexity. My general approach to simplification is to first get it working; even in a complicated fashion, then start removing stuff, until it stops working. If I can't get it working on the latest simplicity, I am forced to add the last thing back. But I can usually get it going, which may sometimes require a rearchitecting. One of the happiest times for me, when I'm writing code, is when I get to toss out a whole bunch of painstaking work that I did. I'm going through that now. I'm working on an app that has necessary complexity, and I'm finding ways to toss out code all over the place.

lostcolony 5y ago
"First, make it correct. Then, make it beautiful. Then, if you need to, make it performant. Because 9 times out of 10, making it beautiful also makes it performant enough." - Joe Armstrong

henning 5y ago

Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. You get promoted by pulling heroics to build some absurdly complex cloud Kubernetes bullshit, and then pulling more heroics to fix the endless stream of production issues that will result from your overengineered nightmare.

Jtsummers 5y ago
http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight. EDIT: One past discussion here (2015): https://news.ycombinator.com/item?id=8940820 https://news.ycombinator.com/item?id=8940820
- Aqueous 5y ago
  It is very bad. I spent the last year or so rebuilding a system with a fundamental architecture error that it would likely be fighting against for decades. My reward? The refactor was so difficult to do that I ended up getting dinged on timeline. And promotions got delayed The incentive structures at big companies are out of whack. Instead of the original developer, who was promoted to staff engineer, gettting dinged for making the mistake, I got dinged for not fixing it fast enough, and my career slowed down as a result. The refactor was incredibly difficult and launched without issue and I am probably back on track for a promotion, but I have started to make noise about the incentives being askew
  - Buttons840 5y ago
    > I have started to make noise about the incentives being askew Choosing the hard path again I see.
    - Aqueous 5y ago
      Haha, yep. I just think it's fundamentally unjust for the developers doing the (easiest) green field development to get promoted while making significant architecture mistakes, then punishing later developers for pointing out those mistakes and getting stuck with fixing them. Until someone makes noise, this is just going to keep happening.

France_is_bacon 5y ago

“I didn't have time to write a short letter, so I wrote a long one instead.” ― Mark Twain

heartunderbread 5y ago

Classic case of https://en.wikipedia.org/wiki/Occam's_razor https://en.wikipedia.org/wiki/Occam's_razor

ChrisArchitect 5y ago

previous discussion from a year ago when first submitted: https://news.ycombinator.com/item?id=22471355 https://news.ycombinator.com/item?id=22471355

mro_name 5y ago

or put the way round: smart systems have more downtime. In short: smart is stupid.