9 ms·

How Complex Systems Fail (1998)

366 points by mxschumacher 6y ago

soraki_soladead 6y ago
The actual title is “How Complex Systems Fail” (full url: how.complexsystems.fail) which provides a slightly different meaning. The current title “Complex Systems Fail” implies that we should not have complex systems but that’s does not appear to be the intention of the article. Rather we often require complex systems to solve complex problems and the article provides an exposition on failure in such systems: > (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety)
- chrismorgan 6y ago
  I believe HN removes words like “how” or “why” from the start of titles as part of its normalisation, though you can subsequently edit it to fix the title again. I strongly dislike this particular automatic edit because I reckon it’s normally wrong.
  - nitrogen 6y ago
    I wish it would remove the word "just" from titles of the form "X just Y'd a Z".
- namenotrequired 6y ago
  It's interesting the subtitle mentions "patients" when the entire text applies fully for very different contexts, say, cruise ships
  - inpharmer 6y ago
    Relative to aviation, nuclear power, and perhaps cruise ships, the spilled blood of victims of those complex systems resulted in improvements[0][1], while in healthcare the seminal text appeared first in 1999.[2] Healthcare has a lot to learn from other industries with inherent complexity. [0] https://en.wikipedia.org/wiki/Tenerife_airport_disaster#Legacy https://en.wikipedia.org/wiki/Tenerife_airport_disaster#Lega... [1] https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/3mile-isle.html#impact https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/3... [2] https://pubmed.ncbi.nlm.nih.gov/25077248/ https://pubmed.ncbi.nlm.nih.gov/25077248/
ayende 6y ago
This is a short but really good discussion on real world failures and how to build robust systems. I refer to it often when thinking about the design of my systems.
hawktheslayer 6y ago
I see #13 (human expertise) crop up the most often in the complex systems at my work. I think the two main reasons replacing experts is so hard is (1) it's inherently difficult to understand another's work you haven't built from yourself and (2) it's nearly impossible to find someone as enthusiastic for taking on a system and giving it the TLC it needs to keep it going smoothly to the extent that it matches the passion that the original creator poured into it. I find this most acutely whenever the original expert refers to it as "their baby".
mv4 6y ago
For a good analysis of risks and failures associated with complex systems, I recommend "Normal Accidents: Living with High-Risk Technologies" by Charles Perrow (1999).
- csours 6y ago
  I won't make a recommendation either way on this book, but I will say that it came off as much more of an opinionated screed than I was expecting; don't go into it expecting an even keel
wpietri 6y ago
For those interested in the topic, especially point 7 and 8 (root cause and hindsight bias), I strongly recommend Dekker's "A Field Guide to 'Human Error'". It digs in on those points in a very practical way. We recently had a minor catastrophe at work; it was very useful in preparing me to guide people away from individual blame and toward systemic thinking.
- csours 6y ago
  I strongly agree with this recommendation. Personal Experience: During and after the economic collapse of 2007-2009 I wondered who was at fault; who to blame. I kept waiting for a clear answer to "THE root cause". Since then I've read things like Dekker's work, and come to realize that blame is not a productive way of thinking in complex systems. A quick example: in many car accidents, you can easily point to the person who caused the accident; for example the person who runs a red light, texts and drives, or drives drunk is easily found at fault. But what about a case where someone 3 or 4 car lengths ahead makes a quick lane change and an accident occurs behind them?
  - deleted 6y ago
    [deleted]
  - narag 6y ago
    I wrote another comment, before reading the article that says it better than me: Catastrophe requires multiple failures – single point failures are not enough.
  - Joeri 6y ago
    This is why I prefer thinking about root solutions rather than root causes. The answer to the question “how can we make sure something like this cannot happen again?” (for a reasonably wide definition of this). The nice thing is that there are usually many right answers, all of which can be implemented, while when looking for root causes there may not actually be one.
    - travmatt 6y ago
MrXOR 6y ago
Past discussion: https://news.ycombinator.com/item?id=8282923 https://news.ycombinator.com/item?id=8282923 This is a great paper, I bookmarked this webpage.
wenc 6y ago
The accompanying talk by Richard I Cook at the O'Reilly Velocity conference is here: https://www.youtube.com/watch?v=2S0k12uZR14 https://www.youtube.com/watch?v=2S0k12uZR14 When I worked in operations, I would watch this talk every couple of months to remind myself of the principles. The principles are summarized in the list, but the talk adds a bit more context. I find that this context matters for seeing how the principles might be applied.
cs702 6y ago
I find it very instructive to substitute words like "human," "practitioners," and "people" in this essay with the word "AI," and re-read the essay from the standpoint of building autonomous AI agents that can safely run complex systems whose failures can be catastrophic, like driving a car or maneuvering a rocket. The essay becomes a kind of "guiding principles for building and evaluating autonomous AI." Here are five sample paragraphs in which I replaced all words referring to human beings to "AI:" -- Hindsight biases post-accident assessments of AI performance. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to the AI at the time than was actually the case. This means that ex post facto accident analysis of the AI performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of the AI before the accident of those same factors. It seems that the AI “should have known” that the factors would “inevitably” lead to an accident. Hindsight bias remains the primary obstacle to accident investigation, especially when AI performance is involved. All AI actions are gambles. After accidents, the overt failure often appears to have been inevitable and the AI’s actions as blunders or deliberate willful disregard of certain impending failure. But all AI actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That AI actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated. Actions at the sharp end resolve all ambiguity. Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of AIs at the sharp end of the system. After an accident, AI actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure. Views of ‘cause’ limit the effectiveness of defenses against future events. Post-accident remedies for “AI error” are usually predicated on obstructing activities that can “cause” accidents. These end-of-the-chain measures do little to reduce the likelihood of further accidents. In fact that likelihood of an identical accident is already extraordinarily low because the pattern of latent failures changes constantly. Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult. Failure free operations require AI experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where AIs can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, AIs are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing AIs with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope. -- PS. The original essay, published circa 2002, is also available here as a PDF: https://www.researchgate.net/publication/228797158_How_complex_systems_fail https://www.researchgate.net/publication/228797158_How_compl...
rawgabbit 6y ago
Agree with Pt 11. “After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure.” Management pressure to produce reminds me of the phrase “We don’t know where we are going but we want to get there as soon as possible.” Garbage in garbage out. And no. Endless meetings are not a good substitute for anything.
- vegetablepotpie 6y ago
  Management pressure to produce is coupled closely with practitioner actions are gambles Pt. 10, and these two things lead to less efficiency. I see the function of management at my workplace is to get practitioners to take bigger gambles to produce more favorable outcomes. This is convenient for management because they can capture the upside of a gamble and focus blame on an individual on the downside of a gamble. One of the practitioner reactions I've seen to this in government contracting is to become hyper specialized in one specific job function. Don't be a jack of all trades, don't wear multiple hats. Don't spring up to help someone. This leads to two outcomes. 1. When a practitioner is confronted with a novel problem, they will say that's not their job and that persons X, Y, or Z can help. Person X directs you to person A, persons Y and Z direct you to B, person A also directs you to person B and person B directs you back to person X. You ultimately find that no one is responsible for anything. 2. When an operation does need to be performed, it takes tiny contributions from many people. This leads to long chains of activities that take months to do simple operations, such as purchasing items because there is inevitably someone who is out sick or on vacation for two weeks. This in turn leads to more hiring, and then management hits back with initiatives to streamline operations. It is a fascinating cycle.
dukeofdoom 6y ago
I think we should build complexity using sort of adversarial system, where things compete like in Nature. The great lakes is a complex eco system. I remember when Zebra mussels first appeared in the Great Lakes. Tens of articles appeared, how this was an epic natural disaster, that would kill the Great Lakes Eco system, because Zebra mussels would eat all the food. Now decades later, it turns out that Zebra mussels are great at filter feeding, and have thrived. But this also cleared up the water greatly. To the point that the lakes can look almost tropical, ocean clear, on sunny summer days. Which is probably great for all kinds of plants that now thrive on the extra sunshine reaching deeper down. It seems like the system just rearranged itself, and new opportunities appeared for some other plants to thrive.
- TeMPOraL 6y ago
  In nature, everything is fine, including nothing at all, because nature doesn't care either way - it just is. If Zebra mussels turned the Great Lakes sterile, it would be fine by nature. When we care about a system, we care about specifics - we want it to look in some way, do some thing or provide us with some things. E.g. we want clean-looking, odorless lakes safe for recreation and full of fish to eat. That's a specific outcome that may or may not be what the natural selection will provide; to ensure we get what we want, we need to supply a strong selection pressure of our design. Adversarial processes are effective given sufficient time, but about the least efficient way to build a system after just waiting for it to appear at random. Humanity's dominance over other life on Earth is in big way based on the ability to predict and invent things "up front", without any competitive selection. I feel we should refine that skill and find more opportunities to use it, because it's much less wasteful.
0xbadcafebee 6y ago
> The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes. I've heard this line before and I don't buy it. Methods exist that have been proven to identify root causes and those have led to significant reductions in failures. Half this page talks about how systems are made resilient to failures over time. But to do that you need to identify the failures' root causes! The Five Whys is one example of an effective method to find root causes. The solution may not be simple, but it does lead to more robust systems.
- bobsomers 6y ago
  Avoiding the trap of "root causes" is exactly what has lead to such a significant reduction in failures! Avoiding "root cause" thinking doesn't mean that there isn't something that can be found and fixed. The intent is to keep your mind open to the fact that these systems and the interaction of their components are phenomenally complex. There are many contributing factors to a loss, not just a single error. Root cause analysis is harmful because it artificially limits your investigation to a predetermined "single" error or fault. Once the first or most salient error or fault is found (very commonly a human operator), the investigation stops because the "root cause" was found. Instead of stopping at the root cause, you keep digging in attempt to find all the possible contributing factors, no matter how big or small. You can prevent many future failures of the system, not just the same one, by learning about many places where your defenses and mitigations were weak. NTSB reports are a great example of how useful this approach is. Instead of identifying a root cause, they list many contributing factors, all of which can be improved to raise the level of safety significantly rather the fixing a single root cause. As a recent example, think of the Boeing 737 MAX. What was the "root cause"? Was it: * A software bug in the MCAS system? * The removal of a redundant method for sensing angle of attack? * The decision to make the "AoA Disagree" light an optional upgrade? * Making substantially unstable changes to the existing airframe? * The desire for it be be "another 737 variant" for pilot classification? * The insufficient pilot training w.r.t. to the new MCAS system? * Organizational pressures to not lose the AA contract to Airbus? * Executive leadership that prioritized business over safety? The answer is all of the above, and more! By identifying all of these past "a software bug in the MCAS system" we have the opportunity to fix many safety issues which could be contributing factors to future failures. There is an even more in depth example of this in Chapter 5 of Nancy Leveson's book Engineering a Safer World. She spends over 60 pages describing the environment and circumstances that lead to a US Air Force F-15 shooting down a friendly US Army Black Hawk over northern Iraq in 1994.
afarrell 6y ago
For those looking for ways to talk to non-engineers about this topic (without them rolling their eyes and telling you to stop making excuses), I highly recommend the book Leadership is Language.
master_yoda_1 6y ago
You forget about Lithium ion battery and nuclear fusion and many more examples. All the problems can't be solved by simple solution you need to do complex things.
- kaiomagalhaes 6y ago
  The article doesn't say we can't have complex systems. It simply states they are doomed to fail in some way, and that we need to be prepared for that
  - master_yoda_1 6y ago
    That is not true. Nuclear reactor are far more complex and not doomed to fail one day. So stop the bullshit.
  - master_yoda_1 6y ago
    I think by complex system you mean a shitty system build by jokers which is doomed to fail. You can’t just build any bullshit and called it complex.
jwdomb 6y ago
This reads like a crisp summary of a subset of John Gall's "Systemantics": https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics
adrianmonk 6y ago
I would modify the section on root causes. It's not a fundamentally bad idea to look for root causes (plural). It's essential to not stop at immediate causes. More or less, you start at immediate causes and then recursively go deeper until you get to things that cannot be changed and/or are not within your control. One real danger is believing in a root cause or the root cause (singular) as if there is only one. The desire to only have to change (or understand) one thing is dangerous because it leads you to stop investigating/analyzing as soon as you have found something to blame it on. As you go deeper through the chain(s) of causes, you also should not focus exclusively on the last (root) because that means you aren't thinking about defense in depth. TLDR, follow the chain of causes all the way to its ends, realize it may have multiple ends, and don't focus only on the surface stuff or only on the root stuff.
- di4na 6y ago
  You are making the assumption that causality is useful and exist in these complex systems. Dr Cook base his treaty on what we know from the research of them, which is that these causal links are far less stronger than you think. This means that searching along a causal tree is indeed detrimental fundamentally.
  - adrianmonk 6y ago
    I'll give an example to illustrate. Situation: my software service stops working because it ran out of disk space. Insufficient focus on root cause: I add more disk, do nothing else, and assume it's solved. Excessive focus on root cause only: I investigate, find there's a leak where old data never gets deleted, call it the root cause, fix the leak, and do nothing else. Balanced approach: I fix the leak AND stop running at such tight margins (allocate breathing room) AND add automated monitoring for low disk space. The lesson to look for the root cause is valuable. Without it, disk space fills up rapidly and my other measures to insulate against that can only go so far. But if it's all I do, that's also bad.
deleted 6y ago
[deleted]
somurzakov 6y ago
>>Safety is a characteristic of systems and not of their components >>Safety is an emergent property of systems; it does not reside in a person, device or department of an organization or system. Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system. This means that safety cannot be manipulated like a feedstock or raw material. The state of safety in any system is always dynamic; continuous systemic change insures that hazard and its management are constantly changing. fits perfectly the description of Security of complex IT systems, nice way to explain why IT security marketplace is a wild wild west
marnett 6y ago
Is this the primary source of this https://fs.blog/2014/04/how-complex-systems-fail/ https://fs.blog/2014/04/how-complex-systems-fail/ ?
- di4na 6y ago
  As this is a copy of the paper, probably. I would point out that Dr Cook would probably disagree greatly with the whole idea of AntiFragile, or at least has the few time i have been able to talk with him. This link you offer is probably not the best context in which to comment on this paper.
- jeffreygoesto 6y ago
  The Morning Paper [0] has a nice discussion and quotes the original paper from 2000. The same link is at the very bottom of the post, behind "original". However, it seems that MIT rearranged something and the link has gone stale. Maybe someone at MIT could make it appear under the original URL [1] again? [0] https://blog.acolyer.org/2016/02/10/how-complex-systems-fail/ https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... [1] http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...
crdrost 6y ago
This is a wonderful article, but I feel like it is kind of missing a certain depth. Read Donella Meadows’ classic Thinking in Systems for instance. So like, yes, of necessity the largeness of a system causes the existence of feedback loops such that it maintains an equilibrium state, a homeostasis. This is kind of just a restatement of what a system is, and to characterize it as “the system is constantly failing,” while that is totally true, seems a little bit sensationalist. Like, you are one of these complex systems, do I say that you are constantly failing to draw breath? No, but I imagine some of your alveolar sacs routinely fail to fully inflate and others are too covered with mucous to absorb oxygen and whatever else, and often your breath runs ragged. Similarly with “you cannot identify a root cause,” a kid dies in a pool and we routinely have a medical examiner pronounce that the cause of death was drowning, which in the respiratory sense can be fairly well isolated to a root cause, “could not breathe air because no air was supplied by the system upstream.” The failure to find a root cause essentially only becomes hard because there is another aspect of complex systems which Meadows discusses, which is that they bleed into each other and into the World At Large, they do not have clearly defined boundaries until we impose them for analysis. Draw the boundaries larger than the respiratory system, say the social system, and now the reason that the kid died has something to do with parents never having been trained on what drowning actually looks like, so they heard some light splashing and looked over and said “aww isn’t he enjoying the water” not realizing that this was actually a “holy fuck my son is struggling to keep his mouth above water” moment—this perhaps speaks to a larger problem of not having good water education in society, maybe that is the root cause. Stuff like that. In that sense yes there are failures at all levels, but at any given level of explanation byou can more or less say that this is the thing that at that level explains the problem. So my servers go down, there is a valid root cause at the level of the individual server; there is also one at the level of the cluster as a whole and what that was doing; there is a valid cause at our developer and operations practices; there is a different cause at the level of company culture too. I don’t think that it makes sense to say that there is no root cause. The root cause of the Challenger explosion was that it was too cold to fly that day and they still flew the shuttle anyway. One can zoom in (“the fuel tanks were supposed to be sealed by rubber rings, but at the cold temperature of launch day the rubber no longer sealed the tanks”) or out (“NASA leadership built a false confidence in the quality of their work bolstered by a sort of management double-speak that disconnected them from the real risks they were playing with, making them foolhardy in ordering the launch”) and one can even sort of pan sideways (“why was the weather so cold that day?” or “what financial and engineering considerations made them choose this construction with these o-rings, several years ago?”). But this “middle explanation” of “it was too cold to fly that day and they flew the shuttle anyway” is a good summary that gives an entry-point to the problem, which is the real function of a root cause analysis (you want to collect information about how to fix the problem and know whether it was fixed). The fact that this is a little bit over-sensational detracts a little from my enjoyment but I do think that the heart of the article is great and fun to think about. If folks like thinking about this stuff I really recommend Meadows's book though, systems thinking is not present in most curricula and that is a real shame.
bitminer 6y ago
>>[The] defenses include obvious technical components (e.g. backup systems, ‘safety’ features of equipment) and human components (e.g. training, knowledge) but also a variety of organizational, institutional, and regulatory defenses (e.g. policies and procedures, certification, work rules, team training). This omits "design" for defenses against problems. Example: the chemical industries in many countries in the 1960s had horrendous accident records: many employees were dying on the job. (For many reasons) the owners re-engineered their plants to substantially reduce overall accident rates. "Days since a lost-time accident" became a key performance indicator. A key engineering process was introduced: HAZOP. The chemical flows were evaluated under all conditions: full-on, full-stop, and any other situations contrary to the design. Hazards from equipment failures or operational mistakes are thus identified and the design is adjusted to mitigate them. This was s.o.p. in the 1980s. See Wikipedia for an intro. Similar approaches could help IT and other systems.
treelovinhippie 6y ago
Complex system: nature. Complicated system: things mentioned in this article.
victor106 6y ago
We implemented a Microservices architecture that was promised and looked to be so simple on paper but became exponentially complex as each domain was implemented so we went back to a simpler monolithic service and it was much simpler to operant manage.
ngneer 6y ago
Great post. Uncanny how much of it maps almost 1:1 to security, which is too an emergent property of a complex system.
dang 6y ago
If curious see also 2019 https://news.ycombinator.com/item?id=20380055 https://news.ycombinator.com/item?id=20380055 2017 https://news.ycombinator.com/item?id=15002683 https://news.ycombinator.com/item?id=15002683 2017 https://news.ycombinator.com/item?id=14127543 https://news.ycombinator.com/item?id=14127543 2014: https://news.ycombinator.com/item?id=8282923 https://news.ycombinator.com/item?id=8282923
ineedasername 6y ago
"Complex systems run in degraded mode." This is the point that struck me the most. "Good enough" is good enough when there are so many other more urgent needs to address and you've been able to keep a system functioning despite extensive deferred maintenance. It is difficult to get allocation of resources to a system that's still working because it's robust when there are less robust (and probably less critical) systems that won't work without more resources being thrown at it.
b0ner_t0ner 6y ago
Slightly related, .fail TLDs were delegated back in 2014: https://icannwiki.org/.fail https://icannwiki.org/.fail
edrobap 6y ago
There is a remarkable resemblance between complex systems and hacks, except for point 7 and 8 that are trivial in case of latter. Perhaps most complex systems are hack that have been around for long enough to make everyone believe that they are well thought of to be called systems.

366 points by mxschumacher 6y ago

The actual title is “How Complex Systems Fail” (full url: how.complexsystems.fail) which provides a slightly different meaning. The current title “Complex Systems Fail” implies that we should not have complex systems but that’s does not appear to be the intention of the article. Rather we often require complex systems to solve complex problems and the article provides an exposition on failure in such systems: > (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety)

chrismorgan 6y ago
I believe HN removes words like “how” or “why” from the start of titles as part of its normalisation, though you can subsequently edit it to fix the title again. I strongly dislike this particular automatic edit because I reckon it’s normally wrong.
- nitrogen 6y ago
  I wish it would remove the word "just" from titles of the form "X just Y'd a Z".
namenotrequired 6y ago
It's interesting the subtitle mentions "patients" when the entire text applies fully for very different contexts, say, cruise ships
- inpharmer 6y ago
  Relative to aviation, nuclear power, and perhaps cruise ships, the spilled blood of victims of those complex systems resulted in improvements[0][1], while in healthcare the seminal text appeared first in 1999.[2] Healthcare has a lot to learn from other industries with inherent complexity. [0] https://en.wikipedia.org/wiki/Tenerife_airport_disaster#Legacy https://en.wikipedia.org/wiki/Tenerife_airport_disaster#Lega... [1] https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/3mile-isle.html#impact https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/3... [2] https://pubmed.ncbi.nlm.nih.gov/25077248/ https://pubmed.ncbi.nlm.nih.gov/25077248/

ayende 6y ago

This is a short but really good discussion on real world failures and how to build robust systems. I refer to it often when thinking about the design of my systems.

hawktheslayer 6y ago

I see #13 (human expertise) crop up the most often in the complex systems at my work. I think the two main reasons replacing experts is so hard is (1) it's inherently difficult to understand another's work you haven't built from yourself and (2) it's nearly impossible to find someone as enthusiastic for taking on a system and giving it the TLC it needs to keep it going smoothly to the extent that it matches the passion that the original creator poured into it. I find this most acutely whenever the original expert refers to it as "their baby".

mv4 6y ago

For a good analysis of risks and failures associated with complex systems, I recommend "Normal Accidents: Living with High-Risk Technologies" by Charles Perrow (1999).

csours 6y ago
I won't make a recommendation either way on this book, but I will say that it came off as much more of an opinionated screed than I was expecting; don't go into it expecting an even keel

wpietri 6y ago

For those interested in the topic, especially point 7 and 8 (root cause and hindsight bias), I strongly recommend Dekker's "A Field Guide to 'Human Error'". It digs in on those points in a very practical way. We recently had a minor catastrophe at work; it was very useful in preparing me to guide people away from individual blame and toward systemic thinking.

csours 6y ago
I strongly agree with this recommendation. Personal Experience: During and after the economic collapse of 2007-2009 I wondered who was at fault; who to blame. I kept waiting for a clear answer to "THE root cause". Since then I've read things like Dekker's work, and come to realize that blame is not a productive way of thinking in complex systems. A quick example: in many car accidents, you can easily point to the person who caused the accident; for example the person who runs a red light, texts and drives, or drives drunk is easily found at fault. But what about a case where someone 3 or 4 car lengths ahead makes a quick lane change and an accident occurs behind them?
- deleted 6y ago
  [deleted]
- narag 6y ago
  I wrote another comment, before reading the article that says it better than me: Catastrophe requires multiple failures – single point failures are not enough.
- Joeri 6y ago
  This is why I prefer thinking about root solutions rather than root causes. The answer to the question “how can we make sure something like this cannot happen again?” (for a reasonably wide definition of this). The nice thing is that there are usually many right answers, all of which can be implemented, while when looking for root causes there may not actually be one.
  - travmatt 6y ago

MrXOR 6y ago

Past discussion: https://news.ycombinator.com/item?id=8282923 https://news.ycombinator.com/item?id=8282923 This is a great paper, I bookmarked this webpage.

wenc 6y ago

The accompanying talk by Richard I Cook at the O'Reilly Velocity conference is here: https://www.youtube.com/watch?v=2S0k12uZR14 https://www.youtube.com/watch?v=2S0k12uZR14 When I worked in operations, I would watch this talk every couple of months to remind myself of the principles. The principles are summarized in the list, but the talk adds a bit more context. I find that this context matters for seeing how the principles might be applied.

cs702 6y ago

I find it very instructive to substitute words like "human," "practitioners," and "people" in this essay with the word "AI," and re-read the essay from the standpoint of building autonomous AI agents that can safely run complex systems whose failures can be catastrophic, like driving a car or maneuvering a rocket. The essay becomes a kind of "guiding principles for building and evaluating autonomous AI." Here are five sample paragraphs in which I replaced all words referring to human beings to "AI:" -- Hindsight biases post-accident assessments of AI performance. Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to the AI at the time than was actually the case. This means that ex post facto accident analysis of the AI performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of the AI before the accident of those same factors. It seems that the AI “should have known” that the factors would “inevitably” lead to an accident. Hindsight bias remains the primary obstacle to accident investigation, especially when AI performance is involved. All AI actions are gambles. After accidents, the overt failure often appears to have been inevitable and the AI’s actions as blunders or deliberate willful disregard of certain impending failure. But all AI actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That AI actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated. Actions at the sharp end resolve all ambiguity. Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of AIs at the sharp end of the system. After an accident, AI actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure. Views of ‘cause’ limit the effectiveness of defenses against future events. Post-accident remedies for “AI error” are usually predicated on obstructing activities that can “cause” accidents. These end-of-the-chain measures do little to reduce the likelihood of further accidents. In fact that likelihood of an identical accident is already extraordinarily low because the pattern of latent failures changes constantly. Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult. Failure free operations require AI experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where AIs can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, AIs are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing AIs with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope. -- PS. The original essay, published circa 2002, is also available here as a PDF: https://www.researchgate.net/publication/228797158_How_complex_systems_fail https://www.researchgate.net/publication/228797158_How_compl...

rawgabbit 6y ago

Agree with Pt 11. “After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure.” Management pressure to produce reminds me of the phrase “We don’t know where we are going but we want to get there as soon as possible.” Garbage in garbage out. And no. Endless meetings are not a good substitute for anything.

vegetablepotpie 6y ago
Management pressure to produce is coupled closely with practitioner actions are gambles Pt. 10, and these two things lead to less efficiency. I see the function of management at my workplace is to get practitioners to take bigger gambles to produce more favorable outcomes. This is convenient for management because they can capture the upside of a gamble and focus blame on an individual on the downside of a gamble. One of the practitioner reactions I've seen to this in government contracting is to become hyper specialized in one specific job function. Don't be a jack of all trades, don't wear multiple hats. Don't spring up to help someone. This leads to two outcomes. 1. When a practitioner is confronted with a novel problem, they will say that's not their job and that persons X, Y, or Z can help. Person X directs you to person A, persons Y and Z direct you to B, person A also directs you to person B and person B directs you back to person X. You ultimately find that no one is responsible for anything. 2. When an operation does need to be performed, it takes tiny contributions from many people. This leads to long chains of activities that take months to do simple operations, such as purchasing items because there is inevitably someone who is out sick or on vacation for two weeks. This in turn leads to more hiring, and then management hits back with initiatives to streamline operations. It is a fascinating cycle.

dukeofdoom 6y ago

I think we should build complexity using sort of adversarial system, where things compete like in Nature. The great lakes is a complex eco system. I remember when Zebra mussels first appeared in the Great Lakes. Tens of articles appeared, how this was an epic natural disaster, that would kill the Great Lakes Eco system, because Zebra mussels would eat all the food. Now decades later, it turns out that Zebra mussels are great at filter feeding, and have thrived. But this also cleared up the water greatly. To the point that the lakes can look almost tropical, ocean clear, on sunny summer days. Which is probably great for all kinds of plants that now thrive on the extra sunshine reaching deeper down. It seems like the system just rearranged itself, and new opportunities appeared for some other plants to thrive.

TeMPOraL 6y ago
In nature, everything is fine, including nothing at all, because nature doesn't care either way - it just is. If Zebra mussels turned the Great Lakes sterile, it would be fine by nature. When we care about a system, we care about specifics - we want it to look in some way, do some thing or provide us with some things. E.g. we want clean-looking, odorless lakes safe for recreation and full of fish to eat. That's a specific outcome that may or may not be what the natural selection will provide; to ensure we get what we want, we need to supply a strong selection pressure of our design. Adversarial processes are effective given sufficient time, but about the least efficient way to build a system after just waiting for it to appear at random. Humanity's dominance over other life on Earth is in big way based on the ability to predict and invent things "up front", without any competitive selection. I feel we should refine that skill and find more opportunities to use it, because it's much less wasteful.

0xbadcafebee 6y ago

> The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes. I've heard this line before and I don't buy it. Methods exist that have been proven to identify root causes and those have led to significant reductions in failures. Half this page talks about how systems are made resilient to failures over time. But to do that you need to identify the failures' root causes! The Five Whys is one example of an effective method to find root causes. The solution may not be simple, but it does lead to more robust systems.

bobsomers 6y ago
Avoiding the trap of "root causes" is exactly what has lead to such a significant reduction in failures! Avoiding "root cause" thinking doesn't mean that there isn't something that can be found and fixed. The intent is to keep your mind open to the fact that these systems and the interaction of their components are phenomenally complex. There are many contributing factors to a loss, not just a single error. Root cause analysis is harmful because it artificially limits your investigation to a predetermined "single" error or fault. Once the first or most salient error or fault is found (very commonly a human operator), the investigation stops because the "root cause" was found. Instead of stopping at the root cause, you keep digging in attempt to find all the possible contributing factors, no matter how big or small. You can prevent many future failures of the system, not just the same one, by learning about many places where your defenses and mitigations were weak. NTSB reports are a great example of how useful this approach is. Instead of identifying a root cause, they list many contributing factors, all of which can be improved to raise the level of safety significantly rather the fixing a single root cause. As a recent example, think of the Boeing 737 MAX. What was the "root cause"? Was it: * A software bug in the MCAS system? * The removal of a redundant method for sensing angle of attack? * The decision to make the "AoA Disagree" light an optional upgrade? * Making substantially unstable changes to the existing airframe? * The desire for it be be "another 737 variant" for pilot classification? * The insufficient pilot training w.r.t. to the new MCAS system? * Organizational pressures to not lose the AA contract to Airbus? * Executive leadership that prioritized business over safety? The answer is all of the above, and more! By identifying all of these past "a software bug in the MCAS system" we have the opportunity to fix many safety issues which could be contributing factors to future failures. There is an even more in depth example of this in Chapter 5 of Nancy Leveson's book Engineering a Safer World. She spends over 60 pages describing the environment and circumstances that lead to a US Air Force F-15 shooting down a friendly US Army Black Hawk over northern Iraq in 1994.

afarrell 6y ago

For those looking for ways to talk to non-engineers about this topic (without them rolling their eyes and telling you to stop making excuses), I highly recommend the book Leadership is Language.

master_yoda_1 6y ago

You forget about Lithium ion battery and nuclear fusion and many more examples. All the problems can't be solved by simple solution you need to do complex things.

kaiomagalhaes 6y ago
The article doesn't say we can't have complex systems. It simply states they are doomed to fail in some way, and that we need to be prepared for that
- master_yoda_1 6y ago
  That is not true. Nuclear reactor are far more complex and not doomed to fail one day. So stop the bullshit.
- master_yoda_1 6y ago
  I think by complex system you mean a shitty system build by jokers which is doomed to fail. You can’t just build any bullshit and called it complex.

jwdomb 6y ago

This reads like a crisp summary of a subset of John Gall's "Systemantics": https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics

adrianmonk 6y ago

I would modify the section on root causes. It's not a fundamentally bad idea to look for root causes (plural). It's essential to not stop at immediate causes. More or less, you start at immediate causes and then recursively go deeper until you get to things that cannot be changed and/or are not within your control. One real danger is believing in a root cause or the root cause (singular) as if there is only one. The desire to only have to change (or understand) one thing is dangerous because it leads you to stop investigating/analyzing as soon as you have found something to blame it on. As you go deeper through the chain(s) of causes, you also should not focus exclusively on the last (root) because that means you aren't thinking about defense in depth. TLDR, follow the chain of causes all the way to its ends, realize it may have multiple ends, and don't focus only on the surface stuff or only on the root stuff.

di4na 6y ago
You are making the assumption that causality is useful and exist in these complex systems. Dr Cook base his treaty on what we know from the research of them, which is that these causal links are far less stronger than you think. This means that searching along a causal tree is indeed detrimental fundamentally.
- adrianmonk 6y ago
  I'll give an example to illustrate. Situation: my software service stops working because it ran out of disk space. Insufficient focus on root cause: I add more disk, do nothing else, and assume it's solved. Excessive focus on root cause only: I investigate, find there's a leak where old data never gets deleted, call it the root cause, fix the leak, and do nothing else. Balanced approach: I fix the leak AND stop running at such tight margins (allocate breathing room) AND add automated monitoring for low disk space. The lesson to look for the root cause is valuable. Without it, disk space fills up rapidly and my other measures to insulate against that can only go so far. But if it's all I do, that's also bad.

deleted 6y ago

[deleted]

somurzakov 6y ago

>>Safety is a characteristic of systems and not of their components >>Safety is an emergent property of systems; it does not reside in a person, device or department of an organization or system. Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system. This means that safety cannot be manipulated like a feedstock or raw material. The state of safety in any system is always dynamic; continuous systemic change insures that hazard and its management are constantly changing. fits perfectly the description of Security of complex IT systems, nice way to explain why IT security marketplace is a wild wild west

marnett 6y ago

Is this the primary source of this https://fs.blog/2014/04/how-complex-systems-fail/ https://fs.blog/2014/04/how-complex-systems-fail/ ?

di4na 6y ago
As this is a copy of the paper, probably. I would point out that Dr Cook would probably disagree greatly with the whole idea of AntiFragile, or at least has the few time i have been able to talk with him. This link you offer is probably not the best context in which to comment on this paper.
jeffreygoesto 6y ago
The Morning Paper [0] has a nice discussion and quotes the original paper from 2000. The same link is at the very bottom of the post, behind "original". However, it seems that MIT rearranged something and the link has gone stale. Maybe someone at MIT could make it appear under the original URL [1] again? [0] https://blog.acolyer.org/2016/02/10/how-complex-systems-fail/ https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... [1] http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...

crdrost 6y ago

This is a wonderful article, but I feel like it is kind of missing a certain depth. Read Donella Meadows’ classic Thinking in Systems for instance. So like, yes, of necessity the largeness of a system causes the existence of feedback loops such that it maintains an equilibrium state, a homeostasis. This is kind of just a restatement of what a system is, and to characterize it as “the system is constantly failing,” while that is totally true, seems a little bit sensationalist. Like, you are one of these complex systems, do I say that you are constantly failing to draw breath? No, but I imagine some of your alveolar sacs routinely fail to fully inflate and others are too covered with mucous to absorb oxygen and whatever else, and often your breath runs ragged. Similarly with “you cannot identify a root cause,” a kid dies in a pool and we routinely have a medical examiner pronounce that the cause of death was drowning, which in the respiratory sense can be fairly well isolated to a root cause, “could not breathe air because no air was supplied by the system upstream.” The failure to find a root cause essentially only becomes hard because there is another aspect of complex systems which Meadows discusses, which is that they bleed into each other and into the World At Large, they do not have clearly defined boundaries until we impose them for analysis. Draw the boundaries larger than the respiratory system, say the social system, and now the reason that the kid died has something to do with parents never having been trained on what drowning actually looks like, so they heard some light splashing and looked over and said “aww isn’t he enjoying the water” not realizing that this was actually a “holy fuck my son is struggling to keep his mouth above water” moment—this perhaps speaks to a larger problem of not having good water education in society, maybe that is the root cause. Stuff like that. In that sense yes there are failures at all levels, but at any given level of explanation byou can more or less say that this is the thing that at that level explains the problem. So my servers go down, there is a valid root cause at the level of the individual server; there is also one at the level of the cluster as a whole and what that was doing; there is a valid cause at our developer and operations practices; there is a different cause at the level of company culture too. I don’t think that it makes sense to say that there is no root cause. The root cause of the Challenger explosion was that it was too cold to fly that day and they still flew the shuttle anyway. One can zoom in (“the fuel tanks were supposed to be sealed by rubber rings, but at the cold temperature of launch day the rubber no longer sealed the tanks”) or out (“NASA leadership built a false confidence in the quality of their work bolstered by a sort of management double-speak that disconnected them from the real risks they were playing with, making them foolhardy in ordering the launch”) and one can even sort of pan sideways (“why was the weather so cold that day?” or “what financial and engineering considerations made them choose this construction with these o-rings, several years ago?”). But this “middle explanation” of “it was too cold to fly that day and they flew the shuttle anyway” is a good summary that gives an entry-point to the problem, which is the real function of a root cause analysis (you want to collect information about how to fix the problem and know whether it was fixed). The fact that this is a little bit over-sensational detracts a little from my enjoyment but I do think that the heart of the article is great and fun to think about. If folks like thinking about this stuff I really recommend Meadows's book though, systems thinking is not present in most curricula and that is a real shame.

bitminer 6y ago

>>[The] defenses include obvious technical components (e.g. backup systems, ‘safety’ features of equipment) and human components (e.g. training, knowledge) but also a variety of organizational, institutional, and regulatory defenses (e.g. policies and procedures, certification, work rules, team training). This omits "design" for defenses against problems. Example: the chemical industries in many countries in the 1960s had horrendous accident records: many employees were dying on the job. (For many reasons) the owners re-engineered their plants to substantially reduce overall accident rates. "Days since a lost-time accident" became a key performance indicator. A key engineering process was introduced: HAZOP. The chemical flows were evaluated under all conditions: full-on, full-stop, and any other situations contrary to the design. Hazards from equipment failures or operational mistakes are thus identified and the design is adjusted to mitigate them. This was s.o.p. in the 1980s. See Wikipedia for an intro. Similar approaches could help IT and other systems.

treelovinhippie 6y ago

Complex system: nature. Complicated system: things mentioned in this article.

victor106 6y ago

We implemented a Microservices architecture that was promised and looked to be so simple on paper but became exponentially complex as each domain was implemented so we went back to a simpler monolithic service and it was much simpler to operant manage.

ngneer 6y ago

Great post. Uncanny how much of it maps almost 1:1 to security, which is too an emergent property of a complex system.

dang 6y ago

If curious see also 2019 https://news.ycombinator.com/item?id=20380055 https://news.ycombinator.com/item?id=20380055 2017 https://news.ycombinator.com/item?id=15002683 https://news.ycombinator.com/item?id=15002683 2017 https://news.ycombinator.com/item?id=14127543 https://news.ycombinator.com/item?id=14127543 2014: https://news.ycombinator.com/item?id=8282923 https://news.ycombinator.com/item?id=8282923

ineedasername 6y ago

"Complex systems run in degraded mode." This is the point that struck me the most. "Good enough" is good enough when there are so many other more urgent needs to address and you've been able to keep a system functioning despite extensive deferred maintenance. It is difficult to get allocation of resources to a system that's still working because it's robust when there are less robust (and probably less critical) systems that won't work without more resources being thrown at it.

b0ner_t0ner 6y ago

Slightly related, .fail TLDs were delegated back in 2014: https://icannwiki.org/.fail https://icannwiki.org/.fail

edrobap 6y ago

There is a remarkable resemblance between complex systems and hacks, except for point 7 and 8 that are trivial in case of latter. Perhaps most complex systems are hack that have been around for long enough to make everyone believe that they are well thought of to be called systems.