4 ms·

If you are interested in this topic, worth reading Charles Perrow's classic work, "Normal Accidents". The book is an analysis of industrial accidents (eg Three

by pge 6y ago

If you are interested in this topic, worth reading Charles Perrow's classic work, "Normal Accidents". The book is an analysis of industrial accidents (eg Three Mile Island) and their root causes. His observations suggest that correctly mapping the left side of a bowtie is virtually impossible in tightly coupled systems (ie. the possible pathways to a failure mode are too complex to be understood ahead of time). Mapping risk incorrectly can provide a dangerously false sense of confidence (perhaps a reference to Black Swans is in order here, but those were popularized 30 years after Perrow)

thetwentyone 6y ago
Interesting recommendation. I've been wanting to read something that talks about important failures and the root/structural causes. E.g. I loved reading into the Challenger disaster and the follow-up investigations and was interested in similar types of events to learn from. Do you think that book is along those lines or something different?
- pge 6y ago
  yes - Perrow spends much of the book (too much, perhaps) on case studies, with Three Mile Island being the most significant.
PaulAJ 6y ago
Another view which addresses the complexity is STAMP (Systems-Theoretic Accident Model and. Processes) by Nancy Leveson. Rather than look at scenarios she sees a system as a set of nested control loops, where each loop involves two-way information flow between a system and a controller. STAMP analyses the information flow looking for messages which might cause erroneous behaviour if they are wrong. This is based on the observation that in many system failures every component is found to have operated within its specification, but the system still failed. The 737-MAX is a good example; the MCAS did exactly what it was supposed to, given its inputs. Even the failed sensor was within spec, in the sense that the fleet-wide sensor failure rate no higher than expected. The problem wasn't a component failure, it was the interaction between a system (the airframe) and a controller (MCAS) once erroneous information started to flow. This was then exacerbated by the next control level up, where information about the MCAS failed to flow to the pilot who was controlling it.

by pge 6y ago

thetwentyone 6y ago

Interesting recommendation. I've been wanting to read something that talks about important failures and the root/structural causes. E.g. I loved reading into the Challenger disaster and the follow-up investigations and was interested in similar types of events to learn from. Do you think that book is along those lines or something different?

pge 6y ago
yes - Perrow spends much of the book (too much, perhaps) on case studies, with Three Mile Island being the most significant.

PaulAJ 6y ago

Another view which addresses the complexity is STAMP (Systems-Theoretic Accident Model and. Processes) by Nancy Leveson. Rather than look at scenarios she sees a system as a set of nested control loops, where each loop involves two-way information flow between a system and a controller. STAMP analyses the information flow looking for messages which might cause erroneous behaviour if they are wrong. This is based on the observation that in many system failures every component is found to have operated within its specification, but the system still failed. The 737-MAX is a good example; the MCAS did exactly what it was supposed to, given its inputs. Even the failed sensor was within spec, in the sense that the fleet-wide sensor failure rate no higher than expected. The problem wasn't a component failure, it was the interaction between a system (the airframe) and a controller (MCAS) once erroneous information started to flow. This was then exacerbated by the next control level up, where information about the MCAS failed to flow to the pilot who was controlling it.