4 ms·

A lot of this is based on heavy assumptions about systems and risk/safety analysis. The biggest assumption this post is making is that humans should be involved

A lot of this is based on heavy assumptions about systems and risk/safety analysis. The biggest assumption this post is making is that humans should be involved at all. Systems do not have to facilitate operators in building accurate mental models. In fact, safe systems disregard mental models, because a mental model is a human thing, and humans are fallible. Remove the human and you have a safer system. Safety is not a dynamic control problem, it's a quality-state management problem. You need to maintain a state of quality assurance/quality control to ensure safety. When it falters, so does safety. Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Also fwiw, there's often not a root cause, but instead multiple causes (or coincidental states). For getting better at tracking down causes of failures (and preventing them), I recommend learning the Toyota Production System, then reading Out Of The Crisis. That'll kickstart your brain enough to be more effective than 99.99% of people.

kqr 1y ago
> mental model It should be more clear in the article that this term is used more broadly. The "mental model" is the function that converts feedback into control actions. In that sense, even simple automated controllers have "mental models". > Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Sometimes it isn't but often it is. Yes, that makes the system more complex, along with other properties such as organicism, non-linearity, parallelism, interactivity, long-term recurrences, etc. These properties are inconvenient but we cannot just wish them away. We have to design for the system we have. > Toyota, Deming, SPC I have read more books in this area than most people, and I don't really see how you came away without an appreciation for the importance of humans in the loop, and the dangers of overautomation. Could you illustrate more clearly what you mean?

kqr 1y ago

> mental model It should be more clear in the article that this term is used more broadly. The "mental model" is the function that converts feedback into control actions. In that sense, even simple automated controllers have "mental models". > Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Sometimes it isn't but often it is. Yes, that makes the system more complex, along with other properties such as organicism, non-linearity, parallelism, interactivity, long-term recurrences, etc. These properties are inconvenient but we cannot just wish them away. We have to design for the system we have. > Toyota, Deming, SPC I have read more books in this area than most people, and I don't really see how you came away without an appreciation for the importance of humans in the loop, and the dangers of overautomation. Could you illustrate more clearly what you mean?