4 ms·

In The Field Guide to Human Error Investigations Dekker talks about how “number of minor incidents” correlates inversely with “number of fatal incidents” for ai

In The Field Guide to Human Error Investigations Dekker talks about how “number of minor incidents” correlates inversely with “number of fatal incidents” for airlines (scaled per flight hour or whatever). I have forgotten whether this was all airlines or Western only. I wonder if it still holds. The rest of the book is also quite a good read including a fun take on Murphy’s Law that goes “Everything that can go wrong will go right” which is the basis for normalization of deviance: where a group of people (an org whatever) slowly shifts from their performance metrics as they “get away with it”. I wonder how modern organizations fight this. Most critically I imagine warfighting ability can experience massive multipliers based on adherence to design. But also civilian performance to a lesser extent (outcome is often less catastrophically binary). Anyway, I got a lot of mileage out of the safety book wrt software engineering.

aqueueaqueue 2y ago
Are chaos monkeys relevant here. And at some level: testing! You definitely find more severe issues of you actually test stuff. In production.
AlotOfReading 2y ago
Safety literature focuses on straightforward ways to build highly reliable systems involving humans. That applies to almost everyone on HN. What's funny is that the suggestions are usually pretty "common sense". You already know most of the information in Sidney Dekker's books and the NASA guidelines. They're essentially the same principles we all like to see in code. Things like: Consider the human factors. Make doing the right thing easy. Make wrong things obvious. Trust, but verify. Keep the signal to noise ratio high in communication. Etc.
- terribleperson 2y ago
  One big lesson, I think (at least as flying does safety), is that even if the ideas are common sense, you have to take the common sense and turn it into rules and processes, and make sure people stick to those. And if it's a role where time matters, you have to drill those so people stick to them when seconds count.
  - AlotOfReading 2y ago
    There's a lot of value in the books. Applying "common sense" consistently and intentionally is extremely difficult. It's not a deep magic entirely divorced from the practices we already know and agree with, but rather the organizational equivalent of going to the gym.

aqueueaqueue 2y ago

Are chaos monkeys relevant here. And at some level: testing! You definitely find more severe issues of you actually test stuff. In production.

AlotOfReading 2y ago

Safety literature focuses on straightforward ways to build highly reliable systems involving humans. That applies to almost everyone on HN. What's funny is that the suggestions are usually pretty "common sense". You already know most of the information in Sidney Dekker's books and the NASA guidelines. They're essentially the same principles we all like to see in code. Things like: Consider the human factors. Make doing the right thing easy. Make wrong things obvious. Trust, but verify. Keep the signal to noise ratio high in communication. Etc.

terribleperson 2y ago
One big lesson, I think (at least as flying does safety), is that even if the ideas are common sense, you have to take the common sense and turn it into rules and processes, and make sure people stick to those. And if it's a role where time matters, you have to drill those so people stick to them when seconds count.
- AlotOfReading 2y ago
  There's a lot of value in the books. Applying "common sense" consistently and intentionally is extremely difficult. It's not a deep magic entirely divorced from the practices we already know and agree with, but rather the organizational equivalent of going to the gym.