2 ms·
This reminds me of a link to a talk by Nancy Leveson that a guy linked to while taking Uncle Bob behind the woodshed for a shellacking because of some ignorant
by Gibbon1 8y ago
This reminds me of a link to a talk by Nancy Leveson that a guy linked to while taking Uncle Bob behind the woodshed for a shellacking because of some ignorant things he said about software safety.
https://www.youtube.com/watch?v=WBktiCyPLo4 https://www.youtube.com/watch?v=WBktiCyPLo4
If you don't want to watch an hour long video, she says most serious accidents and near misses happen not because something failed (and lets face it software 'bugs' are just another type of failure). But because bad interactions between properly functioning systems[1].
One of her comments is you need a system that watches for stuff like that. Lion Air Flight 610 crash happened because there wasn't a system that could make a determination that the pilots needed hard control over the aircraft. A safety system being repeatably overridden by the pilots for 11 minutes should have resulted in that system being shut down.
[1] One thing I got beat into me over the last 20 years developing sensor networks is how little information is actually available to systems like a MCAS system. That makes it's reasoning about the world brittle. Humans are better at managing path dependent state based on inane sources of information. The pilots utterly knew the aircraft wasn't stalling just by feel. While the MCAS 'knew' it was based on a messed up sensor.
- Yetanfou 8y ago> The pilots utterly knew the aircraft wasn't stalling just by feel. While the MCAS 'knew' it was based on a messed up sensor. ...except when the pilots get it wrong like they did on AF447 which crashed into the Atlantic Ocean when 'the crew failed to recognize the aircraft had stalled and consequently did not make inputs that would have made it possible to recover from the stall.' [1]. That accident was caused by a combination of equipment failure - iced up Pitot tubes - in combination with an erroneous reaction by the pilots. [1] https://en.wikipedia.org/wiki/Air_France_Flight_447 https://en.wikipedia.org/wiki/Air_France_Flight_447
- Gibbon1 8y agoAF447 crashed because the auto pilot handed control of the airplane over to the pilots when it was in the coffin corner with iced over pitot tubes[1]. And then stall warning system gave them very confusing information. It's not really the same situation. In one the aircraft was absolutely flyable without instruments. The other it was not flyable without instruments. [1] Absolutely last thing you want in that situation.