4 ms·

There are many accidents in aviation that show that 99% reliable automation kills. It is just reliable enough that you don’t expect it to fail, and you are not

There are many accidents in aviation that show that 99% reliable automation kills. It is just reliable enough that you don’t expect it to fail, and you are not ready to handle if it does. This is why self driving cars of today, that claim that humans should take over in case of failure are nonsense. We have a huge body of knowledge showing that when faced with a system that almost always works, humans suck at overseeing it and taking over when needed.

throwawaylinux 4y ago
This case was perfectly reliable automation. The automated systems behaved exactly as designed. The point of the article isn't that the automation just needs to be made more reliable and that will solve the problem, it's that systems like this are becoming too complicated to understand all aspects about how they work. You would have thought overriding the TOGA switches should have sounded an immediate alarm though. It seems perfectly reasonable that a very dangerous command or input would be inhibited by the automatic systems, but it seems completely crazy that any command or input would ever be inhibited silently. I'm actually flabbergasted that this is not a fundamental rule of aircraft control systems design at Boeing.
- tass 4y ago
  Another poster pointed out that the alarm might also fail to sound, or maybe the switch didn’t actually work, so you need to check it anyway. On Boeing planes, many automations such as TOGO typically have a corresponding physical counterpart - in this case, autothrottle slides the throttle controls upwards to indicate that the engines were asked to go to full power. There’s also the engine instruments which show the engines reaction to the requested throttle level. It’s not hard to know that it didn’t set the throttle to full power, but the pilots didn’t check or didn’t know to check. An alarm would help, but would also be another fallible automation to be relied upon.
  - throwawaylinux 4y ago
    I appreciate there would be indicators to confirm most things, and procedure probably says those checks should be done and alarms should not be relied upon. I still think it's crazy that there aren't alarms for any situation where the airplane disregards an input like this.
jacquesm 4y ago
And invariably: that 1% triggers when you need it to work the most.

throwawaylinux 4y ago

This case was perfectly reliable automation. The automated systems behaved exactly as designed. The point of the article isn't that the automation just needs to be made more reliable and that will solve the problem, it's that systems like this are becoming too complicated to understand all aspects about how they work. You would have thought overriding the TOGA switches should have sounded an immediate alarm though. It seems perfectly reasonable that a very dangerous command or input would be inhibited by the automatic systems, but it seems completely crazy that any command or input would ever be inhibited silently. I'm actually flabbergasted that this is not a fundamental rule of aircraft control systems design at Boeing.

tass 4y ago
Another poster pointed out that the alarm might also fail to sound, or maybe the switch didn’t actually work, so you need to check it anyway. On Boeing planes, many automations such as TOGO typically have a corresponding physical counterpart - in this case, autothrottle slides the throttle controls upwards to indicate that the engines were asked to go to full power. There’s also the engine instruments which show the engines reaction to the requested throttle level. It’s not hard to know that it didn’t set the throttle to full power, but the pilots didn’t check or didn’t know to check. An alarm would help, but would also be another fallible automation to be relied upon.
- throwawaylinux 4y ago
  I appreciate there would be indicators to confirm most things, and procedure probably says those checks should be done and alarms should not be relied upon. I still think it's crazy that there aren't alarms for any situation where the airplane disregards an input like this.

jacquesm 4y ago

And invariably: that 1% triggers when you need it to work the most.