6 ms·

I am a designer and implementor of vital speed and distance measurement systems, with triply modular redundancy, used in mass transit application. I cannot eve

by crocal 8y ago

I am a designer and implementor of vital speed and distance measurement systems, with triply modular redundancy, used in mass transit application. I cannot even imagine what the designers of this thing are going through now. It must be terrible. To make it worse, it's a confusing topic. There are two pillars to the design of such system. 1. Faulty sensor must be detected with a very high probability. The typical way to achieve that is redundancy and diversification. The exact amount of redundancy depends on the reliability of the sensor considered. In most cases, it is sufficient to have 2 sensors, but of different models, in order to avoid common mode of failure. 2. In case of a failure, the system must have a graceful degradation, and in aeronautics, this means a clean handover to the pilot. So, in the case of this MCAS thing, having two sensors is not necessarily a bad thing, and what M. Kornecki reports is 100% correct. What looks strange is the way a single failure was managed by the software, and how the procedure to recover was quite complicated and, even worse, not exported to the training material of the pilots. In my world, we called this "exported safety requirement application conditions" - SRACs, and verification of their proper allocation is a big chunk of the safety case. More than discussing architecture, in my view, the investigation must explain why the organisation failed to perform this activity. The case for a third sensor can be made to decrease the likelihood of having to bypass the system ("belt and suspenders", as says M. Kornacki), but based on my experience, it will not be sufficient. As other correctly report here, it's not a silver bullet. Ultimately, the degradation scenario and pilot handover is part of the overall system safety.

devit 8y ago
How expensive are these sensors compared to the cost of the plane? Any reason to not just use at least 9 of them, pick the median value and forget about it?
- consto 8y ago
  It's incredibly unlikely, but not entirely unforseeable that enough sensors would fail at the same time for the same reason. All four engines on the 747 BA9 failed at the same time due to volcanic ash [0]. If you just take the median of nine, it's possible that a mere five failed sensors could break MCAS. And then you are back to the issue of handing over to the pilots. [0] https://en.m.wikipedia.org/wiki/British_Airways_Flight_9 https://en.m.wikipedia.org/wiki/British_Airways_Flight_9
- crocal 8y ago
  Yes there is. If you put too many sensors then there is always one in need of repair. And then it becomes dangerous: the pilot and maintainer ignores the alarm, delay replacement, leading slowly to a failure. This is why the redundancy choices and availability calculations should always be made pondering the maintenance burden.
  - iSnow 8y ago
    Well, I've been working in mass mobility and they had 4x redundancy with different sensor types. The more sensors went bad, the more restricted the operation regime became as it went through degraded modes. And the on-board systems were of course aware of the status of the sensors. I think that created the right incentives: you could ignore a faulty sensor or two, but the results were increased travel times, so costs to the operator. I always wonder why we don't have massive arrays of micro-sensors on planes spread out over the plane. If a handful your 65k sensors start to fail, you can delay repair for quite some time - otoh, if sensors start to fail all over the plane, you know you have to wake up the pilot hard, since it's a systemic problem like icing.
    - madengr 8y ago
      Wiring harnesses are a major weight savings. If you wired them all in serial, then that wire is a single point of failure. Otherwise you’d have 65k wires going to each one.
      - magduf 8y ago
        No, you put them on a bus like so many other things in modern vehicles today. They don't need their own separate wires or even their own bus. With lots of sensors, you're going to have several independent buses anyway, so a failure on that bus would only make you lose a fraction of your sensors.
nindalf 8y ago
> even worse, not exported to the training material of the pilots According to this article (https://www.vox.com/business-and-finance/2019/3/29/18281270/737-max-faa-scandal-explained https://www.vox.com/business-and-finance/2019/3/29/18281270/...), this was a requirement of the 737-MAX while it was being developed. It needed to be sold as pretty much the same as it's predecessors, therefore requiring minimal retraining. > The training piece is important because a key selling feature of the 737 Max was the idea that since it wasn’t really a new plane, pilots didn’t really need to be retrained for the new equipment. As the New York Times reported, “For many new airplane models, pilots train for hours on giant, multimillion-dollar machines, on-the-ground versions of cockpits that mimic the flying experience and teach them new features” while the experienced 737 Max pilots were allowed light refresher courses that you could do on an iPad. > That let Boeing get the planes into customers’ hands quickly and cheaply, but evidently at the cost of increasing the possibility of pilots not really knowing how to handle the planes, with dire consequences for everyone involved.
- trhway 8y ago
  >at the cost of increasing the possibility of pilots not really knowing how to handle the planes not excusing Boeing here, just extrapolating the situation to where there are so many automated systems doing so much and such complicated stuff that handover to pilot is just not an option (especially if plane is designed to fully utilize the automated systems capabilities). Like F-16 which just can't flown manually, or autonomous vehicles (Uber/Tesla/ect.) where handover to human driver in reality is absolutely not option. I mean, look how we completely lost the ability to stick-shift the cars.
crististm 8y ago
I find it surprising when a system is considered "redundant" when the _pilot_ is part of redundancy!? If this is an accepted solution then it is very easy to be abused: sensors fail, never mind, pilot has eyes and can deactivate a malfunctioning _whatever_.
- crocal 8y ago
  Depending on the mode of operation, the pilot will or will not be part of the redundancy. The trick in this type of application is the management of mode transitions. When sensors fail, the system will transition to degraded modes of operations where the pilot will have more responsibility. The safety case will rely on demonstrating that transition to such degraded modes is low enough to be compatible with the human rate of errors. A modern system should not be declared safe if it relies on pilot _whatever_. Not sayin’ it did not happen in the past. Unfortunately.
  - crististm 8y ago
    I don't understand (all) your points. I'm questioning whether a system as the one I described can be abused to the point where redundancy is offloaded to pilot flying the plane even when it shouldn't be?
    - crocal 8y ago
      It can be abused, but it should not be possible if the process driving certification of such system is followed properly. The standards define checks and balances designed precisely to detect such abuse. In particular, the pilot’s rate of error is quantified and it should not be possible to abuse it without going unnoticed.
oldjokes 8y ago
My car is only five years old but already has two sensors that have gone bad in the engine. Neither of them are vital and everything works great despite the occasional check engine light. Redundancy. It's a time intensive process to pop the head off and replace sensors. Nobody wants to pay their mechanic $100 an hour if they can avoid it. I can't even imagine a sensor failure having zero redundancy, much less changing control inputs to a complicated system like a multi-engine jet. This is completely against everything I know about engineering and I find even discussion of this being possible to be deeply offensive. The fact that it happened twice and hundreds of people are dead is just mind boggling.
- pytyper2 8y ago
  "...everything works great despite the occasional check engine light..." that may make you feel better but it's not true. The life and efficiency of your engine is being reduced. What year, make, model are you driving? When you sell it do you plan on telling the next owner that you drove it religiously with the check engine light on?
  - oldjokes 8y ago
    The engine isn't magic, you can read the codes yourself, figure out what they mean, look up the replacement guidelines yourself, etc. It's really simple actually, anyone can do it.
- linuxftw 8y ago
  Failed sensors can lead to premature failure of various components, especially catalytic converters: http://ricksfreeautorepairadvice.com/causes-catalytic-converter-go-bad/ http://ricksfreeautorepairadvice.com/causes-catalytic-conver... Don't put off a $150 job and cost yourself a $2k job. Most sensor replacements don't require removing the cylinder heads.
  - oldjokes 8y ago
    You're assuming I don't know what the codes are and that they are potentially harmful, neither of which is true.
jordache 8y ago
>I cannot even imagine what the designers of this thing are going through now. It must be terrible. Their poor decisions led to this. Single sensor as input to autopilot to augment the nose!?
- Aromasin 8y ago
  The turn-over of employees in aerospace is often quite quick. It's very likely that most of the people that designed this system are long gone, and the current design team are the ones picking up the pieces.
  - jordache 8y ago
    Then they shouldn't have any personal guilt..
    - baddox 8y ago
      Guilt isn’t the only emotion that can be described as terrible.
Cacti 8y ago
My understanding is that the recovery scenario wasn’t supposed to be complicated at all, because in the event of a MCAS failure, it would always manifest as a runaway trim scenario, and that pilots are trained heavily on recognizing as correcting for that scenario, and that by following the standard runaway trim solution/checklist/process that this would necessarily involve disabling MCAS whether the pilots were aware of it or not. So the question is, did they, and if so did this fail, and if not, why didn’t they? Is this actually a direct problem with MCAS or is it closer to an issue of basic pilot training that only happened to involve MCAS? I suspect as the investigations unfold that our understanding of the accidents is only going to get more complex and nuanced.
- ncallaway 8y ago
  One significant problem as I understand it was the assumption that a failing MCAS would present as runaway stabilizer trim. When, in actuality, it may not have looked much like runaway stabilizer trim to the pilots. My understanding is in a runaway stabilizer trim situation the trim wheel (a medium sized wheel near the pilot and co-pilots knees) spins continuously and makes a loud clacking noise as part of the spinning. This makes runaway stabilizer trim easy to identify. The wheel is spinning (for no identifiable reason) and making noise. The MCAS failure, however applies trim discretely with intervals of 10-20s. This means it might be very hard to spot the adjustments to the trim wheel if you glance down briefly. A pilot might notice the control column is heavy, glance down to see if it's runaway trim see the wheel not moving, rule out that scenario and move on to other diagnostic steps.
  - AceyMan 8y ago
    NB. It seems this moment news is breaking that the Ethiopia pilots indeed followed the checklist — yet they still lost the airplane. This is getting worse for Boeing at every turn.
    - Cacti 8y ago
      Well, it sounds like they turned off the autopilot but then turned it back on, which is not part of the checklist. It’s not clear from the reporting exactly what happened and I would hesitate to jump to conclusions. Accidents like this are almost always hugely complicated and are the result of many things going wrong in a specific order at a specific time, rather than a single factor. In fact we don’t even know what the cause of the crash was, and we won’t actually know for several months—there have been a number of crashes where everyone “knew” what the cause was early on, only for it later to come to light that is was something completely unrelated.
sixothree 8y ago
Are AOA sensors really so prone to failure that they caused the downing of two airplanes?
- Xixi 8y ago
  Like all sensors out there in the atmosphere, icing is an issue. So not only are they error prone, but failures are sometimes not independent. Airbus planes from the A320 family have three such sensors, but it so happened that on November 5th, 2014, an A321 had two sensors freeze in the wrong position during flight. Two sensors against one win, so the flight envelope protection system kicked in and the aircraft started to nose dive. It lost 4000 feet in altitude before the crew regained control... Details of the flight if you want to do some research: Lufthansa Airbus A321-200, registration D-AIDP, performing flight LH-1829 from Bilbao, SP (Spain) to Munich (Germany) on November 5th, 2014.
- ams6110 8y ago
  Normally no, from what I have read they are very reliable. And the sensors are the same ones used on all 737s. So it seems that there is some other problem with the MAX that is leading to the a false "nose high" indication.
- linuxftw 8y ago
  I wouldn't think so. My personal theory is AoA is sending valid data to one of the flight computers, that flight computer is sending invalid data to the MCAS, or the MCAS has a software problem. I suspect AoA is sending signed data, computer expects unsigned data, or possibly LSB/MSB conversion problem. Or it could just be faulty logic somewhere.
  - salawat 8y ago
    The sensors on Lion Air were 10+ degrees out of calibratiom IIRC. Critical AoA is somewhere in the realm of 17 to 20 degrees. That doesn't leave much room for where MCAS wouldn't be a problem. Also, AoA sensors are not normally safety critical. The MAX is the first 737 airframe for which this is the case.
hbarka 8y ago
I enjoyed your comment because I’m particularly interested in Tesla’s Autopilot system and your mention of “graceful degradation and clean handover” sounds like it should apply to driving autonomy levels as well.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- wjnc 8y ago
  In the way we use cars there seems no time for graceful handover. You're about the same time interval away from the next car as the time to gain situational awareness, probably less. A plane is usually quite some minutes away from the ground and other planes, or f.e. when landing on autopilot the pilots are not distracted. If you'd apply this standard, the bar for autopilot in cars would be very high indeed. (That might be warranted, although in a risk / reward perspective humans are terrible drivers too.)

by crocal 8y ago

devit 8y ago

How expensive are these sensors compared to the cost of the plane? Any reason to not just use at least 9 of them, pick the median value and forget about it?

consto 8y ago
It's incredibly unlikely, but not entirely unforseeable that enough sensors would fail at the same time for the same reason. All four engines on the 747 BA9 failed at the same time due to volcanic ash [0]. If you just take the median of nine, it's possible that a mere five failed sensors could break MCAS. And then you are back to the issue of handing over to the pilots. [0] https://en.m.wikipedia.org/wiki/British_Airways_Flight_9 https://en.m.wikipedia.org/wiki/British_Airways_Flight_9
crocal 8y ago
Yes there is. If you put too many sensors then there is always one in need of repair. And then it becomes dangerous: the pilot and maintainer ignores the alarm, delay replacement, leading slowly to a failure. This is why the redundancy choices and availability calculations should always be made pondering the maintenance burden.
- iSnow 8y ago
  Well, I've been working in mass mobility and they had 4x redundancy with different sensor types. The more sensors went bad, the more restricted the operation regime became as it went through degraded modes. And the on-board systems were of course aware of the status of the sensors. I think that created the right incentives: you could ignore a faulty sensor or two, but the results were increased travel times, so costs to the operator. I always wonder why we don't have massive arrays of micro-sensors on planes spread out over the plane. If a handful your 65k sensors start to fail, you can delay repair for quite some time - otoh, if sensors start to fail all over the plane, you know you have to wake up the pilot hard, since it's a systemic problem like icing.
  - madengr 8y ago
    Wiring harnesses are a major weight savings. If you wired them all in serial, then that wire is a single point of failure. Otherwise you’d have 65k wires going to each one.
    - magduf 8y ago
      No, you put them on a bus like so many other things in modern vehicles today. They don't need their own separate wires or even their own bus. With lots of sensors, you're going to have several independent buses anyway, so a failure on that bus would only make you lose a fraction of your sensors.

nindalf 8y ago

> even worse, not exported to the training material of the pilots According to this article (https://www.vox.com/business-and-finance/2019/3/29/18281270/737-max-faa-scandal-explained https://www.vox.com/business-and-finance/2019/3/29/18281270/...), this was a requirement of the 737-MAX while it was being developed. It needed to be sold as pretty much the same as it's predecessors, therefore requiring minimal retraining. > The training piece is important because a key selling feature of the 737 Max was the idea that since it wasn’t really a new plane, pilots didn’t really need to be retrained for the new equipment. As the New York Times reported, “For many new airplane models, pilots train for hours on giant, multimillion-dollar machines, on-the-ground versions of cockpits that mimic the flying experience and teach them new features” while the experienced 737 Max pilots were allowed light refresher courses that you could do on an iPad. > That let Boeing get the planes into customers’ hands quickly and cheaply, but evidently at the cost of increasing the possibility of pilots not really knowing how to handle the planes, with dire consequences for everyone involved.

trhway 8y ago
>at the cost of increasing the possibility of pilots not really knowing how to handle the planes not excusing Boeing here, just extrapolating the situation to where there are so many automated systems doing so much and such complicated stuff that handover to pilot is just not an option (especially if plane is designed to fully utilize the automated systems capabilities). Like F-16 which just can't flown manually, or autonomous vehicles (Uber/Tesla/ect.) where handover to human driver in reality is absolutely not option. I mean, look how we completely lost the ability to stick-shift the cars.

crististm 8y ago

I find it surprising when a system is considered "redundant" when the _pilot_ is part of redundancy!? If this is an accepted solution then it is very easy to be abused: sensors fail, never mind, pilot has eyes and can deactivate a malfunctioning _whatever_.

crocal 8y ago
Depending on the mode of operation, the pilot will or will not be part of the redundancy. The trick in this type of application is the management of mode transitions. When sensors fail, the system will transition to degraded modes of operations where the pilot will have more responsibility. The safety case will rely on demonstrating that transition to such degraded modes is low enough to be compatible with the human rate of errors. A modern system should not be declared safe if it relies on pilot _whatever_. Not sayin’ it did not happen in the past. Unfortunately.
- crististm 8y ago
  I don't understand (all) your points. I'm questioning whether a system as the one I described can be abused to the point where redundancy is offloaded to pilot flying the plane even when it shouldn't be?
  - crocal 8y ago
    It can be abused, but it should not be possible if the process driving certification of such system is followed properly. The standards define checks and balances designed precisely to detect such abuse. In particular, the pilot’s rate of error is quantified and it should not be possible to abuse it without going unnoticed.

oldjokes 8y ago

My car is only five years old but already has two sensors that have gone bad in the engine. Neither of them are vital and everything works great despite the occasional check engine light. Redundancy. It's a time intensive process to pop the head off and replace sensors. Nobody wants to pay their mechanic $100 an hour if they can avoid it. I can't even imagine a sensor failure having zero redundancy, much less changing control inputs to a complicated system like a multi-engine jet. This is completely against everything I know about engineering and I find even discussion of this being possible to be deeply offensive. The fact that it happened twice and hundreds of people are dead is just mind boggling.

pytyper2 8y ago
"...everything works great despite the occasional check engine light..." that may make you feel better but it's not true. The life and efficiency of your engine is being reduced. What year, make, model are you driving? When you sell it do you plan on telling the next owner that you drove it religiously with the check engine light on?
- oldjokes 8y ago
  The engine isn't magic, you can read the codes yourself, figure out what they mean, look up the replacement guidelines yourself, etc. It's really simple actually, anyone can do it.
linuxftw 8y ago
Failed sensors can lead to premature failure of various components, especially catalytic converters: http://ricksfreeautorepairadvice.com/causes-catalytic-converter-go-bad/ http://ricksfreeautorepairadvice.com/causes-catalytic-conver... Don't put off a $150 job and cost yourself a $2k job. Most sensor replacements don't require removing the cylinder heads.
- oldjokes 8y ago
  You're assuming I don't know what the codes are and that they are potentially harmful, neither of which is true.

jordache 8y ago

>I cannot even imagine what the designers of this thing are going through now. It must be terrible. Their poor decisions led to this. Single sensor as input to autopilot to augment the nose!?

Aromasin 8y ago
The turn-over of employees in aerospace is often quite quick. It's very likely that most of the people that designed this system are long gone, and the current design team are the ones picking up the pieces.
- jordache 8y ago
  Then they shouldn't have any personal guilt..
  - baddox 8y ago
    Guilt isn’t the only emotion that can be described as terrible.

Cacti 8y ago

My understanding is that the recovery scenario wasn’t supposed to be complicated at all, because in the event of a MCAS failure, it would always manifest as a runaway trim scenario, and that pilots are trained heavily on recognizing as correcting for that scenario, and that by following the standard runaway trim solution/checklist/process that this would necessarily involve disabling MCAS whether the pilots were aware of it or not. So the question is, did they, and if so did this fail, and if not, why didn’t they? Is this actually a direct problem with MCAS or is it closer to an issue of basic pilot training that only happened to involve MCAS? I suspect as the investigations unfold that our understanding of the accidents is only going to get more complex and nuanced.

ncallaway 8y ago
One significant problem as I understand it was the assumption that a failing MCAS would present as runaway stabilizer trim. When, in actuality, it may not have looked much like runaway stabilizer trim to the pilots. My understanding is in a runaway stabilizer trim situation the trim wheel (a medium sized wheel near the pilot and co-pilots knees) spins continuously and makes a loud clacking noise as part of the spinning. This makes runaway stabilizer trim easy to identify. The wheel is spinning (for no identifiable reason) and making noise. The MCAS failure, however applies trim discretely with intervals of 10-20s. This means it might be very hard to spot the adjustments to the trim wheel if you glance down briefly. A pilot might notice the control column is heavy, glance down to see if it's runaway trim see the wheel not moving, rule out that scenario and move on to other diagnostic steps.
- AceyMan 8y ago
  NB. It seems this moment news is breaking that the Ethiopia pilots indeed followed the checklist — yet they still lost the airplane. This is getting worse for Boeing at every turn.
  - Cacti 8y ago
    Well, it sounds like they turned off the autopilot but then turned it back on, which is not part of the checklist. It’s not clear from the reporting exactly what happened and I would hesitate to jump to conclusions. Accidents like this are almost always hugely complicated and are the result of many things going wrong in a specific order at a specific time, rather than a single factor. In fact we don’t even know what the cause of the crash was, and we won’t actually know for several months—there have been a number of crashes where everyone “knew” what the cause was early on, only for it later to come to light that is was something completely unrelated.

sixothree 8y ago

Are AOA sensors really so prone to failure that they caused the downing of two airplanes?

Xixi 8y ago
Like all sensors out there in the atmosphere, icing is an issue. So not only are they error prone, but failures are sometimes not independent. Airbus planes from the A320 family have three such sensors, but it so happened that on November 5th, 2014, an A321 had two sensors freeze in the wrong position during flight. Two sensors against one win, so the flight envelope protection system kicked in and the aircraft started to nose dive. It lost 4000 feet in altitude before the crew regained control... Details of the flight if you want to do some research: Lufthansa Airbus A321-200, registration D-AIDP, performing flight LH-1829 from Bilbao, SP (Spain) to Munich (Germany) on November 5th, 2014.
ams6110 8y ago
Normally no, from what I have read they are very reliable. And the sensors are the same ones used on all 737s. So it seems that there is some other problem with the MAX that is leading to the a false "nose high" indication.
linuxftw 8y ago
I wouldn't think so. My personal theory is AoA is sending valid data to one of the flight computers, that flight computer is sending invalid data to the MCAS, or the MCAS has a software problem. I suspect AoA is sending signed data, computer expects unsigned data, or possibly LSB/MSB conversion problem. Or it could just be faulty logic somewhere.
- salawat 8y ago
  The sensors on Lion Air were 10+ degrees out of calibratiom IIRC. Critical AoA is somewhere in the realm of 17 to 20 degrees. That doesn't leave much room for where MCAS wouldn't be a problem. Also, AoA sensors are not normally safety critical. The MAX is the first 737 airframe for which this is the case.

hbarka 8y ago

I enjoyed your comment because I’m particularly interested in Tesla’s Autopilot system and your mention of “graceful degradation and clean handover” sounds like it should apply to driving autonomy levels as well.

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
wjnc 8y ago
In the way we use cars there seems no time for graceful handover. You're about the same time interval away from the next car as the time to gain situational awareness, probably less. A plane is usually quite some minutes away from the ground and other planes, or f.e. when landing on autopilot the pilots are not distracted. If you'd apply this standard, the bar for autopilot in cars would be very high indeed. (That might be warranted, although in a risk / reward perspective humans are terrible drivers too.)