15 ms·

Boeing 737 Max Crashes: Sensors Vulnerable to Failure

102 points by bushido 8y ago

jngreenlee 8y ago
Realize some pilots lurk here...was the angle of attack sensor ever associated with an automatic system prior to MCAS? Seems like previous errors would have been less impactful due to human observation and synthesis with other instruments, window observation, etc. If so, it shows how GIGO works in automatic systems...this should have been scrutinized as AOA sensor was not above the 1:1,000,000 threshold for single-input systems w/o failover.
- dreamcompiler 8y ago
  > was the angle of attack sensor ever associated with an automatic system prior to MCAS? Yes [0]. The stick shaker and stick pusher. But those are overridable by pilot muscle power. With the MCAS, there can be situations where that isn't the case, and we have two crashes as a (probable) result. [0] https://philip.greenspun.com/blog/2018/11/11/boeing-737-crash-is-first-mass-killing-by-software/ https://philip.greenspun.com/blog/2018/11/11/boeing-737-cras...
- Xixi 8y ago
  Not a pilot, but angle of attack sensors are used with flight envelope protection systems in Airbus airliners (and probably newer Boeing planes too, though I'm not sure). Take for instance this particular incident on an A321 aircraft [1]: "two angle of attack sensors having frozen in their positions during climb at an angle, that caused the fly by wire protection to assume the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated forcing the aircraft to pitch down, which could not be corrected even by full back stick input. The crew eventually disconnected the related Air Data Units and was able to recover the aircraft.". A321 have three AoA sensors, so if two freeze in the same wrong position, they outvote the good sensor... [1] https://avherald.com/h?article=47d74074 https://avherald.com/h?article=47d74074 EDIT: incomplete citation
- howard941 8y ago
  There are at least two A320-family AoA<>automation incidents listed here: https://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_the_Airbus_A320_family https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
  - lutorm 8y ago
    Specifically: "On 5 November 2014, Lufthansa Flight 1829, an Airbus A321 was flying from Bilbao to Munich when the aircraft, while on autopilot, lowered the nose into a descent reaching 4000 fpm. The uncommanded pitch-down was caused by two angle of attack sensors that were jammed in their positions, causing the fly by wire protection to believe the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated, forcing the aircraft to pitch down, which could not be corrected even by full stick input. The crew disconnected the related Air Data Units and were able to recover the aircraft."
    - cmurf 8y ago
      That's disgusting. The computer thinks the plane in a otherwise normal climb (attitude, airspeed, power, rate of climb) is stalling? That's just...we cannot call this kind of automation anything like a pilot. The term "autopilot" is too much anthropomorphizing. A pilot reacting this way in this same situation with the same information, they would be incompetent or crazy to trust two AOA sensors and assume a stall. If a crash ensued they absolutely would be blamed for it. Not the goddamn sensors. And calling such a thing a safeguard when it can fail danger like this? A 4000fpm decent as a correction for a stall in a normal climb? That's just batshit. I'm now in a dive recovery operation, itself with higher risks than a stall in the midst of a climb! A stall in a climb is not more dangerous than a 4000fpm dive. Think of throwing a football, it's stalling the entire way, all that's going to happen is the plane stops climbing as fast as it was and the nose is supposed to come down (that's why planes must have positive static stability) and viola we're flying again! Let's pretend it is true you are somehow stalling in a normal climb (an instantaneous increase in angle of attack): what do does a pilot need to do? Neutralize controls, that's it. Maybe the attitude goes somewhere between 5 and 0 degrees. But a 4000fpm decent? pUke. (I'm a pilot and former CFII, so I feel sufficiently entitled with my biases to comment with such editorialization.) The only reason we humans tolerate such unsophisticated systems arriving at weird and dangerous edge cases is because the automation has in fact helped to make flying so much safer. But so have many other things: components are so much more reliable, turbine engines run for many thousands of hours between overhauls, they're really incredible. The training is a lot better. CRM has made things way safer! It's not just the automation. And we really shouldn't be cutting automation so much slack either. I mean for fucks sake, 4000fpm? Crazy. But it's an edge case and it should not be accepted with "oh well the pilots recovered! AOK!" Obviously if the pilots had not recovered a central theme would be "the pilots didn't recover! INCOMPETENT PILOTS! THEY DIDN'T FOLLOW THE CHECKLIST! WARGARBLE! Automation is not anything like a human. So accusing it of being crazy, or trying to murder people isn't exactly accurate. But I think it's reasonable to hold it accountable to things that look like "insane" behavior when it fails this spectacularly. Its just a sequence of narrowly defined laws, with each law doing its specific thing, without any regard to other information. Very myopic worldview. And then having more authority to take (wrong) action than pilots have to correct it?! What a bad sequence!
- PaulHoule 8y ago
  Airbus learned this the hard way in this incident: https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T which I think had to do with New Zealand regulators following up on https://en.wikipedia.org/wiki/Qantas_Flight_72 https://en.wikipedia.org/wiki/Qantas_Flight_72 A group of New Zealand pilots and regulators pressured a German pilot to test the envelope protection systems during a simulated emergency landing (so well simulated that the pilot did not seem to know where he was going ahead of time) The AoA vanes were out of commission with this aircraft, the flight protection system malfunctioned and the plane flew into the ground. That is, the danger got caught on a test flight and only 7 were killed. Compare that to the problem happening on not one but two revenue flights. Had the FAA required that pilots ride out an MCAS failure in a simulator they probably would have learned that pilots could not ride out an MCAS failure before passengers were put at risk.
  - prolepunk 8y ago
    AoA vanes not just failed, they failed because of incorrect maintenance procedure. >One of the contributing causes was incorrect maintenance procedures which allowed water to enter the angle of attack (AOA) sensors. During fuselage rinsing with water before painting, three days before the flight, the AOA sensors were unprotected. As specified in the Structure Repair Manual by Airbus, it is mandatory to fit a protection device on AOA sensors before these tasks. In this case several failures all happening at the same time cause the crash: software, pilots following protocols and maintenance.
    - alexis_fr 8y ago
      Wow, that still happens? Not the first accident caused by incorrectly covering and uncovering probes during maintenance. I think once the Pitot tubes were still covered with their caps.
dreamcompiler 8y ago
This thing is a small fin on a swivel. It's a wind vane about 6 inches long, sticking out into a 500 mph airstream. It's a heavy-duty piece of metal, but that swivel joint makes it vulnerable to bird strikes, freezing, etc.
- dboreham 8y ago
  Hmm...if it works that way it seems quite easy to determine if it is working -- if it isn't swiveling then it is probably broken. Perhaps also you could make an active version of this sensor where there's a stepper that pushes it a bit either side of the current neutral position, then measure resistance to that push with a strain gauge. It might be possible to thereby know that you're pushing against flowing air, not bird guts jammed in the mechanism, for example.
  - dreamcompiler 8y ago
    Probably could be done but it would require significant reengineering because it introduces new failure modes. Best approach is what everybody else does: Assume the AoA sensor can fail and design the software accordingly. Boeing didn't do that and the results were fatal.
    - noir_lord 8y ago
      Use an odd number > 1 and sound an alert if they can't reach a quorum. Triple redundancy is pretty common in safety critical systems.
  - cesarb 8y ago
    > if it isn't swiveling then it is probably broken According to https://www.satcom.guru/2019/04/what-happened-on-et302.html https://www.satcom.guru/2019/04/what-happened-on-et302.html it was swiveling freely because it was broken.
mansoor_ 8y ago
Everything is vunerable to failure
AceyMan 8y ago
I wonder if it's feasible to build a guard to cover the AoA sensor to mitigate the ability of foreign objects damaging the vane. Clearly it'd need to be very transparent to airflow—something like the face guards of an NFL player—a heavy gauge titanium cage or something like that. Btw, the Wright Bros knew the importance of AoA and included one on the Wright Flyer. Implementation? — a length of straight stick parallel to the wing chord with a bit of ribbon affixed to the end :-D (KISS at work!)
- subb 8y ago
  I wonder if they could take the mesure from the underside of the plane (90 degrees difference from the front)? It might help for bird strike, not much for ice.
  - CamperBob2 8y ago
    I wonder if a small array of microphones couldn't be used to accomplish the same thing by cross-correlating the sound made by airflow hitting adjacent surfaces at different angles. Seems like a bad idea to use mechanical vanes for this, especially if you're going to bet the whole farm on a single instrument as Boeing seems to have done.
- lawlessone 8y ago
  >omething like the face guards of an NFL player it would probably just get blocked then by bits of bird.
- VBprogrammer 8y ago
  AoA sensors are sensitive to very minor deviations in the surface they are mounted to, there are aircraft which don't allow any minor damage in that area because it can cause misreadings.
  - HeyLaughingBoy 8y ago
    You seem to have a pretty high level of knowledge of avionics. Do I remember correctly that you worked in the field? If so, do you have any idea of what the allowable range of an AoA sensor might be? i.e., on some systems I've worked on, we had a range that a sensor was expected to be in (normal, just do regular control loop), an outer range that was unlikely (stop reporting for a while/hold control output constant to see if it gets better or worse before warning operator), and a narrower range that was flat out impossible (notify operator of sensor failure/put system into a graceful shutdown). I assume that the engineers designing this system had something similar, but given that MCAS activated with an AoA indication of 80 degrees (IIRC) and that's essentially impossible on a 737 (or at least would correlate with a hell of a lot of other warnings), was there anything they could do to warn the operator (pilot) that there was likely a bad sensor input?
jpm_sd 8y ago
I am surprised that there is no ultrasonic method of measuring AoA.
- CamperBob2 8y ago
  Exactly, using mechanical vanes for this sounds pretty stupid, given what can be done with acoustical signal processing. It must be a more difficult problem than it seems.
  - HeyLaughingBoy 8y ago
    Not necessarily. It could be that after doing the safety analysis, the mechanical method is less likely to fail. Remember that there are usually many ways to skin a cat. Often the one chosen hits the sweet spot of reliability, manufacturability, cost (both initial purchase and operational), maintainability (e.g., how easy is it to test/inspect), etc. When you have decades of experience with something and it works well enough, there's generally no reason to improve it when your engineering resources can be focused on more pressing problems. I was a team lead on a new system that was based on the IBM ISA bus in 2013. The reasons were simply that it worked well enough and we had tons of experience with it and could reuse a lot of custom-built hardware and code. The only downside (and reason we switched), was that it was getting harder to find blade PCs that still had an ISA bus and the system was expected to be in the field for 15+ years.
    - jpm_sd 8y ago
      I was hardware lead on a system that was the 2013 revision to an existing ISA-bus blade PC architecture. We threw it all out and switched to an off-the-shelf DIN rail PC from Beckhoff with snap-on EtherCAT modules. It took a little extra development time but it was 100% worth it. Sometimes you gotta throw it out.
newsoul2019 8y ago
One thing I haven't seen fully explained ( maybe it was but I missed it ). Were these vanes damaged on both of these downed flights? Why didn't all the other 737 MAX 8's crash around the world, Was it something about the flight profile or the weather? Can the plane be flown as-is with a sufficiently conservative flight profile?
- VBprogrammer 8y ago
  The speculation on the Ethiopian flight was a bird strike. The data was normal until a few seconds after take off. Another issue could be the failure mode, if it failed at or near 0 AoA it wouldn't cause too many safety of flight issues on a normal flight.
  - linuxftw 8y ago
    Well, unless you need the MCAS to kick in, and the plane plummets to the earth in a hard stall because it didn't.
    - VBprogrammer 8y ago
      If a pilot goes anywhere near a stall on a normal commercial flight then they have fucked something up. MCAS is only required to maintain a positive slope to AoA - stick force curve. MCAS resulted in 2 crashes because the implementation was completely brain dead. Not because the aircraft is impossible to fly without it.
      - linuxftw 8y ago
        > MCAS is only required to maintain a positive slope to AoA - stick force curve. You and other keeps parroting this line, I don't believe it to be true. MCAS is an anti-stall device, show me once source that says otherwise. > Not because the aircraft is impossible to fly without it. Not impossible, but possibly also not safe. The system was implemented to prevent stalls in the MAX 8 planes due to it's flight characteristics.
gordon_freeman 8y ago
So if the sensors have a higher rate of failure then it looks like changing MCAS software to rely on both sensors may not be enough as there is a higher probability of both sensors malfunctioning and sending bad data to MCAS at the same moment. In that case, I am not sure how would MCAS behave.
- stackola 8y ago
  If these sensors have a common failure mode (eg. always showing lowest possible value after blowing up), this could be bad. If they just produce junk, it's unlikely they would agree, and MCAS would be turned off.
  - cjbprime 8y ago
    The common failure mode would be e.g. freezing, perhaps in identical positions, while on the ground before takeoff. You can't easily detect it before takeoff because the sensor requires airflow.
    - blattimwind 8y ago
      I assume you can walk up to it and turn it by hand to at least see that the mechanical part is moving freely as-of now.
      - bgdam 8y ago
        Can't always test the mechanics like that. Ex: Pitot tubes are just hollow tubes. The interiors of pitot tubes getting iced over has crashed planes. This wouldn't even be visible from the outside, and there is no test you could do.
        maxmunzel 8y ago
        Couldn’t you just blow in?
salawat 8y ago
This is not even remotely surprising, and by all standards should be taken into account in your implementation of dependent systems' gradual degradation fault modes.
markphip 8y ago
Shouldn’t it be easy to detect a stuck sensor and then stop using its data unless it recovers?
shadykiller 8y ago
Shouldn't the headline say "Boeing 737 Max" ? There's a big difference compared to regular 737s
- oliveshell 8y ago
  I think it’s well-enough implied by ‘crashes’. Regular 737s haven’t been crashing recently.
  - deleted 8y ago
    [deleted]
- Someone1234 8y ago
  Yes it should, and that's actually the original title of the article: > Boeing 737 Max Crashes: Sensors Vulnerable to Failure Mods should likely change it.
- sctb 8y ago
  Updated. Thanks!
ansible 8y ago
I'm wondering now if MCAS should be an active system. Perhaps it should instead just warn the pilots that it thinks a stall is imminent instead, like when you get the stick shaking.
- jhayward 8y ago
  As I understand it, that would require specific training for pilots of the MAX 8, and (not sure) perhaps a new type certification. Those are expensive and it was a specific goal to avoid them.
  - magnamerc 8y ago
    Having to get a new type certificate would likely kill the MAX. I work in aircraft certification, and I know it would be a very long and slow process on the order of years.
    - jhayward 8y ago
      Not only is it a long and expensive process, but the 737 type cert grandfathers in a whole lot of things that would not be permitted in a new type. So they would have to re-engineer a brand new plane from the wheels up. Keeping the 737 type certification was a powerful incentive. Oversight investigation should look in to whether it should have been permitted by the regulator, and what other perverse incentives may exist in the certification process.
- Someone1234 8y ago
  They cannot do that, because then the Max loses its shared type rating due to the larger engines. MCAS exists to give the aircraft more similar flight characteristics to its predecessors.
  - tzs 8y ago
    MCAS apparently exists for more than that. A few days ago someone posted a link and some quotes from the FAA regulations on pitch stability. They require that the control force must be positive up to and throughout a stall. Without MCAS there would be a point on that MAX where you no longer needed positive control force to continue pitching up. Without MCAS or some other system to automatically counter that, so that the pilot needs positive control force to continue pitching up, the plane could not be certified even if they did go for a new type rating.
Lind5 8y ago
Sadly, another "the sensor did it" scenario. https://semiengineering.com/dirty-data-is-the-sensor-malfunctioning/ https://semiengineering.com/dirty-data-is-the-sensor-malfunc... The tragic example of the Lion Air crash of a new Boeing MAX 8 aircraft, which on Oct. 29 killed all aboard, may be heading toward “the sensor did it” category. The black box recovered from the flight showed inconsistent data from one of the two angle-of-attack (AOA) sensors. With one half of the data apparently incorrect, it was enough to trigger this plane’s anti-stall system into a nose down action, which the pilots wrestled all the way into the Java sea.
- 05 8y ago
  MCAS is not an ‘anti-stall system’, it’s a cheap type certification compliance hack
  - Doxin 8y ago
    It’s a cheap type certification compliance hack that prevents stalls.
- deleted 8y ago
  [deleted]
digikata 8y ago
All equipment on the plane has a potential for failure... the question is what systematic design and pilot training is in place to take action in the face of failures.
- iscrewyou 8y ago
  I don’t think it’s fair to put pilot training and systematic design right next to each other. If the design in itself, and the subsequent factors of safety are at fault, pilots can’t recover from that fast enough. Why put them in a dangerous situation and expect them to recover. How long could this go on even if we didn’t have these two plane crashes. The pilots/airlines could be at fault for not training too much but the bigger issue is the system in the question. It’s like saying the drivers don’t know how to drive because the manufacturer didnt put it front and center on the driver’s manual that the car suddenly swerves right. As ic drivers should always have their hands tight on the wheels and be ready for situations like that. It’s not fair.
  - digikata 8y ago
    The pilot training is part of the overall systematic design, neither the equipment, nor the pilots exist alone.
- peteradio 8y ago
  What? Does the degree of failure not matter at all? That makes no sense. The question is whether all these parts are actually meeting the specified failure tolerance rate.
  - digikata 8y ago
    It matters, but if the rate of failure is high enough to be expected to be encountered within an all-fleet, constant flight environment for the lifetime of the aircraft, then you plan for failures as well as having the pilot know what to do when there is a failure. In air sensors like this, their failures are higher than the rate at which you can ignore. They fail on their own, they get bumped and bent on the ground, they ice over in the air, or have objects striking them.
velox_io 8y ago
I'm just amazed by how poorly this was implemented. Software engineers expect hardware to fail. This wasn't just the work of a newb (or I hope it wasn't). This was planned; both the requirements and the failure modes. Then implemented by another team, and the quality verified by another team, even a 3rd party. There wasn't even a cutoff if it started operating outside it's expected range, surely one of those people would have noticed the shortfalls. This software should have the same level of engineering as other systems/ parts of the plane. Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent. This software wasn't even a crucial feature. It was only needed when high power is applied to the engines (like during takeoff) to stop the nose feeling too light by existing 737 pilots. This wasn't a jet-fighter, which would become unstable without the aid of computers. It's a shame 338 died because of what is effectively a 'schoolboy error'.
- js2 8y ago
  > Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Pitch != AoA. https://aviation.stackexchange.com/questions/2287/can-the-pitch-be-very-different-from-the-angle-of-attack https://aviation.stackexchange.com/questions/2287/can-the-pi... http://www.aerospaceweb.org/question/aerodynamics/q0165.shtml http://www.aerospaceweb.org/question/aerodynamics/q0165.shtm...
- yutyut 8y ago
  >Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! You would need a sensor to measure relative wind. Angle of attack is the angle between the wing and the relative wind.
bookofjoe 8y ago
The elephant in the room, which I've not yet seen alluded to: Boeing wants at all costs (which will be very high) to avoid having pilots be required to requalify with new MCAS software in flight simulators.
lurker0094 8y ago
So, 140 failures in 30 years of aviation... ~4 failures per year FOR ALL FLIGHTS IN A YEAR. That's unreliable?
- peteradio 8y ago
  140 incidents. Untold number of averted failures.
cmurf 8y ago
Cessna 150/152/172 stall warning "horn" works by negative pressure. https://www.youtube.com/watch?v=Q1JRTCBkWKQ https://www.youtube.com/watch?v=Q1JRTCBkWKQ And then the Cessna 182 has an aerodynamic switch that activates an electric "horn". And in either case they can get jammed up with bugs. But we do a lot of training with slow flight, approach to stall, full stall, and recoveries including recovery from the ensuing dive and secondary stalls.
ggm 8y ago
Root cause analysis: excessive pressure to keep type rating caused lax engineering to be considered acceptable. Fix:let the FAA do it's job independently
- mjevans 8y ago
  Fix: Make Engineers actually liable for the correct classification of systems and require signed justification for NOT defaulting to the highest possible standard.
  - ggm 8y ago
    ...which demands a fully independent FAA. Not the one we seem to have which appears on the face of it, compromised by the consequences of embedded self assessment and 'same type' rules. Why was it in the national interest to have Boeing get 'same type' ? Because of the cost of competing with airbus. The regulator was hamstrung in the interests of 'efficiency' (I'm not an American and this is an argument i read which resonates. I'd welcome good rebuttal)
  - rusticpenn 8y ago
    This also causes a lot of over engineering. ( Which maybe good for Boeing right about now)
    - ggm 8y ago
      I'd dispute it causes the over-engineering. Engineering principles cause safety margins, and regulators enforce safety margins. As the industry decides it can live with reduced safety margins they get reduced. When they get too close to a risk window, they get raised. I know the over-engineering term exists, but I think it's based on flawed logic. Not having margins results in under-engineering, which is significantly worse in almost all cases. Its not a symmetric world out there. Some things are worse than others.
- zepearl 8y ago
  On a technical level: personally I think that the root cause could be a lack of "integration" of the system (MCAS) with the rest. Meaning that e.g. 1) if the pilots' yokes/sticks are being pulled and 2) if the altitude based on gps and/or air pressure and/or radar is decreasing and 3) if the speed registered by the airspeed probe is high respectively increasing and 4) if the fuel being used by the engines to keep or even increase the speed is very low and 5) if the G-force registered by <some sensor, if it exists?> is lower than usual (neutral or even negative?) 6) etc... (anything else excluding AoA?) ...then the airplane is mooost probably "going down", even if the AoA says the opposite. As the MCAS auto-trim has apparently a big potential impact on how the airplane flies (surfaces on the tail are big and their angle can move a lot), it should therefore not rely on just 1 (or even 2 duplicate) lonely sensor(s) but on multiple types that provide multiple point-of-views of the aircraft's situation to then abstract/determine/confirm the current high-level situation (in this case "is the aircraft going up or down?") and only after such result to react/take countermeasures to save the aircraft respectively help the pilots and/or lower costs for the flight. If this is not totally wrong, is this (the total integration of all sensors) how Airbus did it with their super-automated systems like e.g. their side-sticks/joysticks just telling the airplane what pilots "want"? (meaning, did they start already from scratch creating a totally integrated system, if they did create such an integrated system?) (not sure about how automated the new Boeing airplanes are, so I did not mention them...)
cmurf 8y ago
Too much is being made about the sensors. From my pilot's perspective it's far worse that: a.) MCAS can mistrim the airplane faster than pilots can even recognize what's going on. b.) Boeing and FAA directives still don't account for a.), and therefore they're still giving pilot's inadequate advice. And it's sufficiently bad that I think it's intentional because the advice is consistent with prior Boeing 737 models: runaway stabilizer. If they came up with a better checklist to run that only applies to the MAX, that is suggestive that at least difference training is needed, and might even suggest a different type certificate is needed. Much of this is discussed here which I found to be decent: https://www.satcom.guru/2019/04/what-happened-on-et302.html https://www.satcom.guru/2019/04/what-happened-on-et302.html incredibly it just dumps the stab and leaves it to the pilot to get it back. Electric trim must be used to neutralize control column pitch forces before cutout!
ngcc_hk 8y ago
3 is the minimum for any critical system.

102 points by bushido 8y ago

Realize some pilots lurk here...was the angle of attack sensor ever associated with an automatic system prior to MCAS? Seems like previous errors would have been less impactful due to human observation and synthesis with other instruments, window observation, etc. If so, it shows how GIGO works in automatic systems...this should have been scrutinized as AOA sensor was not above the 1:1,000,000 threshold for single-input systems w/o failover.

dreamcompiler 8y ago
> was the angle of attack sensor ever associated with an automatic system prior to MCAS? Yes [0]. The stick shaker and stick pusher. But those are overridable by pilot muscle power. With the MCAS, there can be situations where that isn't the case, and we have two crashes as a (probable) result. [0] https://philip.greenspun.com/blog/2018/11/11/boeing-737-crash-is-first-mass-killing-by-software/ https://philip.greenspun.com/blog/2018/11/11/boeing-737-cras...
Xixi 8y ago
Not a pilot, but angle of attack sensors are used with flight envelope protection systems in Airbus airliners (and probably newer Boeing planes too, though I'm not sure). Take for instance this particular incident on an A321 aircraft [1]: "two angle of attack sensors having frozen in their positions during climb at an angle, that caused the fly by wire protection to assume the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated forcing the aircraft to pitch down, which could not be corrected even by full back stick input. The crew eventually disconnected the related Air Data Units and was able to recover the aircraft.". A321 have three AoA sensors, so if two freeze in the same wrong position, they outvote the good sensor... [1] https://avherald.com/h?article=47d74074 https://avherald.com/h?article=47d74074 EDIT: incomplete citation
howard941 8y ago
There are at least two A320-family AoA<>automation incidents listed here: https://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_the_Airbus_A320_family https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
- lutorm 8y ago
  Specifically: "On 5 November 2014, Lufthansa Flight 1829, an Airbus A321 was flying from Bilbao to Munich when the aircraft, while on autopilot, lowered the nose into a descent reaching 4000 fpm. The uncommanded pitch-down was caused by two angle of attack sensors that were jammed in their positions, causing the fly by wire protection to believe the aircraft entered a stall while it climbed through FL310. The Alpha Protection activated, forcing the aircraft to pitch down, which could not be corrected even by full stick input. The crew disconnected the related Air Data Units and were able to recover the aircraft."
  - cmurf 8y ago
    That's disgusting. The computer thinks the plane in a otherwise normal climb (attitude, airspeed, power, rate of climb) is stalling? That's just...we cannot call this kind of automation anything like a pilot. The term "autopilot" is too much anthropomorphizing. A pilot reacting this way in this same situation with the same information, they would be incompetent or crazy to trust two AOA sensors and assume a stall. If a crash ensued they absolutely would be blamed for it. Not the goddamn sensors. And calling such a thing a safeguard when it can fail danger like this? A 4000fpm decent as a correction for a stall in a normal climb? That's just batshit. I'm now in a dive recovery operation, itself with higher risks than a stall in the midst of a climb! A stall in a climb is not more dangerous than a 4000fpm dive. Think of throwing a football, it's stalling the entire way, all that's going to happen is the plane stops climbing as fast as it was and the nose is supposed to come down (that's why planes must have positive static stability) and viola we're flying again! Let's pretend it is true you are somehow stalling in a normal climb (an instantaneous increase in angle of attack): what do does a pilot need to do? Neutralize controls, that's it. Maybe the attitude goes somewhere between 5 and 0 degrees. But a 4000fpm decent? pUke. (I'm a pilot and former CFII, so I feel sufficiently entitled with my biases to comment with such editorialization.) The only reason we humans tolerate such unsophisticated systems arriving at weird and dangerous edge cases is because the automation has in fact helped to make flying so much safer. But so have many other things: components are so much more reliable, turbine engines run for many thousands of hours between overhauls, they're really incredible. The training is a lot better. CRM has made things way safer! It's not just the automation. And we really shouldn't be cutting automation so much slack either. I mean for fucks sake, 4000fpm? Crazy. But it's an edge case and it should not be accepted with "oh well the pilots recovered! AOK!" Obviously if the pilots had not recovered a central theme would be "the pilots didn't recover! INCOMPETENT PILOTS! THEY DIDN'T FOLLOW THE CHECKLIST! WARGARBLE! Automation is not anything like a human. So accusing it of being crazy, or trying to murder people isn't exactly accurate. But I think it's reasonable to hold it accountable to things that look like "insane" behavior when it fails this spectacularly. Its just a sequence of narrowly defined laws, with each law doing its specific thing, without any regard to other information. Very myopic worldview. And then having more authority to take (wrong) action than pilots have to correct it?! What a bad sequence!
PaulHoule 8y ago
Airbus learned this the hard way in this incident: https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T which I think had to do with New Zealand regulators following up on https://en.wikipedia.org/wiki/Qantas_Flight_72 https://en.wikipedia.org/wiki/Qantas_Flight_72 A group of New Zealand pilots and regulators pressured a German pilot to test the envelope protection systems during a simulated emergency landing (so well simulated that the pilot did not seem to know where he was going ahead of time) The AoA vanes were out of commission with this aircraft, the flight protection system malfunctioned and the plane flew into the ground. That is, the danger got caught on a test flight and only 7 were killed. Compare that to the problem happening on not one but two revenue flights. Had the FAA required that pilots ride out an MCAS failure in a simulator they probably would have learned that pilots could not ride out an MCAS failure before passengers were put at risk.
- prolepunk 8y ago
  AoA vanes not just failed, they failed because of incorrect maintenance procedure. >One of the contributing causes was incorrect maintenance procedures which allowed water to enter the angle of attack (AOA) sensors. During fuselage rinsing with water before painting, three days before the flight, the AOA sensors were unprotected. As specified in the Structure Repair Manual by Airbus, it is mandatory to fit a protection device on AOA sensors before these tasks. In this case several failures all happening at the same time cause the crash: software, pilots following protocols and maintenance.
  - alexis_fr 8y ago
    Wow, that still happens? Not the first accident caused by incorrectly covering and uncovering probes during maintenance. I think once the Pitot tubes were still covered with their caps.

dreamcompiler 8y ago

This thing is a small fin on a swivel. It's a wind vane about 6 inches long, sticking out into a 500 mph airstream. It's a heavy-duty piece of metal, but that swivel joint makes it vulnerable to bird strikes, freezing, etc.

dboreham 8y ago
Hmm...if it works that way it seems quite easy to determine if it is working -- if it isn't swiveling then it is probably broken. Perhaps also you could make an active version of this sensor where there's a stepper that pushes it a bit either side of the current neutral position, then measure resistance to that push with a strain gauge. It might be possible to thereby know that you're pushing against flowing air, not bird guts jammed in the mechanism, for example.
- dreamcompiler 8y ago
  Probably could be done but it would require significant reengineering because it introduces new failure modes. Best approach is what everybody else does: Assume the AoA sensor can fail and design the software accordingly. Boeing didn't do that and the results were fatal.
  - noir_lord 8y ago
    Use an odd number > 1 and sound an alert if they can't reach a quorum. Triple redundancy is pretty common in safety critical systems.
- cesarb 8y ago
  > if it isn't swiveling then it is probably broken According to https://www.satcom.guru/2019/04/what-happened-on-et302.html https://www.satcom.guru/2019/04/what-happened-on-et302.html it was swiveling freely because it was broken.

mansoor_ 8y ago

Everything is vunerable to failure

AceyMan 8y ago

I wonder if it's feasible to build a guard to cover the AoA sensor to mitigate the ability of foreign objects damaging the vane. Clearly it'd need to be very transparent to airflow—something like the face guards of an NFL player—a heavy gauge titanium cage or something like that. Btw, the Wright Bros knew the importance of AoA and included one on the Wright Flyer. Implementation? — a length of straight stick parallel to the wing chord with a bit of ribbon affixed to the end :-D (KISS at work!)

subb 8y ago
I wonder if they could take the mesure from the underside of the plane (90 degrees difference from the front)? It might help for bird strike, not much for ice.
- CamperBob2 8y ago
  I wonder if a small array of microphones couldn't be used to accomplish the same thing by cross-correlating the sound made by airflow hitting adjacent surfaces at different angles. Seems like a bad idea to use mechanical vanes for this, especially if you're going to bet the whole farm on a single instrument as Boeing seems to have done.
lawlessone 8y ago
>omething like the face guards of an NFL player it would probably just get blocked then by bits of bird.
VBprogrammer 8y ago
AoA sensors are sensitive to very minor deviations in the surface they are mounted to, there are aircraft which don't allow any minor damage in that area because it can cause misreadings.
- HeyLaughingBoy 8y ago
  You seem to have a pretty high level of knowledge of avionics. Do I remember correctly that you worked in the field? If so, do you have any idea of what the allowable range of an AoA sensor might be? i.e., on some systems I've worked on, we had a range that a sensor was expected to be in (normal, just do regular control loop), an outer range that was unlikely (stop reporting for a while/hold control output constant to see if it gets better or worse before warning operator), and a narrower range that was flat out impossible (notify operator of sensor failure/put system into a graceful shutdown). I assume that the engineers designing this system had something similar, but given that MCAS activated with an AoA indication of 80 degrees (IIRC) and that's essentially impossible on a 737 (or at least would correlate with a hell of a lot of other warnings), was there anything they could do to warn the operator (pilot) that there was likely a bad sensor input?

jpm_sd 8y ago

I am surprised that there is no ultrasonic method of measuring AoA.

CamperBob2 8y ago
Exactly, using mechanical vanes for this sounds pretty stupid, given what can be done with acoustical signal processing. It must be a more difficult problem than it seems.
- HeyLaughingBoy 8y ago
  Not necessarily. It could be that after doing the safety analysis, the mechanical method is less likely to fail. Remember that there are usually many ways to skin a cat. Often the one chosen hits the sweet spot of reliability, manufacturability, cost (both initial purchase and operational), maintainability (e.g., how easy is it to test/inspect), etc. When you have decades of experience with something and it works well enough, there's generally no reason to improve it when your engineering resources can be focused on more pressing problems. I was a team lead on a new system that was based on the IBM ISA bus in 2013. The reasons were simply that it worked well enough and we had tons of experience with it and could reuse a lot of custom-built hardware and code. The only downside (and reason we switched), was that it was getting harder to find blade PCs that still had an ISA bus and the system was expected to be in the field for 15+ years.
  - jpm_sd 8y ago
    I was hardware lead on a system that was the 2013 revision to an existing ISA-bus blade PC architecture. We threw it all out and switched to an off-the-shelf DIN rail PC from Beckhoff with snap-on EtherCAT modules. It took a little extra development time but it was 100% worth it. Sometimes you gotta throw it out.

newsoul2019 8y ago

One thing I haven't seen fully explained ( maybe it was but I missed it ). Were these vanes damaged on both of these downed flights? Why didn't all the other 737 MAX 8's crash around the world, Was it something about the flight profile or the weather? Can the plane be flown as-is with a sufficiently conservative flight profile?

VBprogrammer 8y ago
The speculation on the Ethiopian flight was a bird strike. The data was normal until a few seconds after take off. Another issue could be the failure mode, if it failed at or near 0 AoA it wouldn't cause too many safety of flight issues on a normal flight.
- linuxftw 8y ago
  Well, unless you need the MCAS to kick in, and the plane plummets to the earth in a hard stall because it didn't.
  - VBprogrammer 8y ago
    If a pilot goes anywhere near a stall on a normal commercial flight then they have fucked something up. MCAS is only required to maintain a positive slope to AoA - stick force curve. MCAS resulted in 2 crashes because the implementation was completely brain dead. Not because the aircraft is impossible to fly without it.
    - linuxftw 8y ago
      > MCAS is only required to maintain a positive slope to AoA - stick force curve. You and other keeps parroting this line, I don't believe it to be true. MCAS is an anti-stall device, show me once source that says otherwise. > Not because the aircraft is impossible to fly without it. Not impossible, but possibly also not safe. The system was implemented to prevent stalls in the MAX 8 planes due to it's flight characteristics.

gordon_freeman 8y ago

So if the sensors have a higher rate of failure then it looks like changing MCAS software to rely on both sensors may not be enough as there is a higher probability of both sensors malfunctioning and sending bad data to MCAS at the same moment. In that case, I am not sure how would MCAS behave.

stackola 8y ago
If these sensors have a common failure mode (eg. always showing lowest possible value after blowing up), this could be bad. If they just produce junk, it's unlikely they would agree, and MCAS would be turned off.
- cjbprime 8y ago
  The common failure mode would be e.g. freezing, perhaps in identical positions, while on the ground before takeoff. You can't easily detect it before takeoff because the sensor requires airflow.
  - blattimwind 8y ago
    I assume you can walk up to it and turn it by hand to at least see that the mechanical part is moving freely as-of now.
    - bgdam 8y ago
      Can't always test the mechanics like that. Ex: Pitot tubes are just hollow tubes. The interiors of pitot tubes getting iced over has crashed planes. This wouldn't even be visible from the outside, and there is no test you could do.
      - maxmunzel 8y ago
        Couldn’t you just blow in?

salawat 8y ago

This is not even remotely surprising, and by all standards should be taken into account in your implementation of dependent systems' gradual degradation fault modes.

markphip 8y ago

Shouldn’t it be easy to detect a stuck sensor and then stop using its data unless it recovers?

shadykiller 8y ago

Shouldn't the headline say "Boeing 737 Max" ? There's a big difference compared to regular 737s

oliveshell 8y ago
I think it’s well-enough implied by ‘crashes’. Regular 737s haven’t been crashing recently.
- deleted 8y ago
  [deleted]
Someone1234 8y ago
Yes it should, and that's actually the original title of the article: > Boeing 737 Max Crashes: Sensors Vulnerable to Failure Mods should likely change it.
sctb 8y ago
Updated. Thanks!

ansible 8y ago

I'm wondering now if MCAS should be an active system. Perhaps it should instead just warn the pilots that it thinks a stall is imminent instead, like when you get the stick shaking.

jhayward 8y ago
As I understand it, that would require specific training for pilots of the MAX 8, and (not sure) perhaps a new type certification. Those are expensive and it was a specific goal to avoid them.
- magnamerc 8y ago
  Having to get a new type certificate would likely kill the MAX. I work in aircraft certification, and I know it would be a very long and slow process on the order of years.
  - jhayward 8y ago
    Not only is it a long and expensive process, but the 737 type cert grandfathers in a whole lot of things that would not be permitted in a new type. So they would have to re-engineer a brand new plane from the wheels up. Keeping the 737 type certification was a powerful incentive. Oversight investigation should look in to whether it should have been permitted by the regulator, and what other perverse incentives may exist in the certification process.
Someone1234 8y ago
They cannot do that, because then the Max loses its shared type rating due to the larger engines. MCAS exists to give the aircraft more similar flight characteristics to its predecessors.
- tzs 8y ago
  MCAS apparently exists for more than that. A few days ago someone posted a link and some quotes from the FAA regulations on pitch stability. They require that the control force must be positive up to and throughout a stall. Without MCAS there would be a point on that MAX where you no longer needed positive control force to continue pitching up. Without MCAS or some other system to automatically counter that, so that the pilot needs positive control force to continue pitching up, the plane could not be certified even if they did go for a new type rating.

Lind5 8y ago

Sadly, another "the sensor did it" scenario. https://semiengineering.com/dirty-data-is-the-sensor-malfunctioning/ https://semiengineering.com/dirty-data-is-the-sensor-malfunc... The tragic example of the Lion Air crash of a new Boeing MAX 8 aircraft, which on Oct. 29 killed all aboard, may be heading toward “the sensor did it” category. The black box recovered from the flight showed inconsistent data from one of the two angle-of-attack (AOA) sensors. With one half of the data apparently incorrect, it was enough to trigger this plane’s anti-stall system into a nose down action, which the pilots wrestled all the way into the Java sea.

05 8y ago
MCAS is not an ‘anti-stall system’, it’s a cheap type certification compliance hack
- Doxin 8y ago
  It’s a cheap type certification compliance hack that prevents stalls.
deleted 8y ago
[deleted]

digikata 8y ago

All equipment on the plane has a potential for failure... the question is what systematic design and pilot training is in place to take action in the face of failures.

iscrewyou 8y ago
I don’t think it’s fair to put pilot training and systematic design right next to each other. If the design in itself, and the subsequent factors of safety are at fault, pilots can’t recover from that fast enough. Why put them in a dangerous situation and expect them to recover. How long could this go on even if we didn’t have these two plane crashes. The pilots/airlines could be at fault for not training too much but the bigger issue is the system in the question. It’s like saying the drivers don’t know how to drive because the manufacturer didnt put it front and center on the driver’s manual that the car suddenly swerves right. As ic drivers should always have their hands tight on the wheels and be ready for situations like that. It’s not fair.
- digikata 8y ago
  The pilot training is part of the overall systematic design, neither the equipment, nor the pilots exist alone.
peteradio 8y ago
What? Does the degree of failure not matter at all? That makes no sense. The question is whether all these parts are actually meeting the specified failure tolerance rate.
- digikata 8y ago
  It matters, but if the rate of failure is high enough to be expected to be encountered within an all-fleet, constant flight environment for the lifetime of the aircraft, then you plan for failures as well as having the pilot know what to do when there is a failure. In air sensors like this, their failures are higher than the rate at which you can ignore. They fail on their own, they get bumped and bent on the ground, they ice over in the air, or have objects striking them.

velox_io 8y ago

I'm just amazed by how poorly this was implemented. Software engineers expect hardware to fail. This wasn't just the work of a newb (or I hope it wasn't). This was planned; both the requirements and the failure modes. Then implemented by another team, and the quality verified by another team, even a 3rd party. There wasn't even a cutoff if it started operating outside it's expected range, surely one of those people would have noticed the shortfalls. This software should have the same level of engineering as other systems/ parts of the plane. Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Is this a wider issue with software development in general? My guess is that many businesses are moving away from the waterful model, which can often go overboard with planning (not really a problem in aerospace..). To 'Agile', which I don't think many people truly understand (especially some managers), often sidestepping the planning and documentation to software that appears to work fine, and senior management are overjoyed with the progress and become complacent. This software wasn't even a crucial feature. It was only needed when high power is applied to the engines (like during takeoff) to stop the nose feeling too light by existing 737 pilots. This wasn't a jet-fighter, which would become unstable without the aid of computers. It's a shame 338 died because of what is effectively a 'schoolboy error'.

js2 8y ago
> Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! Pitch != AoA. https://aviation.stackexchange.com/questions/2287/can-the-pitch-be-very-different-from-the-angle-of-attack https://aviation.stackexchange.com/questions/2287/can-the-pi... http://www.aerospaceweb.org/question/aerodynamics/q0165.shtml http://www.aerospaceweb.org/question/aerodynamics/q0165.shtm...
yutyut 8y ago
>Every phone I've had in the last decade has come with GPS (which includes altitude), giros and even magnetic compasses. So they can work out the angle of attack without the need for additional sensors, or at the very least a backup! You would need a sensor to measure relative wind. Angle of attack is the angle between the wing and the relative wind.

bookofjoe 8y ago

The elephant in the room, which I've not yet seen alluded to: Boeing wants at all costs (which will be very high) to avoid having pilots be required to requalify with new MCAS software in flight simulators.

lurker0094 8y ago

So, 140 failures in 30 years of aviation... ~4 failures per year FOR ALL FLIGHTS IN A YEAR. That's unreliable?

peteradio 8y ago
140 incidents. Untold number of averted failures.

cmurf 8y ago

Cessna 150/152/172 stall warning "horn" works by negative pressure. https://www.youtube.com/watch?v=Q1JRTCBkWKQ https://www.youtube.com/watch?v=Q1JRTCBkWKQ And then the Cessna 182 has an aerodynamic switch that activates an electric "horn". And in either case they can get jammed up with bugs. But we do a lot of training with slow flight, approach to stall, full stall, and recoveries including recovery from the ensuing dive and secondary stalls.

ggm 8y ago

Root cause analysis: excessive pressure to keep type rating caused lax engineering to be considered acceptable. Fix:let the FAA do it's job independently

mjevans 8y ago
Fix: Make Engineers actually liable for the correct classification of systems and require signed justification for NOT defaulting to the highest possible standard.
- ggm 8y ago
  ...which demands a fully independent FAA. Not the one we seem to have which appears on the face of it, compromised by the consequences of embedded self assessment and 'same type' rules. Why was it in the national interest to have Boeing get 'same type' ? Because of the cost of competing with airbus. The regulator was hamstrung in the interests of 'efficiency' (I'm not an American and this is an argument i read which resonates. I'd welcome good rebuttal)
- rusticpenn 8y ago
  This also causes a lot of over engineering. ( Which maybe good for Boeing right about now)
  - ggm 8y ago
    I'd dispute it causes the over-engineering. Engineering principles cause safety margins, and regulators enforce safety margins. As the industry decides it can live with reduced safety margins they get reduced. When they get too close to a risk window, they get raised. I know the over-engineering term exists, but I think it's based on flawed logic. Not having margins results in under-engineering, which is significantly worse in almost all cases. Its not a symmetric world out there. Some things are worse than others.
zepearl 8y ago
On a technical level: personally I think that the root cause could be a lack of "integration" of the system (MCAS) with the rest. Meaning that e.g. 1) if the pilots' yokes/sticks are being pulled and 2) if the altitude based on gps and/or air pressure and/or radar is decreasing and 3) if the speed registered by the airspeed probe is high respectively increasing and 4) if the fuel being used by the engines to keep or even increase the speed is very low and 5) if the G-force registered by <some sensor, if it exists?> is lower than usual (neutral or even negative?) 6) etc... (anything else excluding AoA?) ...then the airplane is mooost probably "going down", even if the AoA says the opposite. As the MCAS auto-trim has apparently a big potential impact on how the airplane flies (surfaces on the tail are big and their angle can move a lot), it should therefore not rely on just 1 (or even 2 duplicate) lonely sensor(s) but on multiple types that provide multiple point-of-views of the aircraft's situation to then abstract/determine/confirm the current high-level situation (in this case "is the aircraft going up or down?") and only after such result to react/take countermeasures to save the aircraft respectively help the pilots and/or lower costs for the flight. If this is not totally wrong, is this (the total integration of all sensors) how Airbus did it with their super-automated systems like e.g. their side-sticks/joysticks just telling the airplane what pilots "want"? (meaning, did they start already from scratch creating a totally integrated system, if they did create such an integrated system?) (not sure about how automated the new Boeing airplanes are, so I did not mention them...)

cmurf 8y ago

Too much is being made about the sensors. From my pilot's perspective it's far worse that: a.) MCAS can mistrim the airplane faster than pilots can even recognize what's going on. b.) Boeing and FAA directives still don't account for a.), and therefore they're still giving pilot's inadequate advice. And it's sufficiently bad that I think it's intentional because the advice is consistent with prior Boeing 737 models: runaway stabilizer. If they came up with a better checklist to run that only applies to the MAX, that is suggestive that at least difference training is needed, and might even suggest a different type certificate is needed. Much of this is discussed here which I found to be decent: https://www.satcom.guru/2019/04/what-happened-on-et302.html https://www.satcom.guru/2019/04/what-happened-on-et302.html incredibly it just dumps the stab and leaves it to the pilot to get it back. Electric trim must be used to neutralize control column pitch forces before cutout!

ngcc_hk 8y ago

3 is the minimum for any critical system.