33 ms·

Lack of redundancies on Boeing 737 MAX baffles some involved in developing it

136 points by bobsil1 8y ago

js2 8y ago
Despite the headline, the article ends with: That triple-sensor system isn’t foolproof, however. In 2008, on a customer-acceptance flight of an Airbus A320, two of the angle-of-attack sensors froze and those two sensors then outvoted the third. When the pilots went to demonstrate the stall-prevention system, they were not aware of the malfunctioning sensors. The plane crashed, killing the seven people on board. The same problem arose again on a 2014 Airbus A321 Lufthansa flight leaving Spain. Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. This time, after a drop in altitude, the pilots were able to regain control and complete the flight. Any automation system can fail, and when that happens, hope that through some combination of luck, skill, experience, and training, the pilot will get the plane safely to the destination.
- brokenmachine 8y ago
  They need triple temperature sensors on those sensors to sense when they freeze...
  - brucemoose 8y ago
    you're hired!
    - dragonsky67 8y ago
      Ah, it's turtles all the way down.
  - dingaling 8y ago
    Although you're being facetious, AoA sensors do have self-diagnostics that determine when their heating elements have failed. Incodentally on the A380 and A350 Airbus have changed to combined AoA-pitot sensors of which there are now four.
    - magduf 8y ago
      Sounds like Airbus airplanes are much, much safer to fly on than Boeings. Why would anyone trust a company which thinks it's OK to have a flight control system that only uses input from a single sensor that's known to fail?
- argd678 8y ago
  It does seem like a primitive system to use the wind for AoA, seems like there should be better and more reliable ways to determine it. Update: AoA not AoT
  - TylerE 8y ago
    That's what AoA is though. It's the relative wind angle over the wing. The wind matters. You can't calculate AoA simply from pitch or vertical speed or something like that.
    - inferiorhuman 8y ago
      IIRC Boeing calculates a synthetic AoA value from the inertial reference unit in the 777 and 787, so you can derive it.
  - deleted 8y ago
    [deleted]
  - sollibro 8y ago
    Is there another method to measure wind direction, that doesn't involve a small physical winglet?
    - WalterBright 8y ago
      Another way would be to stick a tab in the slipstream, and measure the pressure on its leading edge and both sides. Or just put a strain gauge on the mount of the tab and measure the bending forces (and this would have no ports to freeze or moving parts to jam). I bet there are lots of ways. P.S. I bet you could even put a strain gauge on the wing spar to measure the bending force on the wing, couple that with other inputs like airspeed, and come up with a reasonable AoA value.
      - dingaling 8y ago
        The problem with such approaches is that you have to model all the environmental and performance factors that could cause an identical pseudo-AoA to be calculated, in all flight modes. And to do that you need a range of sensors, all redundant... so now you need additional input validation and sanitisation. For example, measuring slipstream over the wing is not sufficient. You need to known the pressure ratio between above and below, so there's two sensors. But airflow over a wing is not contiguous or straight, so you need more spanwise sensors to start averaging. And the inboard wing will stall first and give erroneous data, so you need to detect that and disregard it... Rather than just measuring the raw value at source.
- taneq 8y ago
  I've had to do a bit of functional safety stuff recently and one of the things they focus on are "common cause failures" (of which this is a perfect example - if one of the sensors freezes, the others are likely to do so as well). If you want to be sure something works, it's not enough to just have two or three copies of the same system. You have to have redundancy in manner of operation as well.
  - inamberclad 8y ago
    To be specific, I'd call that functional redundancy, versus hardware redundancy. Be able to get the same thing done in different ways. NASA's Genesis mission was bit by this a while back. It had redundant parachute sequencers for a sample return capsule, but they were all assembled identically and with the G-sensors backwards. Consequently, they all failed in exactly the same way.
- dragonsky67 8y ago
  If there are three of the same type (make/model?) of sensor teaming to vote on the status being measured I'm not suprised that they can suffer the same failure (iceing) at the same time causing an incorrect state to be upvoted. I would have thought that for any condition there would be more than one mechanism available to sense it. You could get different devices to vote on what is the real condition. This has been done for years in visual cockpits with regards to an artifical horizon. The pilot only has to look out the window (assuming visibliity) to confirm what the instrument is reporting. Of course when the pilot loses visibility and starts to rely on their inner ear for "level" bad things can happen. Many more simple planes, especially things like gliders, have a simple ball bearing in a curved tube to indicate "down" as well as the more convential artifical horizon.
  - danaliv 8y ago
    Are you talking about a turn coordinator? That doesn't show "down." The sum of forces acting in a coordinated turn will keep the ball centered even though the airplane is banked. Also I'm not sure what you're talking about with using the real horizon as a backup to the artificial horizon. It's not meant to be a redundant setup. Artificial horizons aren't required for visual (looking outside) flight, and relying primarily on the instruments in visual flight is an error. In clouds/poor visibility, the cross-check for the artificial horizon is provided by the other flight instruments: airspeed, altimeter, turn coordinator, vertical speed indicator, and directional gyro (or their equivalents on an electronic display).
    - dragonsky67 8y ago
      The point is that you are using a different type of device to cross check the reading from the first. In the case of the faulty sensor, it sounds like they are using the same type of device therefore conditions that impact the effectiveness of that device will likely impact more than one of them.
- leggomylibro 8y ago
  Triple-redundancy isn't 100% foolproof, but it is a lot better than two potentially-faulty sensors. I know this isn't a great example, but triple redundancy is enough of a design cornerstone that Arthur C. Clarke made it one of the defining features of Rendezvous With Rama, a classic sci-fi story about an Oumuamua-like object that people manage to land on: >It was now strikingly apparent that the "city", like so much of Rama, was triplicated. It consisted of three identical, circular complexes or superstructures, rising from a long, oval foundation. Photographs taken from the hub had also indicated that each complex was itself divided into three equal components, like a pie sliced into 120-degree portions. This would greatly simplify the task of exploration; presumably they had to examine only one-ninth of [the city] to see the whole of it. It doesn't exactly sound like a new or controversial idea, you know?
  - WalterBright 8y ago
    The problem with this idea is the "identical" aspect of each redundancy. If it's identical, then it will likely suffer from the same failure mode at the same time as the others. In the case of the AoA freezing, that's exactly what happened. A more robust idea is to have three redundant subsystems, each different.
    - leggomylibro 8y ago
      Yeah, but I get the feeling that it's more of a way to ensure that you probably get at least two components that will 'burn in' well and hopefully not fail too early: https://en.wikipedia.org/wiki/Bathtub_curve https://en.wikipedia.org/wiki/Bathtub_curve Like, you don't expect them to last forever, but you do expect at least 2/3 to meet their lifetime estimates.
      - deathanatos 8y ago
        I don't think it's so much about the lifetime of the component, or whether it eventually fails due to faulty manufacturing or gradual wear and tear. (What the bathtub curve gets at.) It's that if you do make parts that are identical, and you expose them to identical unanticipated circumstances, they're likely to behave identically, and fail identically. I've seen this w/ distributed databases. I overloaded a Cassandra instance once, causing it to OOM. That node failed, but it's distributed, so another node automatically took over, ran the same query, and OOMed, and then the last guy took the query, and very quickly I had 3 dead nodes.
- cletus 8y ago
  Yep, 3 sensors can fail too. Less often however. These sensors exist to solve a problem with the MAX design. Changing and moving the engines increased the likelihood of a stall when the engines could push the nose up (as I understand it). Fine. But here's the kicker: this should be something that pilots should be trained on. They should be aware of how the MAX is different to the previous 737s and know what to do to disable this system if it causes problems. But that might be the end of a common type rating, which is something the airlines (and apparently Boeing) didn't want. The whole 737 MAX situation just looks like a giant clusterf--- that was a kneejerk reaction to the unexpected success of the A320neo to which Boeing had no answer and to avoid years-long development delays and losing more customers to Airbus, it really looks like they made shortcuts. And that's so damaging to their brand it really defies belief.
  - xiphias2 8y ago
    I was hoping to have the damage in the Boing brand reflected more significantly in the share prices.
    - jdc 8y ago
      Maybe the problem hasn't been shouted from a high enough rooftop yet.
      - bdamm 8y ago
        Or more likely it doesn't really matter, and in a couple years nobody will really remember this about Boeing. They'll go back to being a big airplane producer that has a stunning safety record, which they are despite this issue.
        Tsubasachan 8y ago
        The airline industry is growing and there are only two manufacturers who can build hundreds of planes on time. Airbus is already near full capacity. It would be impossible NOT to buy Boeing if you want planes.
- choeger 8y ago
  They should have used (n+1)-th prime sensors, dammit. When will they ever learn?
- gomijacogeo 8y ago
  Although a stuck AoA sensor is also going to have a very characteristic signature - no change in reported AoA. A working AoA sensor will always have some noise in the signal as the air buffets the plane. Similarly, if you maneuver the plane and the indicated AoA doesn't change at all, you can deduce the sensor is stuck and reject it (or turn on the heater, or maybe have a mechanical way to force the AoA sensor through its range of motion, etc).
- rkagerer 8y ago
  I'm admittedly ignorant of aircraft design but years ago when I first found out what those little AoA fins do, I was like, why the heck aren't there more of them plastered all over the place? Like, more than 3.
- magicalhippo 8y ago
  Couldn't one detect the frozen ones by simply noticing the loss of noise in the readings? I imagine an AoA sensor functioning normally to have a small amount of variation between readings due to turbulent air etc. Alternatively, couldn't one correlate the AoA sensor output with the accelerometer output (or some other pitch sensor)? A frozen AoA would have identical readings as the plane changes pitch, a functioning one shouldn't, no?
- DanieI 8y ago
  > Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. If it's typical for the sensor's data to vary from 30.00 to 30.99 degrees as the wind passes over the wings and two sensors are permanently frozen at 10.42, after 8 minutes I think that there might be a high enough probability for the computer's software to decide that two sensors are broken and one is working.
- Cacti 8y ago
  a customer acceptance fight is a shitty way to go
hairytrog 8y ago
The only way to be 100% safe and 100% fool proof is to not fly. For a mechanical system as complex as an airplane, triple redundancy with a good pilot is pretty much the best you can hope for.
- nutjob2 8y ago
  A plane can still fall out of the sky and kill you.
- Tepix 8y ago
  If you want to get somewhere, going by car is a lot more dangerous than flying.
- magduf 8y ago
  Triple redundancy makes planes extremely safe to fly. Airbus has triple redundancy, and Boeing doesn't believe in it because they care more about profits. So the simple answer is to avoid flying on Boeing aircraft, and stick to Airbus ones.
4sak3n 8y ago
"Two is one and one is none." That's pretty much been aeronautical gospel since dinosaurs strapped cardboard to their stubby arms and dreamed of soaring.
- snazzySnappy 8y ago
  Which is to say, if you have redundancy, your thing is reliable, and thus a thing at all. No redundancy renders your life saving equipment worthless. But not, by corollary, that two is none.
- 4sak3n 8y ago
  To elaborate, I'm pointing out that redundancy has been an integral part of aeronautical engineering for ages so to ignore it in any situation, never mind in one where the pilots are kept out of the decision making loop, is criminally negligent.
ptidhomme 8y ago
What I fail to understand is why do they not design an overriding scheme to MCAS, such as an "above-threshold" pull on the stick by the pilot. I can recall that this was a classic design by Boeing. Whatever the number of AoA sensors, the system should be easy and straightforward to disable. Those logics of "let's guess automatically if that sensor is failing" are just pushing the problem further. The real problem is too much reliance on automatic systems and top little human integration in the overall system (the human-machine system, that is to say).
- nutjob2 8y ago
  The problem isn't too much reliance on automatic systems, it's that the systems are a half baked combination of the two. The MCAS system is an example of that. As you say that overriding design is typical in Boeing planes, but MCAS purposefully overrides it, that is, pushing the controls hard, which overrides other automatic functions, has no effect on MCAS.
  - ptidhomme 8y ago
    I'd say this is why they are half-baked : human integration was not thouroughly thought of in the design. When I speak of overriding, I mean human overriding. This is something I think Boeing had got right with its manual overrides : when you design critical systems, you need human backup schemes, but you can go further and make these backup tightly integrated with the rest of the (highly automated) system.
    - CaptainZapp 8y ago
      I'd say this is why they are half-baked : human integration was not thouroughly thought of in the design. When I speak of overriding, I mean human overriding. That's pretty obvious, given by the fact that the system was initially not even documented nor was the scenario allegedly available for training in the brand spanking new 787-MAX simulator.
- ishan1121 8y ago
  The airline manufacturers have to push for automatic systems as they are the one of the features that sells. One of a close friend of mine who works at Airbus said that the company believes the next gen aircrafts would only require a single person in the cockpit, and everything will be handled through automations. Airbus aircrafts are more automated than Boeing. If Boeing does not work on the automation part, it could loose to Airbus. Airbus faced AoA problem back in 2009, when Quantas flight suddenly dropped from the air. But they were able to patch it up within a month and no reported case has come up since then. So I believe Boeing working on Automation is a good thing, and they should be able to fix it up.
bb101 8y ago
It's worth watching Al Jazeera's series on the production of the 787 Dreamliner, and related cultural issues at Boeing after their merger with McDonnell Douglas [1]. Personally, I think it's worth taking with a grain of salt as it appears quite one-sided, but the interviews with Boeing ex-employees and head engineers are enlightening, and could go some way to explaining the lack of redundant systems (i.e. cost and time). [1] https://www.youtube.com/watch?v=rvkEpstd9os https://www.youtube.com/watch?v=rvkEpstd9os
jussij 8y ago
From the information that keeps leaking out it appears the 737 MAX has some serious design faults. What is going to interesting is to see how quickly the FAA certifies the new software changes this second time around. The current perception would be the FAA was too quick to certifying the original MCAS as being safe. I suspect the second time around that certification process is going to take a lot more time and effort and is going to be much harder to achieve. And the final nail in the 737 MAX coffin might be that even when that certification arrives, Boeing then needs to convince paying passenger that is now safe to climb on board the aircraft.
- magduf 8y ago
  >What is going to interesting is to see how quickly the FAA certifies the new software changes this second time around. With a Trump lackey running the FAA, I'm sure they'll rubber-stamp whatever Boeing puts out there. What I want to see is if any of the foreign regulators refuse to certify it. I'd love to see a show-down in international aviation between the Trump administration and better-run nations.
crocal 8y ago
I am a designer and implementor of vital speed and distance measurement systems, with triply modular redundancy, used in mass transit application. I cannot even imagine what the designers of this thing are going through now. It must be terrible. To make it worse, it's a confusing topic. There are two pillars to the design of such system. 1. Faulty sensor must be detected with a very high probability. The typical way to achieve that is redundancy and diversification. The exact amount of redundancy depends on the reliability of the sensor considered. In most cases, it is sufficient to have 2 sensors, but of different models, in order to avoid common mode of failure. 2. In case of a failure, the system must have a graceful degradation, and in aeronautics, this means a clean handover to the pilot. So, in the case of this MCAS thing, having two sensors is not necessarily a bad thing, and what M. Kornecki reports is 100% correct. What looks strange is the way a single failure was managed by the software, and how the procedure to recover was quite complicated and, even worse, not exported to the training material of the pilots. In my world, we called this "exported safety requirement application conditions" - SRACs, and verification of their proper allocation is a big chunk of the safety case. More than discussing architecture, in my view, the investigation must explain why the organisation failed to perform this activity. The case for a third sensor can be made to decrease the likelihood of having to bypass the system ("belt and suspenders", as says M. Kornacki), but based on my experience, it will not be sufficient. As other correctly report here, it's not a silver bullet. Ultimately, the degradation scenario and pilot handover is part of the overall system safety.
- devit 8y ago
  How expensive are these sensors compared to the cost of the plane? Any reason to not just use at least 9 of them, pick the median value and forget about it?
  - consto 8y ago
    It's incredibly unlikely, but not entirely unforseeable that enough sensors would fail at the same time for the same reason. All four engines on the 747 BA9 failed at the same time due to volcanic ash [0]. If you just take the median of nine, it's possible that a mere five failed sensors could break MCAS. And then you are back to the issue of handing over to the pilots. [0] https://en.m.wikipedia.org/wiki/British_Airways_Flight_9 https://en.m.wikipedia.org/wiki/British_Airways_Flight_9
OscarTheGrinch 8y ago
I'm gonna take a punt on the root cause of this mess: cost cutting / bonus seeking managerialisim. In my experience people high up in the decision chain with no technical expertise often have blind faith in what "technology" can accomplish.
- cjbprime 8y ago
  For what it's worth, "take a punt on" means "avoid dealing with", which sounds like the opposite of your intent.
  - OscarTheGrinch 8y ago
    I was more going for the "make a general guess / bet" kind of meaning, which is how the phrase is used today in England / Australia / New Zealand. https://english.stackexchange.com/questions/20742/why-did-this-brit-say-took-a-punt https://english.stackexchange.com/questions/20742/why-did-th... Thanks for the heads up tho, I will retire it from my rhetorical toolkit in favor of something more universally understood.
    - cjbprime 8y ago
      Oh, thanks for the info! Rare to see a phrase with antonymic regional differences ("I'll try it / I won't try it") survive in today's globalized world etc etc.
jackson1way 8y ago
I have a question since many years that I am afraid to ask so straight. There is the AF447 and those 2 Boeing MAX that I remember where people will always talk about redundancy of the sensors, or their maintenance. But really, I want to know why the software and hardware of the aircraft fails to understand that whatever actions it has performed automatically or inputs/actions by the pilots, or any other important event, are making the damn aircraft loose altitude really really fast. In the case of these Boeings, how can your software keep pointing the nose of the aircraft down WHILE it is CONSISTENTLY loosing altitude? How is this not a huge design flaw? I understand that for stall recovery you actually have to loose some altitude to regain speed/lift and recover from the stall. But if you STILL KEEP loosing altitude and there is no turning point, then damnit, whatever caused the loosing altitude should be stopped at least at 500m AGL, no? Switch on a huge yellow light to the pilots „you are 100% on your own now, the computer is out of luck and is shutting down“ and the pilot jumps on the gas pedal, 200% power to the engines, and starts to figure out how to gain altitude. Both Boeing flights were in daylight, no? Pilots should be able to see the ground/water and be able to pull the thing back up just by visibility without any sensors? It‘s a bit of a different case with the AF447, though. How did the AoA sensors and MCAS won the authority here over the only important data: altitude change. Who cares if the AoA shows 30 degrees or -271 degrees, as long as the plane is climbing we are good! No need to point the nose down or do any other stall protection stuff? And that seems to be really what happened with these 2 Boeings. They were climbing but some stupid AoA sensor failed and some even more stupid MCAS decided to dive. And they kept diving until they reached the ocean and MCAS was still confident it was a good idea to dive? In some planes my iPhone is able to get the GPS reading and some free app I got 5 years ago will actually show me ground speed and altitude. Which I always find very exciting. It usually matches with what the entertainment system is showing on the pax screens. Give or take <1%. So reliable altitude reading is solved, no? You have 300 phones on your plane (though only the window seats have a chance for it to work, but still...). Not sure if the GPS is gonna work if the plane is rolling and spinning like crazy, which I dont think happened in any of these cases. I hope I don‘t make it sound stupid or even i-know-it-better, but I really just want to understand.
- whatshisface 8y ago
  I only know a little bit about aerospace programming, but from what I do know complicated chains-of-reasoning are avoided like the plague. With every additional step of program reasoning the probability that you got it right decays exponentially. As a result, simple controllers with clearly defined areas of responsibility are elevated above all else. It is the pilot's job to integrate the information coming to them from all areas of the plane. Yes, a computer could do it, but once you get past three or so sensors the programmer's ability to reason about what might happen shrinks to nothing in comparison to the pilot's ability to see what is happening.
FabHK 8y ago
> “Our proposed software update incorporates additional limits and safeguards to the system and reduces crew workload,” Boeing said in a statement. ... "reduces crew workload" so that they have a chance to figure it out in time and survive. Nice move. EDIT to add: I guess I find it a bit annoying that, instead of admitting that they're fixing a terrible bug, they're cloaking it in some corporate "we're adding even more cool new features" speak.
FabHK 8y ago
The crux of the matter: > if the group had built the MCAS in a way that would depend on two sensors, and would shut the system off if one fails, he thinks the company would have needed to install an alert in the cockpit to make the pilots aware that the safety system was off. > And if that happens, Ludtke said, the pilots would potentially need training on the new alert and the underlying system. That could mean simulator time, which was off the table.
- LarryDarrell 8y ago
  It seems like we are determined to live in a Gilded Age Fantasy Camp. It's like the Titanic not having enough lifeboats. I've been around the block. I know Big Corp will always place value on profits over safety, but we seem to be entering an age where they feel unconstrained.
- gameswithgo 8y ago
  even before that: the plane is a bad design because making a new plane that works properly with the bigger engines was also off the table. so hundreds of people die.
havkom 8y ago
I have no idea of the facts in this case, but based on how organizations function in general, one possibility is that the redundancy issues of MCAS were raised internally by some engineers without result. In that case, the engineers were probably not thanked internally back then and are probably not thanked now either. If their potential concerns would had led to changes on the MCAS design they would probably not have been thanked either, but rather seen as the people who unnecessarily made the project more expensive or less profitable. In conclusion, people who avert or try to avert risk are seldom proportionally thanked in organizations in general.
- linuxftw 8y ago
  > MCAS were raised internally by some engineers I think it's quite generous to Boeing to imply the MCAS system was developed in house. I'm leaning towards 3rd party contract, built to Boeing's spec. Whoever wrote the spec might be 3rd party as well, with Boeing's 'engineers' just performing as project managers.
ken 8y ago
> "A properly trained pilot should be able to solve an MCAS anomaly or any uncommanded flight-control input through procedures that are taught to all 737 pilots" It blows my mind that anyone still talks like this. My first job was writing (non-life-critical) scientific software for detail-obsessed university researchers working in a lab with notebooks and procedures. If "just solve it with training" could work for anyone in the world, it would have been these people. Yet it was immediately and abundantly clear that designing usability into the system would have far more impact on their success rate at any task than documentation or training or even the reliability of the software itself. The movie "Apollo 13" showed astronauts working in a simulator, and engineers giving them faulty sensor inputs to try to trick them. They had to train to be able to identify this possibility, and react correctly to it. This isn't the sort of thing you can figure out on the fly (so to speak). If the solution to flying an aircraft with a faulty MCAS sensor was "the pilot should have read it in a book last month", it's no wonder they were in trouble. The software industry does make some programs with random inputs and hidden internal state, and require users to figure out what's going on and solve it anyway. We call them "games". Perhaps the defining characteristic of a game is that, even if you've read the manual, you won't succeed on the first try. > "But, he said, if he were designing the system from scratch, he would emphasize the training while also building the plane with three sensors." Is even three enough? From an earlier version of the "2001" Wikipedia page [1]: > "In the story HAL features a design with triple redundancy, so that if one of the three modules fails the other two can outvote it. However, there is a [[theorem]] in [[computer science]] that proves that for such [[distributed systems]] a vote-based [[sanity check]] only works if ''less than'' one third of the modules fail. Thus the failure of a single one of HAL's redundant modules would be sufficient to compromise the system, as apparently happened in the movie." I don't know what theorem this is referring to. Help? See also: Segal's Law. [1]: https://en.wikipedia.org/w/index.php?title=2001:_A_Space_Odyssey_(film)&diff=537589&oldid=530391 https://en.wikipedia.org/w/index.php?title=2001:_A_Space_Ody...
- jacobush 8y ago
  Not only that, but the pilots received no training, to uphold the fiction that the 737 MAX flies the same as the old 737.

136 points by bobsil1 8y ago

js2 8y ago

Despite the headline, the article ends with: That triple-sensor system isn’t foolproof, however. In 2008, on a customer-acceptance flight of an Airbus A320, two of the angle-of-attack sensors froze and those two sensors then outvoted the third. When the pilots went to demonstrate the stall-prevention system, they were not aware of the malfunctioning sensors. The plane crashed, killing the seven people on board. The same problem arose again on a 2014 Airbus A321 Lufthansa flight leaving Spain. Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. This time, after a drop in altitude, the pilots were able to regain control and complete the flight. Any automation system can fail, and when that happens, hope that through some combination of luck, skill, experience, and training, the pilot will get the plane safely to the destination.

brokenmachine 8y ago
They need triple temperature sensors on those sensors to sense when they freeze...
- brucemoose 8y ago
  you're hired!
  - dragonsky67 8y ago
    Ah, it's turtles all the way down.
- dingaling 8y ago
  Although you're being facetious, AoA sensors do have self-diagnostics that determine when their heating elements have failed. Incodentally on the A380 and A350 Airbus have changed to combined AoA-pitot sensors of which there are now four.
  - magduf 8y ago
    Sounds like Airbus airplanes are much, much safer to fly on than Boeings. Why would anyone trust a company which thinks it's OK to have a flight control system that only uses input from a single sensor that's known to fail?
argd678 8y ago
It does seem like a primitive system to use the wind for AoA, seems like there should be better and more reliable ways to determine it. Update: AoA not AoT
- TylerE 8y ago
  That's what AoA is though. It's the relative wind angle over the wing. The wind matters. You can't calculate AoA simply from pitch or vertical speed or something like that.
  - inferiorhuman 8y ago
    IIRC Boeing calculates a synthetic AoA value from the inertial reference unit in the 777 and 787, so you can derive it.
- deleted 8y ago
  [deleted]
- sollibro 8y ago
  Is there another method to measure wind direction, that doesn't involve a small physical winglet?
  - WalterBright 8y ago
    Another way would be to stick a tab in the slipstream, and measure the pressure on its leading edge and both sides. Or just put a strain gauge on the mount of the tab and measure the bending forces (and this would have no ports to freeze or moving parts to jam). I bet there are lots of ways. P.S. I bet you could even put a strain gauge on the wing spar to measure the bending force on the wing, couple that with other inputs like airspeed, and come up with a reasonable AoA value.
    - dingaling 8y ago
      The problem with such approaches is that you have to model all the environmental and performance factors that could cause an identical pseudo-AoA to be calculated, in all flight modes. And to do that you need a range of sensors, all redundant... so now you need additional input validation and sanitisation. For example, measuring slipstream over the wing is not sufficient. You need to known the pressure ratio between above and below, so there's two sensors. But airflow over a wing is not contiguous or straight, so you need more spanwise sensors to start averaging. And the inboard wing will stall first and give erroneous data, so you need to detect that and disregard it... Rather than just measuring the raw value at source.
taneq 8y ago
I've had to do a bit of functional safety stuff recently and one of the things they focus on are "common cause failures" (of which this is a perfect example - if one of the sensors freezes, the others are likely to do so as well). If you want to be sure something works, it's not enough to just have two or three copies of the same system. You have to have redundancy in manner of operation as well.
- inamberclad 8y ago
  To be specific, I'd call that functional redundancy, versus hardware redundancy. Be able to get the same thing done in different ways. NASA's Genesis mission was bit by this a while back. It had redundant parachute sequencers for a sample return capsule, but they were all assembled identically and with the G-sensors backwards. Consequently, they all failed in exactly the same way.
dragonsky67 8y ago
If there are three of the same type (make/model?) of sensor teaming to vote on the status being measured I'm not suprised that they can suffer the same failure (iceing) at the same time causing an incorrect state to be upvoted. I would have thought that for any condition there would be more than one mechanism available to sense it. You could get different devices to vote on what is the real condition. This has been done for years in visual cockpits with regards to an artifical horizon. The pilot only has to look out the window (assuming visibliity) to confirm what the instrument is reporting. Of course when the pilot loses visibility and starts to rely on their inner ear for "level" bad things can happen. Many more simple planes, especially things like gliders, have a simple ball bearing in a curved tube to indicate "down" as well as the more convential artifical horizon.
- danaliv 8y ago
  Are you talking about a turn coordinator? That doesn't show "down." The sum of forces acting in a coordinated turn will keep the ball centered even though the airplane is banked. Also I'm not sure what you're talking about with using the real horizon as a backup to the artificial horizon. It's not meant to be a redundant setup. Artificial horizons aren't required for visual (looking outside) flight, and relying primarily on the instruments in visual flight is an error. In clouds/poor visibility, the cross-check for the artificial horizon is provided by the other flight instruments: airspeed, altimeter, turn coordinator, vertical speed indicator, and directional gyro (or their equivalents on an electronic display).
  - dragonsky67 8y ago
    The point is that you are using a different type of device to cross check the reading from the first. In the case of the faulty sensor, it sounds like they are using the same type of device therefore conditions that impact the effectiveness of that device will likely impact more than one of them.
leggomylibro 8y ago
Triple-redundancy isn't 100% foolproof, but it is a lot better than two potentially-faulty sensors. I know this isn't a great example, but triple redundancy is enough of a design cornerstone that Arthur C. Clarke made it one of the defining features of Rendezvous With Rama, a classic sci-fi story about an Oumuamua-like object that people manage to land on: >It was now strikingly apparent that the "city", like so much of Rama, was triplicated. It consisted of three identical, circular complexes or superstructures, rising from a long, oval foundation. Photographs taken from the hub had also indicated that each complex was itself divided into three equal components, like a pie sliced into 120-degree portions. This would greatly simplify the task of exploration; presumably they had to examine only one-ninth of [the city] to see the whole of it. It doesn't exactly sound like a new or controversial idea, you know?
- WalterBright 8y ago
  The problem with this idea is the "identical" aspect of each redundancy. If it's identical, then it will likely suffer from the same failure mode at the same time as the others. In the case of the AoA freezing, that's exactly what happened. A more robust idea is to have three redundant subsystems, each different.
  - leggomylibro 8y ago
    Yeah, but I get the feeling that it's more of a way to ensure that you probably get at least two components that will 'burn in' well and hopefully not fail too early: https://en.wikipedia.org/wiki/Bathtub_curve https://en.wikipedia.org/wiki/Bathtub_curve Like, you don't expect them to last forever, but you do expect at least 2/3 to meet their lifetime estimates.
    - deathanatos 8y ago
      I don't think it's so much about the lifetime of the component, or whether it eventually fails due to faulty manufacturing or gradual wear and tear. (What the bathtub curve gets at.) It's that if you do make parts that are identical, and you expose them to identical unanticipated circumstances, they're likely to behave identically, and fail identically. I've seen this w/ distributed databases. I overloaded a Cassandra instance once, causing it to OOM. That node failed, but it's distributed, so another node automatically took over, ran the same query, and OOMed, and then the last guy took the query, and very quickly I had 3 dead nodes.
cletus 8y ago
Yep, 3 sensors can fail too. Less often however. These sensors exist to solve a problem with the MAX design. Changing and moving the engines increased the likelihood of a stall when the engines could push the nose up (as I understand it). Fine. But here's the kicker: this should be something that pilots should be trained on. They should be aware of how the MAX is different to the previous 737s and know what to do to disable this system if it causes problems. But that might be the end of a common type rating, which is something the airlines (and apparently Boeing) didn't want. The whole 737 MAX situation just looks like a giant clusterf--- that was a kneejerk reaction to the unexpected success of the A320neo to which Boeing had no answer and to avoid years-long development delays and losing more customers to Airbus, it really looks like they made shortcuts. And that's so damaging to their brand it really defies belief.
- xiphias2 8y ago
  I was hoping to have the damage in the Boing brand reflected more significantly in the share prices.
  - jdc 8y ago
    Maybe the problem hasn't been shouted from a high enough rooftop yet.
    - bdamm 8y ago
      Or more likely it doesn't really matter, and in a couple years nobody will really remember this about Boeing. They'll go back to being a big airplane producer that has a stunning safety record, which they are despite this issue.
      - Tsubasachan 8y ago
        The airline industry is growing and there are only two manufacturers who can build hundreds of planes on time. Airbus is already near full capacity. It would be impossible NOT to buy Boeing if you want planes.
choeger 8y ago
They should have used (n+1)-th prime sensors, dammit. When will they ever learn?
gomijacogeo 8y ago
Although a stuck AoA sensor is also going to have a very characteristic signature - no change in reported AoA. A working AoA sensor will always have some noise in the signal as the air buffets the plane. Similarly, if you maneuver the plane and the indicated AoA doesn't change at all, you can deduce the sensor is stuck and reject it (or turn on the heater, or maybe have a mechanical way to force the AoA sensor through its range of motion, etc).
rkagerer 8y ago
I'm admittedly ignorant of aircraft design but years ago when I first found out what those little AoA fins do, I was like, why the heck aren't there more of them plastered all over the place? Like, more than 3.
magicalhippo 8y ago
Couldn't one detect the frozen ones by simply noticing the loss of noise in the readings? I imagine an AoA sensor functioning normally to have a small amount of variation between readings due to turbulent air etc. Alternatively, couldn't one correlate the AoA sensor output with the accelerometer output (or some other pitch sensor)? A frozen AoA would have identical readings as the plane changes pitch, a functioning one shouldn't, no?
DanieI 8y ago
> Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. If it's typical for the sensor's data to vary from 30.00 to 30.99 degrees as the wind passes over the wings and two sensors are permanently frozen at 10.42, after 8 minutes I think that there might be a high enough probability for the computer's software to decide that two sensors are broken and one is working.
Cacti 8y ago
a customer acceptance fight is a shitty way to go

hairytrog 8y ago

The only way to be 100% safe and 100% fool proof is to not fly. For a mechanical system as complex as an airplane, triple redundancy with a good pilot is pretty much the best you can hope for.

nutjob2 8y ago
A plane can still fall out of the sky and kill you.
Tepix 8y ago
If you want to get somewhere, going by car is a lot more dangerous than flying.
magduf 8y ago
Triple redundancy makes planes extremely safe to fly. Airbus has triple redundancy, and Boeing doesn't believe in it because they care more about profits. So the simple answer is to avoid flying on Boeing aircraft, and stick to Airbus ones.

4sak3n 8y ago

"Two is one and one is none." That's pretty much been aeronautical gospel since dinosaurs strapped cardboard to their stubby arms and dreamed of soaring.

snazzySnappy 8y ago
Which is to say, if you have redundancy, your thing is reliable, and thus a thing at all. No redundancy renders your life saving equipment worthless. But not, by corollary, that two is none.
4sak3n 8y ago
To elaborate, I'm pointing out that redundancy has been an integral part of aeronautical engineering for ages so to ignore it in any situation, never mind in one where the pilots are kept out of the decision making loop, is criminally negligent.

ptidhomme 8y ago

What I fail to understand is why do they not design an overriding scheme to MCAS, such as an "above-threshold" pull on the stick by the pilot. I can recall that this was a classic design by Boeing. Whatever the number of AoA sensors, the system should be easy and straightforward to disable. Those logics of "let's guess automatically if that sensor is failing" are just pushing the problem further. The real problem is too much reliance on automatic systems and top little human integration in the overall system (the human-machine system, that is to say).

nutjob2 8y ago
The problem isn't too much reliance on automatic systems, it's that the systems are a half baked combination of the two. The MCAS system is an example of that. As you say that overriding design is typical in Boeing planes, but MCAS purposefully overrides it, that is, pushing the controls hard, which overrides other automatic functions, has no effect on MCAS.
- ptidhomme 8y ago
  I'd say this is why they are half-baked : human integration was not thouroughly thought of in the design. When I speak of overriding, I mean human overriding. This is something I think Boeing had got right with its manual overrides : when you design critical systems, you need human backup schemes, but you can go further and make these backup tightly integrated with the rest of the (highly automated) system.
  - CaptainZapp 8y ago
    I'd say this is why they are half-baked : human integration was not thouroughly thought of in the design. When I speak of overriding, I mean human overriding. That's pretty obvious, given by the fact that the system was initially not even documented nor was the scenario allegedly available for training in the brand spanking new 787-MAX simulator.
ishan1121 8y ago
The airline manufacturers have to push for automatic systems as they are the one of the features that sells. One of a close friend of mine who works at Airbus said that the company believes the next gen aircrafts would only require a single person in the cockpit, and everything will be handled through automations. Airbus aircrafts are more automated than Boeing. If Boeing does not work on the automation part, it could loose to Airbus. Airbus faced AoA problem back in 2009, when Quantas flight suddenly dropped from the air. But they were able to patch it up within a month and no reported case has come up since then. So I believe Boeing working on Automation is a good thing, and they should be able to fix it up.

bb101 8y ago

It's worth watching Al Jazeera's series on the production of the 787 Dreamliner, and related cultural issues at Boeing after their merger with McDonnell Douglas [1]. Personally, I think it's worth taking with a grain of salt as it appears quite one-sided, but the interviews with Boeing ex-employees and head engineers are enlightening, and could go some way to explaining the lack of redundant systems (i.e. cost and time). [1] https://www.youtube.com/watch?v=rvkEpstd9os https://www.youtube.com/watch?v=rvkEpstd9os

jussij 8y ago

From the information that keeps leaking out it appears the 737 MAX has some serious design faults. What is going to interesting is to see how quickly the FAA certifies the new software changes this second time around. The current perception would be the FAA was too quick to certifying the original MCAS as being safe. I suspect the second time around that certification process is going to take a lot more time and effort and is going to be much harder to achieve. And the final nail in the 737 MAX coffin might be that even when that certification arrives, Boeing then needs to convince paying passenger that is now safe to climb on board the aircraft.

magduf 8y ago
>What is going to interesting is to see how quickly the FAA certifies the new software changes this second time around. With a Trump lackey running the FAA, I'm sure they'll rubber-stamp whatever Boeing puts out there. What I want to see is if any of the foreign regulators refuse to certify it. I'd love to see a show-down in international aviation between the Trump administration and better-run nations.

crocal 8y ago

I am a designer and implementor of vital speed and distance measurement systems, with triply modular redundancy, used in mass transit application. I cannot even imagine what the designers of this thing are going through now. It must be terrible. To make it worse, it's a confusing topic. There are two pillars to the design of such system. 1. Faulty sensor must be detected with a very high probability. The typical way to achieve that is redundancy and diversification. The exact amount of redundancy depends on the reliability of the sensor considered. In most cases, it is sufficient to have 2 sensors, but of different models, in order to avoid common mode of failure. 2. In case of a failure, the system must have a graceful degradation, and in aeronautics, this means a clean handover to the pilot. So, in the case of this MCAS thing, having two sensors is not necessarily a bad thing, and what M. Kornecki reports is 100% correct. What looks strange is the way a single failure was managed by the software, and how the procedure to recover was quite complicated and, even worse, not exported to the training material of the pilots. In my world, we called this "exported safety requirement application conditions" - SRACs, and verification of their proper allocation is a big chunk of the safety case. More than discussing architecture, in my view, the investigation must explain why the organisation failed to perform this activity. The case for a third sensor can be made to decrease the likelihood of having to bypass the system ("belt and suspenders", as says M. Kornacki), but based on my experience, it will not be sufficient. As other correctly report here, it's not a silver bullet. Ultimately, the degradation scenario and pilot handover is part of the overall system safety.

devit 8y ago
How expensive are these sensors compared to the cost of the plane? Any reason to not just use at least 9 of them, pick the median value and forget about it?
- consto 8y ago
  It's incredibly unlikely, but not entirely unforseeable that enough sensors would fail at the same time for the same reason. All four engines on the 747 BA9 failed at the same time due to volcanic ash [0]. If you just take the median of nine, it's possible that a mere five failed sensors could break MCAS. And then you are back to the issue of handing over to the pilots. [0] https://en.m.wikipedia.org/wiki/British_Airways_Flight_9 https://en.m.wikipedia.org/wiki/British_Airways_Flight_9

OscarTheGrinch 8y ago

I'm gonna take a punt on the root cause of this mess: cost cutting / bonus seeking managerialisim. In my experience people high up in the decision chain with no technical expertise often have blind faith in what "technology" can accomplish.

cjbprime 8y ago
For what it's worth, "take a punt on" means "avoid dealing with", which sounds like the opposite of your intent.
- OscarTheGrinch 8y ago
  I was more going for the "make a general guess / bet" kind of meaning, which is how the phrase is used today in England / Australia / New Zealand. https://english.stackexchange.com/questions/20742/why-did-this-brit-say-took-a-punt https://english.stackexchange.com/questions/20742/why-did-th... Thanks for the heads up tho, I will retire it from my rhetorical toolkit in favor of something more universally understood.
  - cjbprime 8y ago
    Oh, thanks for the info! Rare to see a phrase with antonymic regional differences ("I'll try it / I won't try it") survive in today's globalized world etc etc.

jackson1way 8y ago

I have a question since many years that I am afraid to ask so straight. There is the AF447 and those 2 Boeing MAX that I remember where people will always talk about redundancy of the sensors, or their maintenance. But really, I want to know why the software and hardware of the aircraft fails to understand that whatever actions it has performed automatically or inputs/actions by the pilots, or any other important event, are making the damn aircraft loose altitude really really fast. In the case of these Boeings, how can your software keep pointing the nose of the aircraft down WHILE it is CONSISTENTLY loosing altitude? How is this not a huge design flaw? I understand that for stall recovery you actually have to loose some altitude to regain speed/lift and recover from the stall. But if you STILL KEEP loosing altitude and there is no turning point, then damnit, whatever caused the loosing altitude should be stopped at least at 500m AGL, no? Switch on a huge yellow light to the pilots „you are 100% on your own now, the computer is out of luck and is shutting down“ and the pilot jumps on the gas pedal, 200% power to the engines, and starts to figure out how to gain altitude. Both Boeing flights were in daylight, no? Pilots should be able to see the ground/water and be able to pull the thing back up just by visibility without any sensors? It‘s a bit of a different case with the AF447, though. How did the AoA sensors and MCAS won the authority here over the only important data: altitude change. Who cares if the AoA shows 30 degrees or -271 degrees, as long as the plane is climbing we are good! No need to point the nose down or do any other stall protection stuff? And that seems to be really what happened with these 2 Boeings. They were climbing but some stupid AoA sensor failed and some even more stupid MCAS decided to dive. And they kept diving until they reached the ocean and MCAS was still confident it was a good idea to dive? In some planes my iPhone is able to get the GPS reading and some free app I got 5 years ago will actually show me ground speed and altitude. Which I always find very exciting. It usually matches with what the entertainment system is showing on the pax screens. Give or take <1%. So reliable altitude reading is solved, no? You have 300 phones on your plane (though only the window seats have a chance for it to work, but still...). Not sure if the GPS is gonna work if the plane is rolling and spinning like crazy, which I dont think happened in any of these cases. I hope I don‘t make it sound stupid or even i-know-it-better, but I really just want to understand.

whatshisface 8y ago
I only know a little bit about aerospace programming, but from what I do know complicated chains-of-reasoning are avoided like the plague. With every additional step of program reasoning the probability that you got it right decays exponentially. As a result, simple controllers with clearly defined areas of responsibility are elevated above all else. It is the pilot's job to integrate the information coming to them from all areas of the plane. Yes, a computer could do it, but once you get past three or so sensors the programmer's ability to reason about what might happen shrinks to nothing in comparison to the pilot's ability to see what is happening.

FabHK 8y ago

> “Our proposed software update incorporates additional limits and safeguards to the system and reduces crew workload,” Boeing said in a statement. ... "reduces crew workload" so that they have a chance to figure it out in time and survive. Nice move. EDIT to add: I guess I find it a bit annoying that, instead of admitting that they're fixing a terrible bug, they're cloaking it in some corporate "we're adding even more cool new features" speak.

FabHK 8y ago

The crux of the matter: > if the group had built the MCAS in a way that would depend on two sensors, and would shut the system off if one fails, he thinks the company would have needed to install an alert in the cockpit to make the pilots aware that the safety system was off. > And if that happens, Ludtke said, the pilots would potentially need training on the new alert and the underlying system. That could mean simulator time, which was off the table.

LarryDarrell 8y ago
It seems like we are determined to live in a Gilded Age Fantasy Camp. It's like the Titanic not having enough lifeboats. I've been around the block. I know Big Corp will always place value on profits over safety, but we seem to be entering an age where they feel unconstrained.
gameswithgo 8y ago
even before that: the plane is a bad design because making a new plane that works properly with the bigger engines was also off the table. so hundreds of people die.

havkom 8y ago

I have no idea of the facts in this case, but based on how organizations function in general, one possibility is that the redundancy issues of MCAS were raised internally by some engineers without result. In that case, the engineers were probably not thanked internally back then and are probably not thanked now either. If their potential concerns would had led to changes on the MCAS design they would probably not have been thanked either, but rather seen as the people who unnecessarily made the project more expensive or less profitable. In conclusion, people who avert or try to avert risk are seldom proportionally thanked in organizations in general.

linuxftw 8y ago
> MCAS were raised internally by some engineers I think it's quite generous to Boeing to imply the MCAS system was developed in house. I'm leaning towards 3rd party contract, built to Boeing's spec. Whoever wrote the spec might be 3rd party as well, with Boeing's 'engineers' just performing as project managers.

ken 8y ago

> "A properly trained pilot should be able to solve an MCAS anomaly or any uncommanded flight-control input through procedures that are taught to all 737 pilots" It blows my mind that anyone still talks like this. My first job was writing (non-life-critical) scientific software for detail-obsessed university researchers working in a lab with notebooks and procedures. If "just solve it with training" could work for anyone in the world, it would have been these people. Yet it was immediately and abundantly clear that designing usability into the system would have far more impact on their success rate at any task than documentation or training or even the reliability of the software itself. The movie "Apollo 13" showed astronauts working in a simulator, and engineers giving them faulty sensor inputs to try to trick them. They had to train to be able to identify this possibility, and react correctly to it. This isn't the sort of thing you can figure out on the fly (so to speak). If the solution to flying an aircraft with a faulty MCAS sensor was "the pilot should have read it in a book last month", it's no wonder they were in trouble. The software industry does make some programs with random inputs and hidden internal state, and require users to figure out what's going on and solve it anyway. We call them "games". Perhaps the defining characteristic of a game is that, even if you've read the manual, you won't succeed on the first try. > "But, he said, if he were designing the system from scratch, he would emphasize the training while also building the plane with three sensors." Is even three enough? From an earlier version of the "2001" Wikipedia page [1]: > "In the story HAL features a design with triple redundancy, so that if one of the three modules fails the other two can outvote it. However, there is a [[theorem]] in [[computer science]] that proves that for such [[distributed systems]] a vote-based [[sanity check]] only works if ''less than'' one third of the modules fail. Thus the failure of a single one of HAL's redundant modules would be sufficient to compromise the system, as apparently happened in the movie." I don't know what theorem this is referring to. Help? See also: Segal's Law. [1]: https://en.wikipedia.org/w/index.php?title=2001:_A_Space_Odyssey_(film)&diff=537589&oldid=530391 https://en.wikipedia.org/w/index.php?title=2001:_A_Space_Ody...

jacobush 8y ago
Not only that, but the pilots received no training, to uphold the fiction that the 737 MAX flies the same as the old 737.