7 ms·
This is a very good analysis, but fatally incomplete. One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it
by ncmncm 8y ago
This is a very good analysis, but fatally incomplete.
One really essential reason those planes crashed was that each time the MCAS triggered, it acted like it was the first time. If it added 1 degree of trim last time, it adds a second this time, a third next time, up to the five degrees that runs the trim all the way to the stops.
A second reason is that, under the design still on file at the FAA, it could only add a maximum of 0.8 degrees (each time). This was raised to 2.4 degrees after testing, so only two hits could, in principle, put you almost to the stops.
A third was that the only way to override the MCAS was to turn off power to the motor that worked the trim. But above 400 knots, the strength needed to dial back the trim with the hand crank was more than actual live pilots have, especially if it is taking all their strength to pull back on the yoke.
A fourth was that, with two flight control computers, the pilot could (partly) turn off a misbehaving one, but there is no way to turn on the other one. You have to land first, to switch over, even though the other is doing all the work to be ready to fly the plane.
A fifth was that it ignored that pilots were desperately pulling back on the yoke, which could have been a clue that it was doing the wrong thing.
A sixth was that, besides comparing redundant sensors, it could have compared what the other flight computer thought it should be doing.
- AWildC182 8y agotl;dr the first thing you learn doing control systems is NEVER use += 1; when it relates to an electro-mechanical device. I learned this a long time ago by putting a robot through a door-frame at max speed. Edit: To add to this in a way that might actually make it useful to someone, motor outputs should almost always be a continuous function of something rather than their own internal state. MCAS should have been coded as something like (vastly oversimplified) if(AoA >= MAX_AOA) trimPosition = g(f(AoA, IAS), trimPosition, dt);//AoA/IAS dependent ramp function else trimPosition = g(trimInput, trimPosition, dt);//ramp function not if(AoA >= MAX_AOA) trimPosition += 1; Because the latter is very likely to result in a runaway even if you have bounds checking somewhere else. Worst case you don't have bounds checking and the position value/register value loops over and the tail just starts spazzing out, flapping up and down as fast as the motor will drive it.
- amluto 8y agoPID controls do exactly what you are objecting to.
- AWildC182 8y agoIt's more nuanced than that. My ramp functions technically have a += 1 somewhere in them. You just don't want to += a motor value directly or otherwise add to state out of a feedback loop. You can verify the PID function in simulation/unit test. It's much harder to unit test the motor/driver/controller on a stand.
- jschwartzi 8y agoIt seems like what you're really getting at is that when you have a control like this where there's a non-linearity in response for a linear change in input quantity, what you really want to do is instead have a non-linear change in input quantity so that there is a linear response. In that case, you would characterize the effect of the control on the response variable and arrive at a table of acceptable values. Then your +=1 becomes an increment of an index which returns the next-highest acceptable value and you no longer produce a non-linear change in the response.
- kortilla 8y agoNo, you’re missing the point a bit. It’s not about the magnitude of the response, it’s about the propensity for a feedback loop. =+1 is shorthand for “ignore everything going on in the world and increase your value”. This is almost never what anyone actually wants so they try to spend a bunch of time guarding against calling that when it’s already at a maximum. Instead, the safe thing to do is only assign to it from a function with a ceiling. val = min(CEIL, val+1) It’s way to easy to get runaways with =+1 even in serious systems like this one. Every time I see that in code I review where the value is some long-lived thing, I just confirm with the author that they don’t care if it overflows, because it’s probably gonna happen.
- Gibbon1 8y ago
- d-sc 8y agoArchitecting solutions is hard. In this case, you need knowledge of motors, flight controls, sensor fusion, etc. It’s easy to find edge cases when they present themself (tragically here). But most electrical-mechanical-software assemblies have similar issues.
- ncmncm 8y agoThus, the standards and procedures that if followed would have prevented the problem. There is a standard that says automated controls are not allowed "authority" such that the pilot cannot counteract it. There is a standard that says failure of a single sensor must not cause a critical failure. Either violation alone makes the design not airworthy, and not certifiable for civil aviation, preventing both crashes.
- ncmncm 8y agoThis case is unusual because, rather than a whole series of things that all had to go wrong before the plane would crash, this system has numerous failure modes that individually were almost enough by themselves to cause a crash. It is only astonishing that it took so long for it to happen. In the Lion Air case, painfully inadequate maintenance contributed.
- gizmo385 8y ago> In this case, you need knowledge of motors, flight controls, sensor fusion, etc. Expertise which should be readily available at one of the world's foremost designers of high performance commercial aircraft.
- acqq 8y ago> A sixth was that, besides comparing redundant sensors, it could have compared what the other flight computer thought it should be doing. If you mean AoA sensors, AFAIK, there was absolutely no redundancy at all the way MCAS was designed. Exactly one sensor was ever used for MCAS. And last time I've read the Boeing's reported coming software changes, they wanted to keep it so, but just to add the notification to the pilot when the sensors disagree.
- ncmncm 8y agoComparing sensors and comparing judgment of the other computer were both things they could have done, that they utterly failed to do, in both cases.
- acqq 8y agoBecause their main premise was that they are selling the "same old" plane, and any change in any workflow would make obvious it is not.
- kakwa_ 8y agoThis analysis is completely right, but in my opinion, focuses too much on the technical aspects. Is MCAS a hack? yes. Is it fixable? yes. Will the 737 MAX continue to fly for two to three decades after all the items above have been addressed? yes. But from an engineering perspective, putting an additional system to "fix" another system feels always a bit weird. Sometimes it's not avoidable (ex: cooling), but when it is avoidable, something is at least a bit wrong. A few hacks like that are manageable, but too many, and you dramatically increase the chances of one of these hacks misbehaving. And if an organization is pushing a lot for this kind of hacks as Boeing did, the issue is not even technical. The story of MCAS reminds me a little about the MD11. The DC10 as a tri-jet could not really compete with new dual engine airplanes in the late 80ies/early 90ies in term of fuel consumption, but McDonell Douglas try to anyway. They optimized the wings, change the engines, add winglets and more significantly reduce the horizontal stabilizer size. This made the MD11 quite hard to land as it needed to go-in with a very high speed for a wide body jet. It was a contributing factor in several accidents (Fedex 14, Fedex 80, Lufthansa Cargo 8460), and pilot training/technical fixes never fully compensate for the design flaw. And in the end, the aircraft kind of failed to reach the target fuel consumption. However, it's still flying today, and it's still a workhorse for cargo companies.
- ncmncm 8y agoUltimately it was not a technical failure: the failure was in allowing the plane to be sold with such a thoroughly bad design. Thus, a management and regulatory failure. Regulatory, because FAA signed off on it, obviously without applying any of the process that would have prevented it. Management, because the cost of this debacle will be many, many times what they saved by trying to skate by with a faulty design.
- Cacti 8y agoThe airlines knew what they were doing when they didn’t pay for the “upgrade” and they doubly knew when they didn’t pay for it after the first crash. Airline management is just as culpable in this debacle.
- 8y ago
- fma 8y agoRegarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor.
- outworlder 8y ago> Regarding #6...Boeing charged $80k to unlock the software feature to gather data from more than 1 angle of attack sensor. 80k for a _light_ in the panel, not to have MCAS take the two sensors into account - which it couldn't do anyway. With two sensors, how do you know which one is right and which one is faulty?
- kortilla 8y agoWell you can’t get quorum but you can certainly shut off when you detect a conflict.
- ReGenGen 8y agoyou can't shut MCAS off AND also NOT train the pilots on the system and MAX flight characteristics. Conflicting system goals.
- bm98 8y agoTo me this is the crux of the issue. And it's non uncommon in software development: The developer fails to check for an error condition or raise an exception because doing so would add too much complexity to the system. So instead it is assumed (or hoped) that it simply can't or won't happen. Problem solved... (Edit: Or that the user will just have to reboot if it happens.)
- ReGenGen 8y agoYes. Not checking for errors is the feature of the MCAT "solution" which hides the system from pilot's training AND involvement with other 727 system/team engineers. A bureaucratic solution to a marketing problem. I wonder how many programers they had to go though to find it.
- m463 8y agoI don't know, I think the 3 points in the article make it glaringly obvious that the root cause is NOT engineering. The decisions made clearly ignored engineering and historical precedent at every turn. It's sad because Boeing has had some wonderful engineers, and Boeing aircraft have traditionally allowed the pilots to have the final say.
- 1v1id 8y ago> So Boeing produced a dynamically unstable airframe, the 737 Max. That is big strike No. 1. Boeing then tried to mask the 737’s dynamic instability with a software system. Big strike No. 2. Finally, the software relied on systems known for their propensity to fail .... Big strike No. 3. The article definitely does partially blame engineering.
- ReGenGen 8y agoWe don't know the MAX is inherently unstable. That's still conjecture at this point. MCAS may have altered fight characteristics to match older 737 to avoid additional pilot training. If the MAX is inherently unstable, then this scandal is much bigger and far reaching.
- deleted 8y ago[deleted]
- cyphar 8y agoThe article is making the point that the MAX is inherently unstable because of the larger engines causing the "pitch up" problem with an increasing angle-of-attack: > Pitch changes with increasing angle of attack, however, are quite another thing. An airplane approaching an aerodynamic stall cannot, under any circumstances, have a tendency to go further into the stall. This is called “dynamic instability,” and the only airplanes that exhibit that characteristic—fighter jets—are also fitted with ejection seats. So arguably the existence of MCAS in the first place indicates that the aircraft design is dynamically unstable (otherwise MCAS wouldn't have been necessary).
- Cacti 8y agoStab trim cutout is not the only way to disable MCAS. Extending the flaps any amount also disables MCAS. Turning on autopilot also disables MCAS, though this isn’t entirely effective since spurious AoA readings may quickly disable the autopilot again.
- cjbprime 8y agoBoeing put out an emergency airworthiness directive after Lion Air. It doesn't tell pilots to lower flaps. AoA sensor failure causes IAS Unreliable warnings, and the checklist for that item demands that flaps be left alone -- if you don't know how fast you're going, lowering flaps could cause a wing stall. It's not reasonable to expect pilots to disobey checklists. We would all be less safe if they did. If pilots are following Boeing's instructions and planes are crashing, that's on Boeing.
- metanoia 8y agoOne question I've had is why Boeing changed the design of the stab cutout switches from the 737NG from a pair of switches - the right side switch that disabled the autopilot control of trim and the left side that disabled the trim motors entirely [1] - to a design with a similar pair of switches, but each controlling the primary and backup control motors of the elevator trim [2] instead of controlling the source of the input. (If you look closely at the yoke trim switches, there are two independent switches that are grouped together by a frame so that they move simultaneously. One drives the primary motor, and the other the secondary). Assuming they'd kept MCAS on the switch associated with the automatics, the procedure would have been to throw the AUTO PILOT switch, disabling MCAS, but keeping the MAIN ELEC switch on, allowing them to trim back to neutral for that speed. [1] https://www.airliners.net/photo/Australia-Air-Force/Boeing-737-7DT-BBJ/865930 https://www.airliners.net/photo/Australia-Air-Force/Boeing-7... [2] http://www.b737.org.uk/mcas.htm#stcs http://www.b737.org.uk/mcas.htm#stcs
- metanoia 8y agoEDIT: Oops, there's only one trim motor on the NG. Both trim switches must be actuated on the yoke switches though. But the input cutoff switches still work the same way I originally stated. Not sure about the Max's # of trim motors, but the Max definitely doesn't have input source cutoff switches.