14 ms·

Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious q

Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious question is why they didn't do this in the first place? One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. The elephant in the room is the requirement to maintain a common type rating with older 737 models. Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. They'd need to announce to the pilots an AoA disagree, and announce that MCAS was disabled. Now what? A pilot certified and trained on the older 737 would not know how the Max now differs from what they trained on. If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating. Hence it seems likely they didn't add the AoA sensor comparison for this reason, reasoning that it was unlikely to be a problem anyway. We now know that reasoning was flawed. What does this mean going forwards? Will EASA and other CAAs refuse to certify the modified 737 Max under the same type rating as the older 737? This certainly seems possible. If they did require a separate type rating, this would likely kill 737 sales, regardless of whether the plane is now safe.

siwatanejo 8y ago
IIRC the two planes that crashed only had a single AOA sensor (the 2nd redundant one being only present in a premium add-on that those airlines didn't purchase), so this software fix would have not changed anything. EDIT: alright thanks for the replies.
- acqq 8y ago
  > the 2nd redundant one No, it was never redundant, in the "premium variant" only the "they don't match" signal would be displayed / sounded, but only one used still, and the pilots would have to turn MCAS off and fly with the plane which behaves differently than the one for which they are trained.
- mhandley 8y ago
  No, all 737s have two AoA sensors. The one on the captain's side feeds the flight control computer on that side, and the one on the first officer's side feeds the flight control computer on that side. At any one time, one computer is in charge. They can compare data, but Boeing decided not to for MCAS.
- chopin 8y ago
  All 737 MAX have two sensors. The Lion Air flight had sensors disagreeing by 20° right from the start. This made the stick shaker go off on one side right after departure (throughout the entire flight) and led to a multitude of other alarms (altitude disagree, unreliable airspeed).
- dalai 8y ago
  From what I've gathered, all MAX planes (including the ones that crashed) have two AoA sensors, but MCAS just uses one. As far as I know there was never an option to have a redundant configuration. The premium add-on was the indication that they disagree.
kelnos 8y ago
> If they did require a separate type rating, this would likely kill 737 sales Would it, though? I'm genuinely asking because I don't know how much all this costs. Certainly certifying pilots for a new aircraft isn't free, and probably isn't cheap, but the MAX line promises significant savings in fuel cost. In the long run, would the latter outweigh the former?
- gsnedders 8y ago
  It likely would; if you have a fleet full of 737s and the 737 MAX turns into a new type rating, there's a much weaker argument as why they must get the 737 MAX rather than an A320neo family aircraft, and that gives Airbus a big in-road with potential customers. That's not to say that Airbus would get all the sales (they won't!), but they then have a much easier time to selling to former 737 customers.
  - adwww 8y ago
    ^^ I just made the exact same comment, but this puts it clearer!
  - isostatic 8y ago
    So Boeing may have to reduce the price a little. C'est la vie. However ti's unlikely to cause major problems, presumably the tooling and spares for the non-max 737 are pretty similar to the 737. If you own a fleet of Boeing mid-range jets, why would you suddenly start to buy airbus?
    - gsnedders 8y ago
      That depends if the certification of the tooling and spares allows them to be used interchangeably between the pre-MAX and the MAX.
  - robin_reala 8y ago
    Out of interest, why wouldn’t Airbus get those sales?
acqq 8y ago
> If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating. Yes. There were no simulators to train pilots (only 4 delivered up to now, vs. 376 planes delivered! -- by the way, the value of all MAX orders, including these still not delivered, is around 600 billion with a b dollars!) and if I'd guess the simulators can't simulate the plane behavior when MCAS is off. Because the selling point is "MAX behaves the same as the old one." Which is just not true.
CraigJPerry 8y ago
According to Blancolirio on YT (a wholehearted thumbs up for his journalism, e.g. the video on atlas prime air is worth a watch, he currently flies as FO on the 777 I believe), there exists an angle of attack disagree light already in the 737max options sheet. There's also an option to purchase an AoA indicator dial, and he said one of the major us carriers did buy that option on their aircraft.
- mhandley 8y ago
  Yes, that's correct. But the implications of AoA disagree are different for the older 737 and the Max with the new software installed. On the Max, this means that the plane will now fly differently at high AoAs than it did if the AoA disgree light was not lit. This is something pilots will need to be trained for.
  - FabHK 8y ago
    > the plane will now fly differently at high AoAs than it did if the AoA disgree light was not lit. ... "at high indicated AoA", with 50% probability, no?
- heisenbit 8y ago
  There is a huge difference of displaying angle of attack as input to educated pilots to feeding it into a system which has the authority to override the pilot with little warning. The problem with the 737MAX airframe is that the pitch up seems to be so strong that nothing else than immediate MCAS feedback can reign it in.
B1FF_PSUVM 8y ago
> this would likely kill 737 sales, regardless of whether the plane is now safe. I suspect a 737 Max is now as saleable as a Samsung Note 7 phone.
- tluyben2 8y ago
  If it's indeed solvable by software then it'll still sell fine, but this is one of the articles I have read where the claim is made that it is a fundamental design issue and cannot safely be fixed. I know nothing about aviation but I do spend a lot of time in planes and I sure as hell won't be boarding a MAX until the fix has been flying around without accident for a few years (I try to fly Airbus only anyway but that's not for safety reasons).
  - dmpk2k 8y ago
    Perhaps it'll sell, but were I a buyer I'd be asking what else that Boeing might have skimped on and FAA rubber stamped. We know the plane was built in a rush, and the single-sensor MCAS is an odd choice. It's unlikely MCAS is the only problem, although hopefully it's the only dangerous one. Don't forget the perception of your ticket-buying customers.
    - mrhappyunhappy 8y ago
      I might be the only idiot but I don’t plan on flying Boeing 737 Max, no matter what kind of software fixes they put in place. Irrational? Maybe.
    - ethbro 8y ago
      > what else that Boeing might have skimped on and FAA rubber stamped Are we pretending this doesn't happen with other manufacturers too? I'd rather fly on a plane that's been thoroughly revetted than one which has only been through a flawed process once.
      - cf141q5325 8y ago
        With manufacturer not being US manufacturers the size of Boeing? Personally I would doubt that. Is there any example where that happened?
Angostura 8y ago
> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. The other problem Boeing faces is that with MCAS enabled the plane no longer necessarily flies like an older 737 - it can try to force its nose down unexpectedly.
- mhandley 8y ago
  The older 737 could do that too - it's known as a stabilizer runaway, and could happen if (for example) the stabilizer trim switch failed closed in the nose-down-trim position. It's something pilots train for, and there are trim-disable switches to handle this. Presumably Boeing believed that this was sufficient for pilots to cope with MCAS too. The problem is that MCAS doesn't run continuously - instead it runs in 10 second bursts, so it looks different from a runaway trim event that pilots had trained for. And MCAS doesn't just result in runaway trim - the failed AoA sensor is also used to correct airspeed, so this gave the pilots an unreliable airspeed indication, and it also triggered the stick shaker on the captain's side, indicating an aerodynamic stall. The combination of all these things together seems to have confused the pilots, and resulted in them not disabling electrical trim. Edit: here's the relevant data from the Lionair flight preliminary report: http://nrg.cs.ucl.ac.uk/mjh/lionair.png http://nrg.cs.ucl.ac.uk/mjh/lionair.png
  - pas 8y ago
    How come the pilots weren't aggressively keeping up with/against the MCAS? They simply gave up? Does that require some very serious effort (pushing many buttons, fiddling with the control), or do they have to simply keep "the joystick" in one direction?
    - mhandley 8y ago
      Originally the captain was flying, and he was keeping up with MCAS using manual electrical trim. But near the end, he handed control to the first officer, so that he could read the quick-reference handbook and troubleshoot. And the first officer, not knowing how much trim the captain had been using, failed to keep up with MCAS. Very sad - if the captain had kept flying, they may have survived.
GuB-42 8y ago
> One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. That's still a possibility. Stupid decisions can emerge out of smart people. Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals. For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production. It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe. There are a lot of safeguards but the complexity is so high that sometimes, something goes through. Especially if the company is under pressure.
- bravo22 8y ago
  That mistake of one team not talking to the other about AoA would have been caught at system level hazard analysis, which is a requirement for any half decent engineering organization churning out safety critical requirements. Heck it is part of required submission to FDA for medical devices which are considered even moderate safety risk, let alone an airplane.
  - deleted 8y ago
    [deleted]
- vonmoltke 8y ago
  > For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production. > It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe. What you describe here would be a major failure of systems engineering on the project. The systems engineers are responsible for flowing top level requirements down to the individual systems. They are responsible for ensuring the specs the engineering teams receive for their systems are correct, and for handling requests to change said specs. If the spec of the output of the AoA sensors does not match the spec flowed down to other teams on the input from those sensors the systems engineers responsible did not do their jobs. Systems engineers exist to manage complexity like this, to ensure that the various engineering teams across the various disciplines are technically coordinated by providing clear, consistent specs and interfaces for them to work to. If that didn't happen, then I would not say those engineers are competent, let alone smart. It would be especially disappointing to me (as a former systems engineer) if this were the case, as systems engineering is the last place I would expect such incompetence from a company like Boeing. It's at the core of everything they do. I want to address one specific point in a different context. > the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere You always expect out-of-spec conditions to be a possibility and have something in place to handle those conditions appropriately. To not do so is incompetence bordering on negligence.
PinguTS 8y ago
As I understand a simple software fix is not possible according to regulation. The problem is as follows, as you described it partly: 2 sensors are not enough. If the MCAS is an important part for the flight safety, a simple redundant safety system is not enough. Because an airplane is not about functional safety but mission critical safety. In functional safety, if there is an error the safety function is triggered and the system is transferred into a safe state. But there is no safe state here. If the system is mission critical, then it is not safe to assume to switch it off in case of an error. That means for mission critical system we need at least 3 readings and with a vote can decide on which reading is most likely the correct reading. If the MCAS would not be part of the mission critical path, then we could ask why is there in the first place? There must be reason why it was introduced. I assume, it is not done by a simple software update, if there are only 2 sensors. It will be partly redesigned to fit the requirements and regulations. But of course, this will not be publicly announced. Think about the share price. They will maintain a communication that assumes that this is an easy (and cheap) fix, a software update.
- bravo22 8y ago
  Yes! But even if you have 2 out of 3 vote you can still have two sensors that can fail, say due to a bird strike or something else and MCAS would have to be disabled which means pilots need to learn to fly the plane w/o MCAS. The one part of the narrative that Boeing has tried to Jedi Mind Trick away is that even if the pilots knew to disable MCAS by pulling the fuse, they're still left with having to fly a plane which no longer flies as the previous generation that they were trained on -- which is what MCAS was designed to do. So you still have a safety issue.
  - ams6110 8y ago
    In normal flight, MCAS is not active. The plane handles pretty much like any other 737. MCAS becomes active at high angles of attack, near the stall, to meet handling requirements for certification.
    - salawat 8y ago
osrec 8y ago
When politics and egos come into play, even teams of very smart software engineers can end up making silly, seemingly incompetent decisions.
skywhopper 8y ago
Yes it all comes back to the requirement from on high to not require any retraining or recertification even though they were delivering essentially a different airplane. Trying to simulate the feel of a different plane via software is adding a huge new layer of complexity and failure risk.
jussij 8y ago
> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. The bigger problem is the MCAS was only added to fix a major design fault, where by the aircraft would automatically pitch up when accelerating. So with the MCAS disabled, the aircraft then runs the risk of stalling when accelerating. I don't understand how design engineers would ever think a software workaround would be a suitable fix for what appears to be a major aerodynamic design flaw.
- ams6110 8y ago
  All aircraft with underslung engines want to pitch up when accelerating. That's just physics. MCAS is there to counter pitch up at high angles of attack, caused by the aerodynamic characteristics of the larger, more forward engine nacelles.
  - jussij 8y ago
    It's not the pitch that is the problem, but instead the degree of pitch. This particular aircraft had a tendency to pitch so badly they had to install a software workaround to automatically catered for the behavior. And presumably it was so extreme they thought an automatic option was needed, out of fear the pilots on their own would not be able to handle the situation.
avip 8y ago
God am I blessed to be building webapps and not be responsible for autopilot systems flying fellow humans 11Km above ground.
- ravenstine 8y ago
  Does anyone know what the typical salary of this kind of software developer is? I hope they are paid several times more than I make as a web developer.
  - vonmoltke 8y ago
    More like a couple times less. I left aerospace partially because the pay sucks.
    - jessaustin 8y ago
      Does this mean that the software developers aren't actually the people "responsible" for the behavior of the system?
      - magnamerc 8y ago
        Certain delegated systems engineers are 'responsible' for the system and they would work for Boeing or the FAA. In a field like Aerospace, you don't have 'software developers' per se. You have systems specialists who write software. You don't move fast and break things, you run simulations of the software over and over again while varying inputs and fixing edge cases, and then you do real world testing.
      - ams6110 8y ago
        I don't work in that area but could image it's much more layered. In consumer/webapp type software a lot of developers are more or less "full stack." In aerospace I could see it being much more rigorous and more like waterfall so by the time the specification gets to the programmer it's already been designed, reviewed, and signed off by several other engineers and the actual programming is a rather rote task. Or I could be totally wrong.
erentz 8y ago
> Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737.... But it’s been reported that this was an option you could buy when you bought the planes. And the crashed planes didn’t have this option. So if that’s correct, then any plane shipped with this optional package would require the recertification. But it appears they don’t either. If they did it would show up as very suspicious and I’m surprised nobody has reported on it: Here buy this plane without this optional package and you don’t need new training. Or buy it with the optional package and you need to learn about these new components we’ve added that may be disabled and undergo new training. It seems too obvious.
- mhandley 8y ago
  An AoA disagree indicator was an option. But it was never an option for this to disable MCAS, as this would have resulted in a change of the flight characteristics that would have required training.
  - erentz 8y ago
    Oh! thank you I didn’t make that connection.
  - michaelcampbell 8y ago
    Would a disagree indicator be an option without the second sensor? With what would it disagree?
    - mhandley 8y ago
      All 737s have two AoA sensors. The left one feeds the left air data computer and the right one feeds the right air data computer. The AoA disagree indicator displays when the two disagree, but is optional and was not fitted on the planes that crashed. The old version of MCAS only uses one AoA sensor as input, even if the AoA disagree indicator was fitted. It does, however, alternate which of the two it uses with each flight.
fixermark 8y ago
Agreed. I think what this may mean going forward is that the CAAs are going to have to consider demanding that the training specifications be designed around a scenario where some (as-yet-to-be-defined) subset of the smart systems are disabled, and if the airframe behaves differently in that configuration, it demands re-training. I'm somewhat surprised acceptance criteria weren't already there. You don't plan for the common case when lives are on the line.
snarf21 8y ago
I agree with you that this was ALL about keeping type rating. I wish the government would offer a whistleblower award to anyone inside Boeing who could prove that this was indeed true especially since it seem that that is how the software originally operated. Companies will do whatever it takes to drive sales and revenue and stock price. Employees don't want to raise their hand and get fired as they have families to support. A true whistleblower program with WITSEC level provisions for protection and monetary support would help cut this down. Once it happens once or twice companies are very disincentivized to continue down this road.
- karlkatzke 8y ago
  The current generation of executives seem to be willing to gamble in this way. They know that as long as they have money, they won't see much (if any) prison and the chances of even getting to a trial will be minimal. I don't think that good whistleblower protection would help any. The justice system needs an overhaul to be blind to someone's wealth and color.
- ken 8y ago
  I bet this would be virtually impossible to prove, and career suicide for anyone who tried it. These aren't mustache-twirling villains who distribute memos that say "Let's ignore safety issues to get this approved faster". They really believe they're doing what's best for all involved. We'll save everyone time and money, and make it easier for pilots. How is that not a good thing? We have no reason to believe safety will be compromised. Predicting the safety implications of design decisions, years in the future, is not an easy task. If the AOA sensors (I think?) were a tiny bit more reliable, we'd never have seen a problem, and the MAX program would be considered a great success in efficiency. I'm sure we've all worked for managers who made decisions we disagreed with, but couldn't prove they were making the wrong one.
  - Timothycquinn 8y ago
    However, they did actively choose to not put in redundancy and the status systems in the cockpits which should have been done out of the box following any common sense and failure mode analysis procedures. This decision itself is enough to bring this to court and as a result the internal communications on these decisions will be explored.
melling 8y ago
"Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious question is why they didn't do this in the first place?" Because you had to pay for the second sensor and the disagree light. https://www.nytimes.com/2019/03/21/business/boeing-safety-features-charge.html https://www.nytimes.com/2019/03/21/business/boeing-safety-fe...
- cjbprime 8y ago
  No, the second sensor is on every plane. The option was just a light that wasn't hooked up to anything else. The light's a red herring here.
  - magduf 8y ago
    The second sensor is on every plane, true, but the MCAS doesn't use it. The light just tells the pilot when the two sensors disagree. The criminally negligent thing is having MCAS only use input from a single sensor, especially when there's a second sensor already there.
tw04 8y ago
I was under the impression the "base model" only came with a single AoA sensor. Adding a second sensor and the warning light if they disagreed was an expensive upgrade that neither of the planes that crashed were equipped with.
ChuckMcM 8y ago
This is pretty much it in a nutshell as far as I can tell. If the sensors don't agree, and MCAS switches off, then the pilots have to be ready to deal with the plane trying to pitch up and stall on their own. When would that happen? Take off and go-arounds. Pilot is coming in for a landing, something goes wrong (too much cross wind, plane on the taxiway, Etc.) what they do is they pull back on the stick and push the throttles up to max to get into a climb. If MCAS is disabled and the pilot hasn't trained to fly the plane without it, there is a risk it will pitch up and stall onto its tail. Not a good place to be.
sixothree 8y ago
Are the angle-of-attack sensors so unreliable to have caused two crashes?

siwatanejo 8y ago

IIRC the two planes that crashed only had a single AOA sensor (the 2nd redundant one being only present in a premium add-on that those airlines didn't purchase), so this software fix would have not changed anything. EDIT: alright thanks for the replies.

acqq 8y ago
> the 2nd redundant one No, it was never redundant, in the "premium variant" only the "they don't match" signal would be displayed / sounded, but only one used still, and the pilots would have to turn MCAS off and fly with the plane which behaves differently than the one for which they are trained.
mhandley 8y ago
No, all 737s have two AoA sensors. The one on the captain's side feeds the flight control computer on that side, and the one on the first officer's side feeds the flight control computer on that side. At any one time, one computer is in charge. They can compare data, but Boeing decided not to for MCAS.
chopin 8y ago
All 737 MAX have two sensors. The Lion Air flight had sensors disagreeing by 20° right from the start. This made the stick shaker go off on one side right after departure (throughout the entire flight) and led to a multitude of other alarms (altitude disagree, unreliable airspeed).
dalai 8y ago
From what I've gathered, all MAX planes (including the ones that crashed) have two AoA sensors, but MCAS just uses one. As far as I know there was never an option to have a redundant configuration. The premium add-on was the indication that they disagree.

kelnos 8y ago

> If they did require a separate type rating, this would likely kill 737 sales Would it, though? I'm genuinely asking because I don't know how much all this costs. Certainly certifying pilots for a new aircraft isn't free, and probably isn't cheap, but the MAX line promises significant savings in fuel cost. In the long run, would the latter outweigh the former?

gsnedders 8y ago
It likely would; if you have a fleet full of 737s and the 737 MAX turns into a new type rating, there's a much weaker argument as why they must get the 737 MAX rather than an A320neo family aircraft, and that gives Airbus a big in-road with potential customers. That's not to say that Airbus would get all the sales (they won't!), but they then have a much easier time to selling to former 737 customers.
- adwww 8y ago
  ^^ I just made the exact same comment, but this puts it clearer!
- isostatic 8y ago
  So Boeing may have to reduce the price a little. C'est la vie. However ti's unlikely to cause major problems, presumably the tooling and spares for the non-max 737 are pretty similar to the 737. If you own a fleet of Boeing mid-range jets, why would you suddenly start to buy airbus?
  - gsnedders 8y ago
    That depends if the certification of the tooling and spares allows them to be used interchangeably between the pre-MAX and the MAX.
- robin_reala 8y ago
  Out of interest, why wouldn’t Airbus get those sales?

acqq 8y ago

> If they'd done this, they'd have needed to provide additional training, and this must have concerned Boeing management that it might jeopardize the common type rating. Yes. There were no simulators to train pilots (only 4 delivered up to now, vs. 376 planes delivered! -- by the way, the value of all MAX orders, including these still not delivered, is around 600 billion with a b dollars!) and if I'd guess the simulators can't simulate the plane behavior when MCAS is off. Because the selling point is "MAX behaves the same as the old one." Which is just not true.

CraigJPerry 8y ago

According to Blancolirio on YT (a wholehearted thumbs up for his journalism, e.g. the video on atlas prime air is worth a watch, he currently flies as FO on the 777 I believe), there exists an angle of attack disagree light already in the 737max options sheet. There's also an option to purchase an AoA indicator dial, and he said one of the major us carriers did buy that option on their aircraft.

mhandley 8y ago
Yes, that's correct. But the implications of AoA disagree are different for the older 737 and the Max with the new software installed. On the Max, this means that the plane will now fly differently at high AoAs than it did if the AoA disgree light was not lit. This is something pilots will need to be trained for.
- FabHK 8y ago
  > the plane will now fly differently at high AoAs than it did if the AoA disgree light was not lit. ... "at high indicated AoA", with 50% probability, no?
heisenbit 8y ago
There is a huge difference of displaying angle of attack as input to educated pilots to feeding it into a system which has the authority to override the pilot with little warning. The problem with the 737MAX airframe is that the pitch up seems to be so strong that nothing else than immediate MCAS feedback can reign it in.

B1FF_PSUVM 8y ago

> this would likely kill 737 sales, regardless of whether the plane is now safe. I suspect a 737 Max is now as saleable as a Samsung Note 7 phone.

tluyben2 8y ago
If it's indeed solvable by software then it'll still sell fine, but this is one of the articles I have read where the claim is made that it is a fundamental design issue and cannot safely be fixed. I know nothing about aviation but I do spend a lot of time in planes and I sure as hell won't be boarding a MAX until the fix has been flying around without accident for a few years (I try to fly Airbus only anyway but that's not for safety reasons).
- dmpk2k 8y ago
  Perhaps it'll sell, but were I a buyer I'd be asking what else that Boeing might have skimped on and FAA rubber stamped. We know the plane was built in a rush, and the single-sensor MCAS is an odd choice. It's unlikely MCAS is the only problem, although hopefully it's the only dangerous one. Don't forget the perception of your ticket-buying customers.
  - mrhappyunhappy 8y ago
    I might be the only idiot but I don’t plan on flying Boeing 737 Max, no matter what kind of software fixes they put in place. Irrational? Maybe.
  - ethbro 8y ago
    > what else that Boeing might have skimped on and FAA rubber stamped Are we pretending this doesn't happen with other manufacturers too? I'd rather fly on a plane that's been thoroughly revetted than one which has only been through a flawed process once.
    - cf141q5325 8y ago
      With manufacturer not being US manufacturers the size of Boeing? Personally I would doubt that. Is there any example where that happened?

Angostura 8y ago

> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. The other problem Boeing faces is that with MCAS enabled the plane no longer necessarily flies like an older 737 - it can try to force its nose down unexpectedly.

mhandley 8y ago
The older 737 could do that too - it's known as a stabilizer runaway, and could happen if (for example) the stabilizer trim switch failed closed in the nose-down-trim position. It's something pilots train for, and there are trim-disable switches to handle this. Presumably Boeing believed that this was sufficient for pilots to cope with MCAS too. The problem is that MCAS doesn't run continuously - instead it runs in 10 second bursts, so it looks different from a runaway trim event that pilots had trained for. And MCAS doesn't just result in runaway trim - the failed AoA sensor is also used to correct airspeed, so this gave the pilots an unreliable airspeed indication, and it also triggered the stick shaker on the captain's side, indicating an aerodynamic stall. The combination of all these things together seems to have confused the pilots, and resulted in them not disabling electrical trim. Edit: here's the relevant data from the Lionair flight preliminary report: http://nrg.cs.ucl.ac.uk/mjh/lionair.png http://nrg.cs.ucl.ac.uk/mjh/lionair.png
- pas 8y ago
  How come the pilots weren't aggressively keeping up with/against the MCAS? They simply gave up? Does that require some very serious effort (pushing many buttons, fiddling with the control), or do they have to simply keep "the joystick" in one direction?
  - mhandley 8y ago
    Originally the captain was flying, and he was keeping up with MCAS using manual electrical trim. But near the end, he handed control to the first officer, so that he could read the quick-reference handbook and troubleshoot. And the first officer, not knowing how much trim the captain had been using, failed to keep up with MCAS. Very sad - if the captain had kept flying, they may have survived.

GuB-42 8y ago

> One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. That's still a possibility. Stupid decisions can emerge out of smart people. Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals. For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production. It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe. There are a lot of safeguards but the complexity is so high that sometimes, something goes through. Especially if the company is under pressure.

bravo22 8y ago
That mistake of one team not talking to the other about AoA would have been caught at system level hazard analysis, which is a requirement for any half decent engineering organization churning out safety critical requirements. Heck it is part of required submission to FDA for medical devices which are considered even moderate safety risk, let alone an airplane.
- deleted 8y ago
  [deleted]
vonmoltke 8y ago
> For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production. > It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe. What you describe here would be a major failure of systems engineering on the project. The systems engineers are responsible for flowing top level requirements down to the individual systems. They are responsible for ensuring the specs the engineering teams receive for their systems are correct, and for handling requests to change said specs. If the spec of the output of the AoA sensors does not match the spec flowed down to other teams on the input from those sensors the systems engineers responsible did not do their jobs. Systems engineers exist to manage complexity like this, to ensure that the various engineering teams across the various disciplines are technically coordinated by providing clear, consistent specs and interfaces for them to work to. If that didn't happen, then I would not say those engineers are competent, let alone smart. It would be especially disappointing to me (as a former systems engineer) if this were the case, as systems engineering is the last place I would expect such incompetence from a company like Boeing. It's at the core of everything they do. I want to address one specific point in a different context. > the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere You always expect out-of-spec conditions to be a possibility and have something in place to handle those conditions appropriately. To not do so is incompetence bordering on negligence.

PinguTS 8y ago

As I understand a simple software fix is not possible according to regulation. The problem is as follows, as you described it partly: 2 sensors are not enough. If the MCAS is an important part for the flight safety, a simple redundant safety system is not enough. Because an airplane is not about functional safety but mission critical safety. In functional safety, if there is an error the safety function is triggered and the system is transferred into a safe state. But there is no safe state here. If the system is mission critical, then it is not safe to assume to switch it off in case of an error. That means for mission critical system we need at least 3 readings and with a vote can decide on which reading is most likely the correct reading. If the MCAS would not be part of the mission critical path, then we could ask why is there in the first place? There must be reason why it was introduced. I assume, it is not done by a simple software update, if there are only 2 sensors. It will be partly redesigned to fit the requirements and regulations. But of course, this will not be publicly announced. Think about the share price. They will maintain a communication that assumes that this is an easy (and cheap) fix, a software update.

bravo22 8y ago
Yes! But even if you have 2 out of 3 vote you can still have two sensors that can fail, say due to a bird strike or something else and MCAS would have to be disabled which means pilots need to learn to fly the plane w/o MCAS. The one part of the narrative that Boeing has tried to Jedi Mind Trick away is that even if the pilots knew to disable MCAS by pulling the fuse, they're still left with having to fly a plane which no longer flies as the previous generation that they were trained on -- which is what MCAS was designed to do. So you still have a safety issue.
- ams6110 8y ago
  In normal flight, MCAS is not active. The plane handles pretty much like any other 737. MCAS becomes active at high angles of attack, near the stall, to meet handling requirements for certification.
  - salawat 8y ago

osrec 8y ago

When politics and egos come into play, even teams of very smart software engineers can end up making silly, seemingly incompetent decisions.

skywhopper 8y ago

Yes it all comes back to the requirement from on high to not require any retraining or recertification even though they were delivering essentially a different airplane. Trying to simulate the feel of a different plane via software is adding a huge new layer of complexity and failure risk.

jussij 8y ago

> The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737. The bigger problem is the MCAS was only added to fix a major design fault, where by the aircraft would automatically pitch up when accelerating. So with the MCAS disabled, the aircraft then runs the risk of stalling when accelerating. I don't understand how design engineers would ever think a software workaround would be a suitable fix for what appears to be a major aerodynamic design flaw.

ams6110 8y ago
All aircraft with underslung engines want to pitch up when accelerating. That's just physics. MCAS is there to counter pitch up at high angles of attack, caused by the aerodynamic characteristics of the larger, more forward engine nacelles.
- jussij 8y ago
  It's not the pitch that is the problem, but instead the degree of pitch. This particular aircraft had a tendency to pitch so badly they had to install a software workaround to automatically catered for the behavior. And presumably it was so extreme they thought an automatic option was needed, out of fear the pilots on their own would not be able to handle the situation.

avip 8y ago

God am I blessed to be building webapps and not be responsible for autopilot systems flying fellow humans 11Km above ground.

ravenstine 8y ago
Does anyone know what the typical salary of this kind of software developer is? I hope they are paid several times more than I make as a web developer.
- vonmoltke 8y ago
  More like a couple times less. I left aerospace partially because the pay sucks.
  - jessaustin 8y ago
    Does this mean that the software developers aren't actually the people "responsible" for the behavior of the system?
    - magnamerc 8y ago
      Certain delegated systems engineers are 'responsible' for the system and they would work for Boeing or the FAA. In a field like Aerospace, you don't have 'software developers' per se. You have systems specialists who write software. You don't move fast and break things, you run simulations of the software over and over again while varying inputs and fixing edge cases, and then you do real world testing.
    - ams6110 8y ago
      I don't work in that area but could image it's much more layered. In consumer/webapp type software a lot of developers are more or less "full stack." In aerospace I could see it being much more rigorous and more like waterfall so by the time the specification gets to the programmer it's already been designed, reviewed, and signed off by several other engineers and the actual programming is a rather rote task. Or I could be totally wrong.

erentz 8y ago

> Suppose they did originally do what the fixed software does now, and disable MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS disabled when this occurs, the plane no longer flies like an older 737.... But it’s been reported that this was an option you could buy when you bought the planes. And the crashed planes didn’t have this option. So if that’s correct, then any plane shipped with this optional package would require the recertification. But it appears they don’t either. If they did it would show up as very suspicious and I’m surprised nobody has reported on it: Here buy this plane without this optional package and you don’t need new training. Or buy it with the optional package and you need to learn about these new components we’ve added that may be disabled and undergo new training. It seems too obvious.

mhandley 8y ago
An AoA disagree indicator was an option. But it was never an option for this to disable MCAS, as this would have resulted in a change of the flight characteristics that would have required training.
- erentz 8y ago
  Oh! thank you I didn’t make that connection.
- michaelcampbell 8y ago
  Would a disagree indicator be an option without the second sensor? With what would it disagree?
  - mhandley 8y ago
    All 737s have two AoA sensors. The left one feeds the left air data computer and the right one feeds the right air data computer. The AoA disagree indicator displays when the two disagree, but is optional and was not fitted on the planes that crashed. The old version of MCAS only uses one AoA sensor as input, even if the AoA disagree indicator was fitted. It does, however, alternate which of the two it uses with each flight.

fixermark 8y ago

Agreed. I think what this may mean going forward is that the CAAs are going to have to consider demanding that the training specifications be designed around a scenario where some (as-yet-to-be-defined) subset of the smart systems are disabled, and if the airframe behaves differently in that configuration, it demands re-training. I'm somewhat surprised acceptance criteria weren't already there. You don't plan for the common case when lives are on the line.

snarf21 8y ago

I agree with you that this was ALL about keeping type rating. I wish the government would offer a whistleblower award to anyone inside Boeing who could prove that this was indeed true especially since it seem that that is how the software originally operated. Companies will do whatever it takes to drive sales and revenue and stock price. Employees don't want to raise their hand and get fired as they have families to support. A true whistleblower program with WITSEC level provisions for protection and monetary support would help cut this down. Once it happens once or twice companies are very disincentivized to continue down this road.

karlkatzke 8y ago
The current generation of executives seem to be willing to gamble in this way. They know that as long as they have money, they won't see much (if any) prison and the chances of even getting to a trial will be minimal. I don't think that good whistleblower protection would help any. The justice system needs an overhaul to be blind to someone's wealth and color.
ken 8y ago
I bet this would be virtually impossible to prove, and career suicide for anyone who tried it. These aren't mustache-twirling villains who distribute memos that say "Let's ignore safety issues to get this approved faster". They really believe they're doing what's best for all involved. We'll save everyone time and money, and make it easier for pilots. How is that not a good thing? We have no reason to believe safety will be compromised. Predicting the safety implications of design decisions, years in the future, is not an easy task. If the AOA sensors (I think?) were a tiny bit more reliable, we'd never have seen a problem, and the MAX program would be considered a great success in efficiency. I'm sure we've all worked for managers who made decisions we disagreed with, but couldn't prove they were making the wrong one.
- Timothycquinn 8y ago
  However, they did actively choose to not put in redundancy and the status systems in the cockpits which should have been done out of the box following any common sense and failure mode analysis procedures. This decision itself is enough to bring this to court and as a result the internal communications on these decisions will be explored.

melling 8y ago

"Boeing's software fix, announced today, is to compare readings from both angle-of-attack sensors and disable MCAS if they disagree significantly. The obvious question is why they didn't do this in the first place?" Because you had to pay for the second sensor and the disagree light. https://www.nytimes.com/2019/03/21/business/boeing-safety-features-charge.html https://www.nytimes.com/2019/03/21/business/boeing-safety-fe...

cjbprime 8y ago
No, the second sensor is on every plane. The option was just a light that wasn't hooked up to anything else. The light's a red herring here.
- magduf 8y ago
  The second sensor is on every plane, true, but the MCAS doesn't use it. The light just tells the pilot when the two sensors disagree. The criminally negligent thing is having MCAS only use input from a single sensor, especially when there's a second sensor already there.

tw04 8y ago

I was under the impression the "base model" only came with a single AoA sensor. Adding a second sensor and the warning light if they disagreed was an expensive upgrade that neither of the planes that crashed were equipped with.

ChuckMcM 8y ago

This is pretty much it in a nutshell as far as I can tell. If the sensors don't agree, and MCAS switches off, then the pilots have to be ready to deal with the plane trying to pitch up and stall on their own. When would that happen? Take off and go-arounds. Pilot is coming in for a landing, something goes wrong (too much cross wind, plane on the taxiway, Etc.) what they do is they pull back on the stick and push the throttles up to max to get into a climb. If MCAS is disabled and the pilot hasn't trained to fly the plane without it, there is a risk it will pitch up and stall onto its tail. Not a good place to be.

sixothree 8y ago

Are the angle-of-attack sensors so unreliable to have caused two crashes?