4 ms·
> One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this. That's still a possibility. Stupid decisions can emerge
by GuB-42 8y ago
> One possibility is incompetence. But Boeing engineers are smart people, so I'm not convinced by this.
That's still a possibility. Stupid decisions can emerge out of smart people.
Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals.
For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production.
It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe.
There are a lot of safeguards but the complexity is so high that sometimes, something goes through. Especially if the company is under pressure.
- bravo22 8y agoThat mistake of one team not talking to the other about AoA would have been caught at system level hazard analysis, which is a requirement for any half decent engineering organization churning out safety critical requirements. Heck it is part of required submission to FDA for medical devices which are considered even moderate safety risk, let alone an airplane.
- deleted 8y ago[deleted]
- vonmoltke 8y ago> For example (I am not saying that happened), the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere. At the same time, the "sensors" team thought that raw, unchecked data was expected. The integration guy didn't read the specs correctly (sometimes, it comes down to a single word), didn't catch that, and checked the OK box. His manager, focused on a more pressing issue took that as granted and it went to production. > It is possible that the engineers did an excellent work, but didn't question the specs they had. The integration guy is normally super reliable but he just had a bad day. And his manager handled the other problem beautifully and overlooked the MCAS/AoA because, normally, the integration guy is reliable. A series of small mistakes that ended up in a catastrophe. What you describe here would be a major failure of systems engineering on the project. The systems engineers are responsible for flowing top level requirements down to the individual systems. They are responsible for ensuring the specs the engineering teams receive for their systems are correct, and for handling requests to change said specs. If the spec of the output of the AoA sensors does not match the spec flowed down to other teams on the input from those sensors the systems engineers responsible did not do their jobs. Systems engineers exist to manage complexity like this, to ensure that the various engineering teams across the various disciplines are technically coordinated by providing clear, consistent specs and interfaces for them to work to. If that didn't happen, then I would not say those engineers are competent, let alone smart. It would be especially disappointing to me (as a former systems engineer) if this were the case, as systems engineering is the last place I would expect such incompetence from a company like Boeing. It's at the core of everything they do. I want to address one specific point in a different context. > the engineers designing MCAS didn't expect incorrect AoA data, thinking the checks were done elsewhere You always expect out-of-spec conditions to be a possibility and have something in place to handle those conditions appropriately. To not do so is incompetence bordering on negligence.
- mtw 8y ago2 planes crashes in less than 5 months. So yes, there was either a system engineering failure, or a completely inept decision in choosing an oversized engine for an antic airframe. Your choice.
- ethbro 8y agoOr a training requirement failure. Or a UX failure. Or a documentation failure. Or an unrelated failure, given that the RCAs haven't been completed on the crashes. It's amazing the hubris software engineers have in assuming everyone else in an idiot. I am not an expert on aircraft-scale hardware/software codesign projects, aerospace engineering, high-reliability engineering (to aerospace standards), or complex systems analysis. I strongly suspect you, and most of HN, aren't either. Boeing has people with all those skillsets. As does Airbus. As do NTSB and the variety of national certification agencies. So before we toss rocks of certainty around in a comment thread, maybe wait and listen to the experts?
- bobthepanda 8y agoGiven that at least some of the expert companies involved have attempted coverups of previous failures before, a little bit of skepticism is healthy. https://en.m.wikipedia.org/wiki/Boeing_737_rudder_issues https://en.m.wikipedia.org/wiki/Boeing_737_rudder_issues Authority should not be free from skepticism, even by people who are not authorities.
- ethbro 8y agoWe can agree that not all skepticism is valuable skepticism though? I came across this Rand report on the NTSB [1, 1999?]. The conclusions were not good about the funding & staffing levels vs modern accident load. Partly due to more incidents, but moreso due to increased systems complexity per incident. So the NTSB, with a budget of around USD$100M, takes multiple years to deliver a report. And they're a professional worldwide standard on accident investigation. Suffice to say, we're not going to crack this case open on HN. Which isn't to say it isn't productive to debate the relative merits of different regulatory approaches, takeaways for other disciplines, lessons learned, and all manner of things. But let's just have some humility in pretending we're all experts on everything [2]. [1] https://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1122z1/MR1122.1.sum.pdf https://www.rand.org/content/dam/rand/pubs/monograph_reports... [2] Though in fairness, I wouldn't be surprised if there were at least one expert on any given topic here. You folks are awesome.
- phkahler 8y ago>> It is possible that the engineers did an excellent work, but didn't question the specs they had. The reason I find that scenario hard to believe is that MCAS very existence is to correct a design problem inherent in the plane. A safety problem - the plane is not safe without MCAS or at least without new training which they wanted to avoid.
- tyingq 8y agoSupposedly the history is that the single sensor input was decided when the MCAS could only exert a fairly small amount of change to the stabilizer. Later, the amount of control given to MCAS was drastically increased, apparently without considering the ramifications of a single sensor feed.
- village-idiot 8y agoSome investigative journalism indicates that this is not what happened. The FAA has different levels of redundancy and uptime requirements based on the outcome of system failure. A system categorized as “catastrophic” failing would lose the plane and all passengers, while “hazardous” might hurt some and kill a few, and so on. The FAA requires that catastrophic systems must have a backup, while hazardous can go without one if it’s reliable enough. Boeing classified the MCAS as “hazardous” only under certain flight characteristics, and they categorized it as a lower level than hazardous during level flight. This means that the crews designing the specification made the decision to depend upon only a single input, and it appears to have been built to spec[0]. It also appears that the FAA was experiencing internal pressure to delegate more certification authority to Boeing, with disastrous results. 0: https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/ https://www.seattletimes.com/business/boeing-aerospace/faile...
- kulu2002 8y agoWhat about integration/ subsystem level testing? It seems MCAS wasn't thoroughly tested with all boundary values. Whats the use of systems like IAHM [1] if they can't predict what faults can occur how safety is ensured/ guaranteed in various scenarios. [1] https://www.phmsociety.org/sites/phmsociety.org/files/Fielded%20Systems%20Panel%20Denver%20161006%20post.pdf https://www.phmsociety.org/sites/phmsociety.org/files/Fielde...
- tomxor 8y ago> Boeing is huge, and what they develop is incredibly complex. There are a lot of people with differing level of competence, ethics, and goals. That is true... however it's also true both of the big aerospace companies have an abundance of very smart engineers who by their very nature would have been responsible enough and inquisitive enough to notice this defect even if it was not their direct responsibility (good engineers will step out of their box). Knowing how vital that sensor is (regardless of the stupid reason for it being vital) and not building in redundancy is such an obvious flaw to any engineer - The reason with almost certainty will have been because that multiple observations of this safety flaw would have been squashed by some big execs due to the business requirements and economics and a severe lack of morals.
- whatshisface 8y ago>good engineers will step out of their box Malignant cultures can do a lot to reduce this. The platonic ideal engineer will cross any number of organizational boundaries to deliver a needed piece of information, but there can be a lot of pressure to leave those boundaries un-crossed. Depending on how political the organization is, a strange engineer (or their manager) dropping in from nowhere to tell you a mistake was made may even be seen as a bad thing.
- vonmoltke 8y agoIn addition aerospace, like most other traditional engineering fields, is heavily biased towards seniority. Years of experience actually matter[1], and more junior engineers often have an uphill battle trying to convince more senior engineers of errors. In aerospace in particular it's a side effect of having systems so complex that first impressions of a subsystem can give you intuitions that are wrong in the context of the larger system, even though they may seem right in isolation. [1] More than they should, in my opinion.
- typeformer 8y agoThe Challenger disaster is a good example of an engineer who did warn the right people but they ignored him; I would bet bottom dollar something similar happened with these Boeing tragedies. The pressure to launch is great and who wants to be the person who stalls "progress".
- vsl 8y agoThe explanation I’ve read (sorry, don’t recall the source) is that this system repurposed a purely advisory one that existed before, for informing the pilot about AoA. Being advisory in nature, it didn’t need duplicate sensors. Next up: you need to implement MCAS and everything is conveniently there in existing code. That, plus toxic culture for internal people doing FAA (yes, really), plus what you write, plus business pressure on certification... Edit: source https://syonyk.blogspot.com/2019/03/boeing-airbus-tesla-and-automation.html https://syonyk.blogspot.com/2019/03/boeing-airbus-tesla-and-...