21 ms·
Imagine the quality of the manufacturing and QA / final inspection to have that kind of issue.
by nolok 3y ago
Imagine the quality of the manufacturing and QA / final inspection to have that kind of issue.
- ethbr1 3y agoI expect even at its worst, software development could learn a lot from aircraft QA. Especially since most shops have pretty much tossed professional career QA out the window.
- darylteo 3y agosoftware development: what is QA?
- bb88 3y agoThat's where failed software devs / management go to.
- ShadowBanThis01 3y agoExcept, of course, from Boeing's aircraft-software QA... which killed hundreds of people already.
- actionfromafar 3y agoThe problem was not really the software in isolation, but that pilots expected the 737 NG to behave exactly like the old version - because Boeing decided it was too expensive to retrain pilots.
- WalterBright 3y agoThe expense for retraining pilots falls on the airline. Retraining has its own problems. No matter how well retraining is done, pilots still make mistakes from doing the right thing for the previous plane that is the wrong thing for the one they are currently flying. Adjusting airplanes to fly the same way is a major safety advantage.
- ethbr1 3y agoArguably, Boeing hit the uncanny safety valley -- similar enough so that pilots and airlines relaxed, but different enough so that relaxation ultimately killed people.
- WalterBright 3y agoThe emergency procedure for runaway trim was the same for both aircraft types, and was not followed. After the first crash, an Emergency Airworthiness Directive was issued to all MAX pilots reiterating the procedure, which was not followed in the second crash, as well as not reacting to an overspeed warning. Unreported by the media, there was another MAX incident before the first crash. The crew had no knowledge of MCAS, but did follow the emergency runaway trim procedure, and continued the flight and landed safely.
- 9659 3y ago"Runaway stab trim". It is a memory item, every pilot should be able to perform it from memory. Turn off the motor, and the trim is manual. There is a crank right there in the cockpit. If it is too hard to turn, change aircraft configuration to reduce the forces required to. Pilot know how to do this. This pilot stuff, they understand the forces on the flight controls and what impacts them. Boeing made an engineering mistake. The pilots also made an operational mistake. Unfortunately, both mistakes at the same time were fatal. I pray that pilot training has improved. And that Boeing has made systems level changes to the aircraft that will preclude it happening in the future. And that is how aviation becomes safer every year; at a significant cost of customers lives.
- ethbr1 3y ago> And that is how aviation becomes safer every year; at a significant cost of customers lives. "Significant" might be inaccurate. It looks like FAA Part 121 accidents over the last 10 years with fatalities have been... 4. [0] For a total of 6 fatalities. [0] https://www.ntsb.gov/Pages/AviationQueryV2.aspx https://www.ntsb.gov/Pages/AviationQueryV2.aspx; 2018 (1 passenger fatality) https://www.ntsb.gov/investigations/Pages/DCA18MA142.aspx https://www.ntsb.gov/investigations/Pages/DCA18MA142.aspx ; 2019 (3 crew fatalities, cargo flight) https://www.ntsb.gov/investigations/Pages/DCA19MA086.aspx https://www.ntsb.gov/investigations/Pages/DCA19MA086.aspx and (1 passenger fatality) https://www.ntsb.gov/investigations/Pages/DCA20MA002.aspx https://www.ntsb.gov/investigations/Pages/DCA20MA002.aspx ; 2022 (1 ramp fatality) https://data.ntsb.gov/carol-repgen/api/Aviation/ReportMain/GenerateNewestReport/106517/pdf https://data.ntsb.gov/carol-repgen/api/Aviation/ReportMain/G...
- unyttigfjelltol 3y agoThe problem was software that prioritized input from a fauly external sensor, over pilot control, and literally crashed planes directly into the ground. At a certain step in the sequence it was not physically possible for a pilot to pull hard enough on the control element to counteract the software. Could they have disabled the system? Only if they could figure out the specific software trying to crash the plane. Is that what you meant by "the problem wasn't the software?" Because the pilots should have been trained to unplug the computer to stop it from crashing the plane?
- ethbr1 3y agoPilots are definitely trained how to disable the autopilot, if needed. Afaic, the fault apportionment was Boeing documentation > airlines >> pilots > Boeing technical design.
- diabeetusman 3y agoNot sure why you’re bringing up autopilot— the MCAS system runs even when the autopilot is disabled. Edit: Also, how does the fault lie with the airlines? Boeing didn’t document the existence of MCAS in the flight manual or training materials.
- ethbr1 3y agoBecause the comment I was replying to >> Because the pilots should have been trained to unplug the computer to stop it from crashing the plane? Yes. The fault lies with the airlines because I don't for a second believe they didn't put pressure on Boeing to get the MAX certified without mandating retraining. And then once that was done, didn't dig into the details too hard about what changes were made. I have a low tolerance for 'I set up all the conditions and incentives to encourage you to break the law... but you should take all the blame when it explodes.' At some point, the customer has to take some responsibility for what they asked for.
- actionfromafar 3y ago
- ShadowBanThis01 3y agoWrong.
- echohack5 3y agoQA was literally invented for the airline industry. Software QA when actually practiced is more advanced now than airline QA.
- paranoidrobot 3y ago> Software QA when actually practiced is more advanced now than airline QA. ...eh, I think "when actually practiced" is doing a lot of carrying there. What do you mean by "actually practiced". Outside of the aerospace and healthcare industries, I'm not sure there are many software shops that are doing QA to a level I would like to trust anyone's life with.
- serf 3y agowhat does advanced mean when comparing things so unlike from each other? also software is the least likely comparison I would have made; software quality is a shit-show on a general level, and the vast public is quite aware of this every time a subway timeboard blue-screens or gets frozen on an AMI screen, or the POS machine that they're forced to interact with at work does something equally as stupid.
- booleandilemma 3y agoNah, in the software world, the truth is QA is where the people who can't get jobs as programmers end up. I've seen testers go on to become programmers, but I've never seen a programmer become a tester. Maybe it's different for real-time or life-critical systems, sure, but I can confidently say this is how it is in web development.
- vegetablepotpie 3y agoI work in the defense industry, it’s very much like the aerospace industry in that we deal with human life as a consequence of our work. We have software QA departments that operate very much like manufacturing or aerospace QA. Software QA provides nothing of value to software development; having it as a dedicated function works against the overtly stated goals of the function and counterintuitively acts to degrade quality within software by mandating strict top down process and brittle end-to-end testing. Although Software QA is intended to be an independent verification body that provides engineering organizations with tools and resources, in practice they function as a moral crumple zone [1] within the complex socio-technical defense industrial system, being one of the groups that the finger will be pointed to when something goes wrong and absorb shock to the business in the event of a failure. As a result they have a strong incentive to highly systematize their work with specific process steps, to shield them from liability, which can be applied generically to all projects. Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. This acts to find problems quickly and resolve them quickly. Software QAs need for high level, top down, generic systemization requires them to work against these principles in practice. Bespoke project specific checks, such as unit testing, is not viewed as contributing to the final product and is discouraged by leadership who see it as waste. To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue. [1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236
- Eisenstein 3y agoIt sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.
- rectang 3y agoIs it plausible that Boeing has "learned" from software/startup/venture-capital culture with regards to tolerating higher risk to minimize costs? I suspect it's rather a case of parallel evolution between McDonnell Douglas brass and software startup culture, since cost-cutting culture goes back many decades (remember "Chainsaw" Al Dunlap[1] ?) — but I wonder if there's a more direct influence. [1] https://en.wikipedia.org/wiki/Albert_J._Dunlap https://en.wikipedia.org/wiki/Albert_J._Dunlap
- Groxx 3y agoIn lots of ways, the "learning" there would just be "capitalism". It's inherently short-sighted unless forced to do otherwise by legislation. Cutting small corners pays off A LOT until the hammer falls, so there's a massive advantage to doing it / you need to do it if competition is doing it, or you eventually shut down as they take all your business. It's inherently a race to the bottom. Sometimes that's a net gain, sometimes it isn't.
- boringg 3y agoBut all those communist airlines had no problems at all - exceptional build quality and operational efficiency! /S
- mc32 3y agoYep, Chernobyl being a prime example. Or Komarov's failed re-entry after complaining about the design faults of the vehicle long before launch. Then there was the more uhhm run of the mill backyard blast furnace campaign which contributed to misallocation of workforce which then led to mass starvation.
- boringg 3y agoThere are many many more examples. I find it so tiresome to see young people just use capitalism as a catch all for the failure of something. It's such a lazy and uninformed argument. I'm not carte blanche defending capitalism - its a mixed bag but it sure outpaces the competing systems put forward to date. It does need some stronger safeguards against industry self regulation - that has a bad track record.
- dilyevsky 3y agoWould you pay at least 2x for your software to have couple more nines of reliability? I’m gonna guess that “no”. At places where it costs $$$ to have bugs shipped to the end customer (e.g phones) or where there’re regulatory requirements they still have dedicated qa.
- bb88 3y agoIt depends. 1% of 10000 is 100. .01% is 1. If someone came up to me and said, "Hey I can save you 99% of expected costs with 1% of your profit.", I might go for it.
- dilyevsky 3y agoWhich is what i said in second part of my comment. For most software businesses the cost of shipping a bug is trivial and/or poorly measured so due to McNamara fallacy it is readily exchanged for well measured cost of having a functioning qa team
- Robotbeat 3y agoI agree. I’m regularly critical of Boeing Defense (particularly space contracts where I’m a huge Boeing skeptic), but I think people are pretty off base if they think Boeing is just completely incompetent. Airliner safety is insanely good. Just vast seas of competence, but when there’s a super rare failure, the incorrect impression people get is that Boeing (or Airbus) is just full of incompetency. Almost nothing that humans do is held to the same standard. Not spaceflight, not software, not healthcare, and certainly not automotive. Flying a 737 Max with a bad door and without the fix to the angle of attack sensor is probably still better per mile than driving. In spite of going at 10 times the speed and miles above the Earth. You can almost argue it’s held to a higher standard than it should, slowing development of cleaner aviation (and therefore killing more people in the future due to tertiary effects of climate change, etc). It kind of annoys me when comment sections are filled with people talking about how incompetent Boeing is. It feels like out of shape slobs on their La-Z-boy chairs talking about how incompetent or slow some professional sports players are. Like, airliner safety is just a totally different league than almost anyone else plays in. On the worst day, their better than almost anyone else is on their best.
- ethbr1 3y ago> Airliner safety is insanely good. Because I dug it up for another comment, commercial carriers operating under Part 121 (roughly: scheduled passenger and cargo operation) had 4 fatal incidents in the last 10 years. [0] Totalling 6 deaths. In 10 years of US commercial carrier aviation. One of those was literally 'the engine exploded and threw part of the turbine into the cabin (and also shredded some of the wing)'!! Which resulted in 1 person dying and a successful landing. [0] https://news.ycombinator.com/item?id=38921664 https://news.ycombinator.com/item?id=38921664
- bb88 3y ago> Totalling 6 deaths. In 10 years of US commercial carrier aviation. Okay, awesome. But how much of that was luck with the 737 Max that they didn't crash on US soil by US airlines?
- 3y ago
- cf1241290841 3y agoCockpit resource management is also something a lot of industries can learn from. As well as human error analysis. How an error came to be is often much more interesting then the personal shortcomings of the person who caused it.
- Symbiote 3y agoAt it's best, software QA and related methods should be equal to airline manufacture. Think of railway signalling systems, control-by-wire bits of modern cars, medical equipment, etc. Where the design of the software is formally proven, and the implementation verified to ensure it fits the design.
- williamdclt 3y agoof course, but most of us aren't working on products where a quality problem would kill hundreds of people. Having aircraft-level QA would be plain silly, you don't expect that level of quality from most other industry like eg guitar manufacturing, do you?
- AdrianB1 3y agoNope. It is most probably caused by operational stress - rudder assembly is moving, fuselage is also working (compression and decompression cycles on take off and landing, thermal expansion and compression). I bet they don't just put red Loctite on it to keep it from getting loose. My bet is design flaws, not manufacturing or QA. EDIT: I saw the pictures of bolts with pins and bolts without pins. The ones with pins cannot get loose, the others can. Let's see what happened.
- 0xEF 3y agoI don't know about you, but in my industry, "QA" also means extensive testing to ensure that part/assembly/etc doesn't break with expected operations. So, yeah, from where I'm standing, this was a QA problem. Something did not get checked or tested as it probably should have.
- ooboe 3y agoThe 4 restraining nuts and bolts on the door have a cotter pin like mechanism to prevent them from loosening. If assembled correctly they cannot loosen unless the pin fails.
- justinclift 3y agoMaybe an inadequate torque specification for the bolt tightening?
- garyfirestorm 3y agoBolts are most likely tightened with a torque wrench or a gun that is set to a torque spec. Over tightening a bolt is as bad as a loose bolt. I speculate these passed QA from Boeing because they might have been correctly torqued to the spec. What happens in field is hard to understand. One possibility is vicinity to the engine can cause extreme vibrations, these can make them loose. Other possibility is the maintenance side of things - maybe a badly calibrated torque wrench could be the reason. Mechanical systems are not inherently immutable.
- FredFS456 3y agoI would expect lock wire or some other method of ensuring the bolt does not un-torque itself. Especially for bolts that are not required to be removed past final assembly...
- kevin_thibedeau 3y agoWhen it's happening on a two month old plane it's a production problem.
- onlyrealcuzzo 3y agoIt's a QA process better than almost every other industry in the world... Sure, it failed, and it isn't perfect. But planes have had a long track record of being absurdly safe.
- MilStdJunkie 3y agoGiven everything I've seen so far, I'd bet good money that what happened here was miscommunication between Spirit and Boeing. Spirit started out locking down the plug, then Boeing asked them to just loosely attach it[1] so Boeing could yank the plug for interior/wiring/AC/paint, then someone at Boeing forgot about the "loosely". So now, they get in a hurry (maybe the AC/interior didn't need any access to work on, which makes sense for this MAX variant, it wouldn't need as many hatches to pull wire) and it went down the Renton line as if the plug was fully installed. It's enough to pass high blow inspection and other inspections, but then over time that "shipment config" attachment vibrated out, and pop goes the plug. Almost certainly systemic issue though, so that sucks. Sucks real bad. They need to get a Tiger Team or whatever together to look at everything with a shipment config, and make sure those "ship kits" don't leak into the real actual airplane configuration. This is . . ok, this is really manufacturing 101 stuff, but well, things happen. I'm in the industry, but haven't touched the MAX, so take this with a grain of salt. [1] Basically a "shipping" or train configuration