11 ms·

I found it interesting that Boeing did proactively tell airlines to inspect 737 MAXs for a possible loose bolt in a different part of the plane (rudder section)

• nolok 3y ago
  Imagine the quality of the manufacturing and QA / final inspection to have that kind of issue.
  • ethbr1 3y ago
    I expect even at its worst, software development could learn a lot from aircraft QA. Especially since most shops have pretty much tossed professional career QA out the window.
    • darylteo 3y ago
      software development: what is QA?
      • bb88 3y ago
        That's where failed software devs / management go to.
    • ShadowBanThis01 3y ago
      Except, of course, from Boeing's aircraft-software QA... which killed hundreds of people already.
      • actionfromafar 3y ago
        The problem was not really the software in isolation, but that pilots expected the 737 NG to behave exactly like the old version - because Boeing decided it was too expensive to retrain pilots.
        • WalterBright 3y ago
          The expense for retraining pilots falls on the airline. Retraining has its own problems. No matter how well retraining is done, pilots still make mistakes from doing the right thing for the previous plane that is the wrong thing for the one they are currently flying. Adjusting airplanes to fly the same way is a major safety advantage.
          • ethbr1 3y ago
            Arguably, Boeing hit the uncanny safety valley -- similar enough so that pilots and airlines relaxed, but different enough so that relaxation ultimately killed people.
            • WalterBright 3y ago
              The emergency procedure for runaway trim was the same for both aircraft types, and was not followed. After the first crash, an Emergency Airworthiness Directive was issued to all MAX pilots reiterating the procedure, which was not followed in the second crash, as well as not reacting to an overspeed warning. Unreported by the media, there was another MAX incident before the first crash. The crew had no knowledge of MCAS, but did follow the emergency runaway trim procedure, and continued the flight and landed safely.
              • 9659 3y ago
                "Runaway stab trim". It is a memory item, every pilot should be able to perform it from memory. Turn off the motor, and the trim is manual. There is a crank right there in the cockpit. If it is too hard to turn, change aircraft configuration to reduce the forces required to. Pilot know how to do this. This pilot stuff, they understand the forces on the flight controls and what impacts them. Boeing made an engineering mistake. The pilots also made an operational mistake. Unfortunately, both mistakes at the same time were fatal. I pray that pilot training has improved. And that Boeing has made systems level changes to the aircraft that will preclude it happening in the future. And that is how aviation becomes safer every year; at a significant cost of customers lives.
    • echohack5 3y ago
      QA was literally invented for the airline industry. Software QA when actually practiced is more advanced now than airline QA.
      • paranoidrobot 3y ago
        > Software QA when actually practiced is more advanced now than airline QA. ...eh, I think "when actually practiced" is doing a lot of carrying there. What do you mean by "actually practiced". Outside of the aerospace and healthcare industries, I'm not sure there are many software shops that are doing QA to a level I would like to trust anyone's life with.
      • serf 3y ago
        what does advanced mean when comparing things so unlike from each other? also software is the least likely comparison I would have made; software quality is a shit-show on a general level, and the vast public is quite aware of this every time a subway timeboard blue-screens or gets frozen on an AMI screen, or the POS machine that they're forced to interact with at work does something equally as stupid.
      • booleandilemma 3y ago
        Nah, in the software world, the truth is QA is where the people who can't get jobs as programmers end up. I've seen testers go on to become programmers, but I've never seen a programmer become a tester. Maybe it's different for real-time or life-critical systems, sure, but I can confidently say this is how it is in web development.
    • vegetablepotpie 3y ago
      I work in the defense industry, it’s very much like the aerospace industry in that we deal with human life as a consequence of our work. We have software QA departments that operate very much like manufacturing or aerospace QA. Software QA provides nothing of value to software development; having it as a dedicated function works against the overtly stated goals of the function and counterintuitively acts to degrade quality within software by mandating strict top down process and brittle end-to-end testing. Although Software QA is intended to be an independent verification body that provides engineering organizations with tools and resources, in practice they function as a moral crumple zone [1] within the complex socio-technical defense industrial system, being one of the groups that the finger will be pointed to when something goes wrong and absorb shock to the business in the event of a failure. As a result they have a strong incentive to highly systematize their work with specific process steps, to shield them from liability, which can be applied generically to all projects. Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. This acts to find problems quickly and resolve them quickly. Software QAs need for high level, top down, generic systemization requires them to work against these principles in practice. Bespoke project specific checks, such as unit testing, is not viewed as contributing to the final product and is discouraged by leadership who see it as waste. To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue. [1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236
      • Eisenstein 3y ago
        It sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.
    • rectang 3y ago
      Is it plausible that Boeing has "learned" from software/startup/venture-capital culture with regards to tolerating higher risk to minimize costs? I suspect it's rather a case of parallel evolution between McDonnell Douglas brass and software startup culture, since cost-cutting culture goes back many decades (remember "Chainsaw" Al Dunlap[1] ?) — but I wonder if there's a more direct influence. [1] https://en.wikipedia.org/wiki/Albert_J._Dunlap https://en.wikipedia.org/wiki/Albert_J._Dunlap
      • Groxx 3y ago
        In lots of ways, the "learning" there would just be "capitalism". It's inherently short-sighted unless forced to do otherwise by legislation. Cutting small corners pays off A LOT until the hammer falls, so there's a massive advantage to doing it / you need to do it if competition is doing it, or you eventually shut down as they take all your business. It's inherently a race to the bottom. Sometimes that's a net gain, sometimes it isn't.
        • boringg 3y ago
          But all those communist airlines had no problems at all - exceptional build quality and operational efficiency! /S
          • mc32 3y ago
            Yep, Chernobyl being a prime example. Or Komarov's failed re-entry after complaining about the design faults of the vehicle long before launch. Then there was the more uhhm run of the mill backyard blast furnace campaign which contributed to misallocation of workforce which then led to mass starvation.
            • boringg 3y ago
              There are many many more examples. I find it so tiresome to see young people just use capitalism as a catch all for the failure of something. It's such a lazy and uninformed argument. I'm not carte blanche defending capitalism - its a mixed bag but it sure outpaces the competing systems put forward to date. It does need some stronger safeguards against industry self regulation - that has a bad track record.
    • dilyevsky 3y ago
      Would you pay at least 2x for your software to have couple more nines of reliability? I’m gonna guess that “no”. At places where it costs $$$ to have bugs shipped to the end customer (e.g phones) or where there’re regulatory requirements they still have dedicated qa.
      • bb88 3y ago
        It depends. 1% of 10000 is 100. .01% is 1. If someone came up to me and said, "Hey I can save you 99% of expected costs with 1% of your profit.", I might go for it.
        • dilyevsky 3y ago
          Which is what i said in second part of my comment. For most software businesses the cost of shipping a bug is trivial and/or poorly measured so due to McNamara fallacy it is readily exchanged for well measured cost of having a functioning qa team
    • Robotbeat 3y ago
      I agree. I’m regularly critical of Boeing Defense (particularly space contracts where I’m a huge Boeing skeptic), but I think people are pretty off base if they think Boeing is just completely incompetent. Airliner safety is insanely good. Just vast seas of competence, but when there’s a super rare failure, the incorrect impression people get is that Boeing (or Airbus) is just full of incompetency. Almost nothing that humans do is held to the same standard. Not spaceflight, not software, not healthcare, and certainly not automotive. Flying a 737 Max with a bad door and without the fix to the angle of attack sensor is probably still better per mile than driving. In spite of going at 10 times the speed and miles above the Earth. You can almost argue it’s held to a higher standard than it should, slowing development of cleaner aviation (and therefore killing more people in the future due to tertiary effects of climate change, etc). It kind of annoys me when comment sections are filled with people talking about how incompetent Boeing is. It feels like out of shape slobs on their La-Z-boy chairs talking about how incompetent or slow some professional sports players are. Like, airliner safety is just a totally different league than almost anyone else plays in. On the worst day, their better than almost anyone else is on their best.
      • ethbr1 3y ago
        > Airliner safety is insanely good. Because I dug it up for another comment, commercial carriers operating under Part 121 (roughly: scheduled passenger and cargo operation) had 4 fatal incidents in the last 10 years. [0] Totalling 6 deaths. In 10 years of US commercial carrier aviation. One of those was literally 'the engine exploded and threw part of the turbine into the cabin (and also shredded some of the wing)'!! Which resulted in 1 person dying and a successful landing. [0] https://news.ycombinator.com/item?id=38921664 https://news.ycombinator.com/item?id=38921664
        • bb88 3y ago
          > Totalling 6 deaths. In 10 years of US commercial carrier aviation. Okay, awesome. But how much of that was luck with the 737 Max that they didn't crash on US soil by US airlines?
          • 3y ago
    • cf1241290841 3y ago
      Cockpit resource management is also something a lot of industries can learn from. As well as human error analysis. How an error came to be is often much more interesting then the personal shortcomings of the person who caused it.
    • Symbiote 3y ago
      At it's best, software QA and related methods should be equal to airline manufacture. Think of railway signalling systems, control-by-wire bits of modern cars, medical equipment, etc. Where the design of the software is formally proven, and the implementation verified to ensure it fits the design.
    • williamdclt 3y ago
      of course, but most of us aren't working on products where a quality problem would kill hundreds of people. Having aircraft-level QA would be plain silly, you don't expect that level of quality from most other industry like eg guitar manufacturing, do you?
  • AdrianB1 3y ago
    Nope. It is most probably caused by operational stress - rudder assembly is moving, fuselage is also working (compression and decompression cycles on take off and landing, thermal expansion and compression). I bet they don't just put red Loctite on it to keep it from getting loose. My bet is design flaws, not manufacturing or QA. EDIT: I saw the pictures of bolts with pins and bolts without pins. The ones with pins cannot get loose, the others can. Let's see what happened.
    • 0xEF 3y ago
      I don't know about you, but in my industry, "QA" also means extensive testing to ensure that part/assembly/etc doesn't break with expected operations. So, yeah, from where I'm standing, this was a QA problem. Something did not get checked or tested as it probably should have.
    • ooboe 3y ago
      The 4 restraining nuts and bolts on the door have a cotter pin like mechanism to prevent them from loosening. If assembled correctly they cannot loosen unless the pin fails.
  • justinclift 3y ago
    Maybe an inadequate torque specification for the bolt tightening?
  • garyfirestorm 3y ago
    Bolts are most likely tightened with a torque wrench or a gun that is set to a torque spec. Over tightening a bolt is as bad as a loose bolt. I speculate these passed QA from Boeing because they might have been correctly torqued to the spec. What happens in field is hard to understand. One possibility is vicinity to the engine can cause extreme vibrations, these can make them loose. Other possibility is the maintenance side of things - maybe a badly calibrated torque wrench could be the reason. Mechanical systems are not inherently immutable.
    • FredFS456 3y ago
      I would expect lock wire or some other method of ensuring the bolt does not un-torque itself. Especially for bolts that are not required to be removed past final assembly...
    • kevin_thibedeau 3y ago
      When it's happening on a two month old plane it's a production problem.
  • onlyrealcuzzo 3y ago
    It's a QA process better than almost every other industry in the world... Sure, it failed, and it isn't perfect. But planes have had a long track record of being absurdly safe.
  • MilStdJunkie 3y ago
    Given everything I've seen so far, I'd bet good money that what happened here was miscommunication between Spirit and Boeing. Spirit started out locking down the plug, then Boeing asked them to just loosely attach it[1] so Boeing could yank the plug for interior/wiring/AC/paint, then someone at Boeing forgot about the "loosely". So now, they get in a hurry (maybe the AC/interior didn't need any access to work on, which makes sense for this MAX variant, it wouldn't need as many hatches to pull wire) and it went down the Renton line as if the plug was fully installed. It's enough to pass high blow inspection and other inspections, but then over time that "shipment config" attachment vibrated out, and pop goes the plug. Almost certainly systemic issue though, so that sucks. Sucks real bad. They need to get a Tiger Team or whatever together to look at everything with a shipment config, and make sure those "ship kits" don't leak into the real actual airplane configuration. This is . . ok, this is really manufacturing 101 stuff, but well, things happen. I'm in the industry, but haven't touched the MAX, so take this with a grain of salt. [1] Basically a "shipping" or train configuration
• AdrianB1 3y ago
  They are not related. Probably different types of bolts, for sure different stress types. Rudder assembly is a moving part, these false door panels are not.
  • MBCook 3y ago
    I think the point is there were at least two sets of loose bolts at the same time.
  • akira2501 3y ago
    > these false door panels are not. It's not a control surface, but it is a "moving part." That's what's baffling to me, that they spent a lot of effort building this hinge and pin roller system, and designed the door to hinge open up to 15 degrees. It makes me wonder if there's maintenance procedures that at some point would require the operation of that door to successfully complete. Otherwise, the mechanism itself seems so incredibly overwrought, with lots of additional bolts, castle nuts, retaining pins, and even sprung hinges at the bottom. Does anyone know why this "plug-type non-plug door" is built this way?
    • Scoundreller 3y ago
      my guess is: to replace gaskets/seals
    • paranoidrobot 3y ago
      It's built to be an actual emergency exit. It needs to be usable depending on how many passengers the interior is configured for. So it has all of the door bits there. Maybe some parts like the emergency escape slide are not installed. e: I should be clear that it's not usable as an emergency exit, as configured by Alaska. However the operator could choose to activate it later and install a usable exit.
      • Pasorrijer 3y ago
        This is not correct. To the passengers, this just looks like another seat next to a window with a plug installed. It's not a door. If there was a reconfiguration to a seating standard that required the extra exit, the plug would be removed and a proper door would be installed, with the associated interior pieces.
• mihaaly 3y ago
  I recall a running joke from my childhood - from a former communist East European counry - about a certain car saying you should finish the assembly at home after purchase, tightening the screws before first use. Despite being a famously poor quality car - even in the sloppy East European practices - that supposed to be a joke not to follow suit!
• otikik 3y ago
  That’s “testing in production”, but beast-sized.