7 ms·

A very thorough preliminary report. I've worked for a long time in quality systems, and this is a perfect example of a systemic failure. They've got work bein

A very thorough preliminary report. I've worked for a long time in quality systems, and this is a perfect example of a systemic failure. They've got work being handed off between Boeing employees and 3rd party contractors with insufficient controls in place to verify that very basic tasks are being performed. I'd be curious to know how many non-conformances they typically see during assembly of a plane and whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them. I'm guessing that the production personnel are under tremendous time constraints and are constantly pressure the quality assurance people to sign off on whatever paperwork is holding up the line, no matter the safety implications. Also, I think a lot of middle and upper level management needs to lose their jobs over this. I hope this mess ends up in textbooks and gets beaten into the head of every MBA student in the country.

lp4vn 3y ago
>I'd be curious to know [...] whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them If management in the aerospace industry works like management in the software industry, then I guess they are pushing for results as agressively as possible without much concern about safety or anything else.
- cezart 3y ago
  At a company I worked in, we had a joke about this: "Good thing we don't build nuclear reactors". In some software projects the level of rush, and the fact that bugs sometimes would leak into production was kinda horrifying. It would've been way more so, if it would've been the kind of project that could kill people in case of failure. Like it happened in Chernobyl with nuclear reactors, or at Boeing with planes. I can't really imagine what these engineers feel when they rush this kind of work knowing what's at stake.
  - toast0 3y ago
    Quality is always a trade-off. If you're deeply into economics, you wonder about the trade-offs of cost to find defects before shipping, difference in cost of addressing defects before and after shipping including costs of mitigation from consequences of defects, % of defects that will never be found after shipping (and are therefore a real cost savings), and in the long game costs of having a reputation for shipping product with defects that could have been reasonably detected. In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste. Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those. In Aerospace, addressing defects after shipping is very expensive, and mitigating the effects of defects is only approximate; you can't restore passengers from backup, economic damages don't really make families whole, but should be an incentive not to let reasonably detectable defects be shipped.
    - wmidwestranger 3y ago
      > Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those. This assumes you're fortunate enough to have a defect at the outer edge of the system. Most times, these problems are created in the initial rush of pushing something out and then tax every effort that depends on them, forever, and ever.
      - masfuerte 3y ago
        Amen.
    - Turing_Machine 3y ago
      As the old saying goes: "Fast, cheap, or good. You can pick a maximum of two."
    - lp4vn 3y ago
      >In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste. It's really a shame that a good percentage of these applications full of bugs and "rapidly changing and undocumented requirements" don't get scrapped and stay many decades afloat until they get replaced by another application also full of bugs and "rapidly changing and undocumented requirements". I think that that's a very sad way of seeing things honestly. In the past the USA put the man on the moon, today repeating the same feat looks almost impossible. I bet that a lot of managers at Boeing also think that building planes like a few decades ago looks almost impossible now.
- trompetenaccoun 3y ago
  That's not how it traditionally worked in aerospace. Commercial airliners are such a safe mode of transportation because from start to finish there are (or used to be?) very high safety standards all around. I can't speak for what's going on with Boeing currently though. Things certainly seem to have deteriorated.
shadowgovt 3y ago
Rumor has it the controls are there, but subvert-able. Apparently, there are two ticketing systems (one for "history of plane," Boeing internal, and one for "day-to-day onsite work," visible by contractors and Boeing management). The work to fix the rivets was logged in the day-to-day, but management and the onsite staff managed to convince themselves that merely opening the plug to fix the vacuum-seal trim did not constitute "removing" the plug, and since there was only an entry in the history-of-plane log for removing, not opening, they didn't log it there (when the intent was "there's no entry for 'just opening' because there's no such thing as 'just opening', breaching the pressure vessel at all constitutes 'removal of plug'"). The final inspection that should have caught the error would have been triggered by the update in the history-of-plane ticketing queue. (And as for 'how many non-conformances,' the same source claims that Spirit is one of the few subcontractors with on-site staff at the factory because their parent company delivers such consistently shoddy out-of-compliance product that they are continuously doing final warrenty-work onsite. So maybe "fire that vendor" should be on the docket too).
- ethbr1 3y ago
  You might want to cite your source on this, which I'm guessing is the purported insider speaking about same? From the portions of the report that hint at corroboration, > Documents and photos show that to perform the replacement of the damaged rivets, access to the rivets required opening the left MED plug (see figure 15). To open the MED plug, the two vertical movement arrestor bolts and two upper guide track bolts had to be removed. > Records show the rivets were replaced per engineering requirements on Non-Conformance (NC) Order 145-8987-RSHK-1296-002NC completed on September 19, 2023, by Spirit AeroSystems personnel. Photo documentation obtained from Boeing shows evidence of the left-hand MED plug closed with no retention hardware (bolts) in the three visible locations (the aft upper guide track is covered with insulation and cannot be seen in the photo)
  - MBCook 3y ago
    Yes what they said about two systems was from the insider account that was posted online.
  - 3y ago
onetimeuse92304 3y ago
> I'd be curious to know how many non-conformances they typically see during assembly of a plane (...) Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible. For example, better quality control usually results in an increase of number of defects, at least temporarily. But that just because large portion of these defects were undetected before. So... you are looking at a number that you have nothing to compare to that also depends on how closely the process is monitored and also depends a lot on the definition of what is non-conformance. It is like trying to give an answer to "what is the length of Britain's coastline?" Everybody knows that you can get whatever answer you want depending on how long the ruler is.
- outworlder 3y ago
  > Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible. That, by itself, should be the kind of thing that should shutdown a company permanently. Remember, this is the aviation industry, where they track the mine where the ore from the bolt was mined, and who tightened the bolt up and with which torque wrench.
  - whythre 3y ago
    Sounds like a lot of steps and paperwork… steps and paperwork that might easily and regularly get fudged.
    - l33tman 3y ago
      The idea is that the system as a whole should be resilient to a certain degree of this, as it will be caught in other places or at least reported at a (hopefully non-fatal) aviation incident that leads to a report and analysis, at which point you backtrack and issue recommendations that actually have clout (like, you can't fly with your plane anywhere in the world if you don't fix this). Of course if the entire industry is corrupt it doesn't work, but to a certain degree I guess it is robust. This time it led to a potential disaster but it will become safer as a result..
mihaaly 3y ago
I have tendency to hope for unrealistic things too. Not a reliable trait of mine, no. But in my clear moments I am afraid that much sooner will come the world peace and union of all nations and religions than unimaginative but determined bean counters learn from millions of catastrophes of the past and future to come giving up pushing their core value and first rule of 'take more, give less' into the infinity and beyond. And giving up their personal wealth with it.
- zmgsabst 3y ago
  I think the hope is people of good conscience stop being political cowards. If you believe in quality engineering, but refuse to engage in the political and business dimensions of an enterprise to fight for that view, then you are just virtue signaling — since you’re refusing to engage with the tools needed to make it happen.
  - lazide 3y ago
    First, being cowards this way would need to stop being the wise choice to make.
    - pooper 3y ago
      At a minimum, we must abolish the right to work at a federal level. Sounds so terrible, doesn't it? Pooper is against the right to work. Pooper wants to take away jobs. No, we can't have that because of politics. And thus, we are stuck.
  - mihaaly 3y ago
    It is always refreshing to meet fellow naives like yourself, I like the company of the likely minded! Engaging in today's political theatre and culture (that is the keen slave of forceful bean counters/collectors btw.) with the prediction of constructive advancement for the quality of life, that is some very hardcore stuff! Even for me. (what I can do beyond raising voice is to show example rather than piss into the headwind hurricane. and hope others will follow, or I can follow. but definitelly not replacing my inner Gandhi with Rocky in a blink and getting into a fistfight with scores of agressive pro boxers simultaneously who are itching to run over anyone in their way. also I can avoid the products of those bean people, not feeding them with my purchases or assistance any way, encouraging others doing the same. vermins should starve, not bloom!)
aniftythrifrty 3y ago
[flagged]
- malfist 3y ago
  That's not what a union does
- xenadu02 3y ago
  On the contrary Boeing has spent the past 2-3 decades trying to screw over the workers as hard as possible. They've massively outsourced parts manufacturing. They don't even make the fuselages anymore! The whole point is to make those people contractors so the contractors can hire cheaper labor and even push work to cheap overseas locations. Can a system like that be made to work? Yes it certainly can. But it says something about Boeing's priorities: find the cheapest labor possible to goose profits (management bonuses).
michaelt 3y ago
> I'd be curious to know how many non-conformances they typically see during assembly of a plane Well, the report says "During the build process, one quality notification (QN NW0002407062) was noted indicating the seal flushness was out of tolerance by 0.01 inches. So I'd say they've had about 2,407,062 quality issues :)
- Arrath 3y ago
  Then its revealed "Well the numbering schema started at QN AXX... and we looped through the alphabet a few times.."
wmidwestranger 3y ago
> Also, I think a lot of middle and upper level management needs to lose their jobs over this. Given that if a worker doing the work raised this problem or took initiative to resolve it, they'ld probably be punished; I completely agree. Reminds me so much of how, once-upon-a-time, there seemed to be actual engineering management in a cooperatively adversarial relationship with business managers but not anymore. Now any sort of engineering in business seems to be completely business managed and business minded. I'm sure it's great for profits while it lasts but I haven't observed engineering becoming better and I suspect business is suffering by overextending itself too, I just don't have any solid observations. (Well, maybe one, my brother does warehouse / logistics management and says, despite there being every reason in the world, he has never seen the accounting software and the inventory software successfully and productively linked. So, big opportunity there for a serious player but maybe not the profitable compared to the issue?)
- wolpoli 3y ago
  Once upon a time, QAs and developers also had a bit of cooperatively adversarial relationship as well. It's no more now as teams are restructured by business, so they cooperate.
deleted 3y ago
[deleted]
rationalfaith 3y ago
[dead]
hipsterstal1n 3y ago
I was speaking with some friends at a Christmas party who work for the Navy - they’ve taken deliveries of planes from Boeing with the same sort of issues that start in the factory. They even went as far as to say the whole lot of planes should’ve been rejected but weren’t. Multiple things not built to spec. The planes they worked on did not share an assembly line with the 737 but another Boeing model…
- fransje26 3y ago
  If I recall properly, sometime after the MAX crash saga, the Air Force simply refused to take delivery of their 767s because of unacceptable build issues, including foreign debris (aluminium rests and shavings) and forgotten tools flying around. I didn't follow how that issue evolved.
fransje26 3y ago
According to the "insider source", 392 non-conforming defects in the fuselage door installation in the last 365 calendar days. > As a result, this check job that should find minimal defects has in the past 365 calendar days recorded 392 nonconforming findings on 737 mid fuselage door installations (so both actual doors for the high density configs, and plugs like the one that blew out). That is a hideously high and very alarming number, and if our quality system on 737 was healthy, it would have stopped the line and driven the issue back to supplier after the first few instances. Source: https://leehamnews.com/2024/01/15/unplanned-removal-installation-inspection-procedure-at-boeing/#comment-509962 https://leehamnews.com/2024/01/15/unplanned-removal-installa...
TurkishPoptart 3y ago
"....after the left mid exit door (MED) plug departed the airplane leading to a rapid decompression" Lol, they said the door plug "departed" instead of "blew the f* off"

lp4vn 3y ago

>I'd be curious to know [...] whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them If management in the aerospace industry works like management in the software industry, then I guess they are pushing for results as agressively as possible without much concern about safety or anything else.

cezart 3y ago
At a company I worked in, we had a joke about this: "Good thing we don't build nuclear reactors". In some software projects the level of rush, and the fact that bugs sometimes would leak into production was kinda horrifying. It would've been way more so, if it would've been the kind of project that could kill people in case of failure. Like it happened in Chernobyl with nuclear reactors, or at Boeing with planes. I can't really imagine what these engineers feel when they rush this kind of work knowing what's at stake.
- toast0 3y ago
  Quality is always a trade-off. If you're deeply into economics, you wonder about the trade-offs of cost to find defects before shipping, difference in cost of addressing defects before and after shipping including costs of mitigation from consequences of defects, % of defects that will never be found after shipping (and are therefore a real cost savings), and in the long game costs of having a reputation for shipping product with defects that could have been reasonably detected. In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste. Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those. In Aerospace, addressing defects after shipping is very expensive, and mitigating the effects of defects is only approximate; you can't restore passengers from backup, economic damages don't really make families whole, but should be an incentive not to let reasonably detectable defects be shipped.
  - wmidwestranger 3y ago
    > Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those. This assumes you're fortunate enough to have a defect at the outer edge of the system. Most times, these problems are created in the initial rush of pushing something out and then tax every effort that depends on them, forever, and ever.
    - masfuerte 3y ago
      Amen.
  - Turing_Machine 3y ago
    As the old saying goes: "Fast, cheap, or good. You can pick a maximum of two."
  - lp4vn 3y ago
    >In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste. It's really a shame that a good percentage of these applications full of bugs and "rapidly changing and undocumented requirements" don't get scrapped and stay many decades afloat until they get replaced by another application also full of bugs and "rapidly changing and undocumented requirements". I think that that's a very sad way of seeing things honestly. In the past the USA put the man on the moon, today repeating the same feat looks almost impossible. I bet that a lot of managers at Boeing also think that building planes like a few decades ago looks almost impossible now.
trompetenaccoun 3y ago
That's not how it traditionally worked in aerospace. Commercial airliners are such a safe mode of transportation because from start to finish there are (or used to be?) very high safety standards all around. I can't speak for what's going on with Boeing currently though. Things certainly seem to have deteriorated.

shadowgovt 3y ago

Rumor has it the controls are there, but subvert-able. Apparently, there are two ticketing systems (one for "history of plane," Boeing internal, and one for "day-to-day onsite work," visible by contractors and Boeing management). The work to fix the rivets was logged in the day-to-day, but management and the onsite staff managed to convince themselves that merely opening the plug to fix the vacuum-seal trim did not constitute "removing" the plug, and since there was only an entry in the history-of-plane log for removing, not opening, they didn't log it there (when the intent was "there's no entry for 'just opening' because there's no such thing as 'just opening', breaching the pressure vessel at all constitutes 'removal of plug'"). The final inspection that should have caught the error would have been triggered by the update in the history-of-plane ticketing queue. (And as for 'how many non-conformances,' the same source claims that Spirit is one of the few subcontractors with on-site staff at the factory because their parent company delivers such consistently shoddy out-of-compliance product that they are continuously doing final warrenty-work onsite. So maybe "fire that vendor" should be on the docket too).

ethbr1 3y ago
You might want to cite your source on this, which I'm guessing is the purported insider speaking about same? From the portions of the report that hint at corroboration, > Documents and photos show that to perform the replacement of the damaged rivets, access to the rivets required opening the left MED plug (see figure 15). To open the MED plug, the two vertical movement arrestor bolts and two upper guide track bolts had to be removed. > Records show the rivets were replaced per engineering requirements on Non-Conformance (NC) Order 145-8987-RSHK-1296-002NC completed on September 19, 2023, by Spirit AeroSystems personnel. Photo documentation obtained from Boeing shows evidence of the left-hand MED plug closed with no retention hardware (bolts) in the three visible locations (the aft upper guide track is covered with insulation and cannot be seen in the photo)
- MBCook 3y ago
  Yes what they said about two systems was from the insider account that was posted online.
- 3y ago

onetimeuse92304 3y ago

> I'd be curious to know how many non-conformances they typically see during assembly of a plane (...) Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible. For example, better quality control usually results in an increase of number of defects, at least temporarily. But that just because large portion of these defects were undetected before. So... you are looking at a number that you have nothing to compare to that also depends on how closely the process is monitored and also depends a lot on the definition of what is non-conformance. It is like trying to give an answer to "what is the length of Britain's coastline?" Everybody knows that you can get whatever answer you want depending on how long the ruler is.

outworlder 3y ago
> Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible. That, by itself, should be the kind of thing that should shutdown a company permanently. Remember, this is the aviation industry, where they track the mine where the ore from the bolt was mined, and who tightened the bolt up and with which torque wrench.
- whythre 3y ago
  Sounds like a lot of steps and paperwork… steps and paperwork that might easily and regularly get fudged.
  - l33tman 3y ago
    The idea is that the system as a whole should be resilient to a certain degree of this, as it will be caught in other places or at least reported at a (hopefully non-fatal) aviation incident that leads to a report and analysis, at which point you backtrack and issue recommendations that actually have clout (like, you can't fly with your plane anywhere in the world if you don't fix this). Of course if the entire industry is corrupt it doesn't work, but to a certain degree I guess it is robust. This time it led to a potential disaster but it will become safer as a result..

mihaaly 3y ago

I have tendency to hope for unrealistic things too. Not a reliable trait of mine, no. But in my clear moments I am afraid that much sooner will come the world peace and union of all nations and religions than unimaginative but determined bean counters learn from millions of catastrophes of the past and future to come giving up pushing their core value and first rule of 'take more, give less' into the infinity and beyond. And giving up their personal wealth with it.

zmgsabst 3y ago
I think the hope is people of good conscience stop being political cowards. If you believe in quality engineering, but refuse to engage in the political and business dimensions of an enterprise to fight for that view, then you are just virtue signaling — since you’re refusing to engage with the tools needed to make it happen.
- lazide 3y ago
  First, being cowards this way would need to stop being the wise choice to make.
  - pooper 3y ago
    At a minimum, we must abolish the right to work at a federal level. Sounds so terrible, doesn't it? Pooper is against the right to work. Pooper wants to take away jobs. No, we can't have that because of politics. And thus, we are stuck.
- mihaaly 3y ago
  It is always refreshing to meet fellow naives like yourself, I like the company of the likely minded! Engaging in today's political theatre and culture (that is the keen slave of forceful bean counters/collectors btw.) with the prediction of constructive advancement for the quality of life, that is some very hardcore stuff! Even for me. (what I can do beyond raising voice is to show example rather than piss into the headwind hurricane. and hope others will follow, or I can follow. but definitelly not replacing my inner Gandhi with Rocky in a blink and getting into a fistfight with scores of agressive pro boxers simultaneously who are itching to run over anyone in their way. also I can avoid the products of those bean people, not feeding them with my purchases or assistance any way, encouraging others doing the same. vermins should starve, not bloom!)

aniftythrifrty 3y ago

[flagged]

malfist 3y ago
That's not what a union does
xenadu02 3y ago
On the contrary Boeing has spent the past 2-3 decades trying to screw over the workers as hard as possible. They've massively outsourced parts manufacturing. They don't even make the fuselages anymore! The whole point is to make those people contractors so the contractors can hire cheaper labor and even push work to cheap overseas locations. Can a system like that be made to work? Yes it certainly can. But it says something about Boeing's priorities: find the cheapest labor possible to goose profits (management bonuses).

michaelt 3y ago

> I'd be curious to know how many non-conformances they typically see during assembly of a plane Well, the report says "During the build process, one quality notification (QN NW0002407062) was noted indicating the seal flushness was out of tolerance by 0.01 inches. So I'd say they've had about 2,407,062 quality issues :)

Arrath 3y ago
Then its revealed "Well the numbering schema started at QN AXX... and we looped through the alphabet a few times.."

wmidwestranger 3y ago

> Also, I think a lot of middle and upper level management needs to lose their jobs over this. Given that if a worker doing the work raised this problem or took initiative to resolve it, they'ld probably be punished; I completely agree. Reminds me so much of how, once-upon-a-time, there seemed to be actual engineering management in a cooperatively adversarial relationship with business managers but not anymore. Now any sort of engineering in business seems to be completely business managed and business minded. I'm sure it's great for profits while it lasts but I haven't observed engineering becoming better and I suspect business is suffering by overextending itself too, I just don't have any solid observations. (Well, maybe one, my brother does warehouse / logistics management and says, despite there being every reason in the world, he has never seen the accounting software and the inventory software successfully and productively linked. So, big opportunity there for a serious player but maybe not the profitable compared to the issue?)

wolpoli 3y ago
Once upon a time, QAs and developers also had a bit of cooperatively adversarial relationship as well. It's no more now as teams are restructured by business, so they cooperate.

deleted 3y ago

[deleted]

rationalfaith 3y ago

[dead]

hipsterstal1n 3y ago

I was speaking with some friends at a Christmas party who work for the Navy - they’ve taken deliveries of planes from Boeing with the same sort of issues that start in the factory. They even went as far as to say the whole lot of planes should’ve been rejected but weren’t. Multiple things not built to spec. The planes they worked on did not share an assembly line with the 737 but another Boeing model…

fransje26 3y ago
If I recall properly, sometime after the MAX crash saga, the Air Force simply refused to take delivery of their 767s because of unacceptable build issues, including foreign debris (aluminium rests and shavings) and forgotten tools flying around. I didn't follow how that issue evolved.

fransje26 3y ago

According to the "insider source", 392 non-conforming defects in the fuselage door installation in the last 365 calendar days. > As a result, this check job that should find minimal defects has in the past 365 calendar days recorded 392 nonconforming findings on 737 mid fuselage door installations (so both actual doors for the high density configs, and plugs like the one that blew out). That is a hideously high and very alarming number, and if our quality system on 737 was healthy, it would have stopped the line and driven the issue back to supplier after the first few instances. Source: https://leehamnews.com/2024/01/15/unplanned-removal-installation-inspection-procedure-at-boeing/#comment-509962 https://leehamnews.com/2024/01/15/unplanned-removal-installa...

TurkishPoptart 3y ago

"....after the left mid exit door (MED) plug departed the airplane leading to a rapid decompression" Lol, they said the door plug "departed" instead of "blew the f* off"