5 ms·

Boeing overhauls quality controls: more high-tech tracking but fewer inspectors

46 points by wallflower 8y ago

_Understated_ 8y ago
I felt compelled to comment on this as I am somewhat conflicted in my feelings towards the automation of the checking process. I am all for progress and automation where possible but I also like to put my faith in a human (or multiple humans) then going over it again "just in case"... in this case it would be someone following behind with another torque wrench to make sure the first guy hadn't missed something or was maybe using a defective wrench. I understand that in many industries, checking samples or batches is the norm. It can make sense. But we're talking about a plane here... the margin for error must be tiny. If my car develops a fault, hell, even if a wheel fell off while driving, I have a good chance of recovering safely but with a plane, something as catastrophic as my car wheel scenario could be massive. Plus, you can't pull over to the side of the road. Also, with my security head on, I am questioning the smart wrenches (I confess to knowing nothing about them!). If the torque is electronically programmed, whats to stop hackers tweaking it a bit. Kind of like in Die Hard 2 where the bad guys reset ground level to be -200ft. How would we know?. Could someone change 300nm to be actually 200nm? I assume the inspector will look at an output to verify these things since they won't be going round with their own torque wrench now. Hackers could have a field day with this as it could be years before the plane develops a massive fault and then you'd never be able to track it. I've read enough about aircraft computer systems to know that they aren't perfect as it is! I dunno. This makes me feel a bit uneasy. I always like to think that planes are over-engineered and triple-inspected by many humans for a reason. Maybe I am looking a bit too deep here!
- jen20 8y ago
  > I also like to put my faith in a human (or multiple humans) then going over it again "just in case"... in this case it would be someone following behind with another torque wrench to make sure the first guy hadn't missed something or was maybe using a defective wrench. A second pass with a torque wrench is probably not such a bad idea, but multiple _checks_ by humans is not necessarily beneficial. A few years ago I worked for a company making small-batch pharmaceuticals (almost exclusively with ingredients drawn and mixed by hand, since the batches were so small), and the consensus (backed by research) there was that adding a second human check made quality worse rather than better - the second checker assumes that the first did the job correctly, and the person doing the work assumes that mistakes will be caught further down the line.
  - keithpeter 8y ago
    Very interesting, thanks for posting. No sampling at all? No samples retained for analysis in the event of a problem later?
    - jen20 8y ago
      Depending on the criticality of the actual substances for the given use (both of which were always known), assay of samples may have been required as an in-process control rather than an out-of-band one, too. By "small-batch" I often mean batches of one-offs, reducing the effectiveness of sampling for quality control!
      - keithpeter 8y ago
        "By "small-batch" I often mean batches of one-offs, reducing the effectiveness of sampling for quality control!" But increasing the importance of a sample retained for later analysis I imagine.
  - sitkack 8y ago
    The solution is for the checkers to be blind, in that they should have no knowledge of ordering or checks. Everything needs to be independent. It isn't that the second check made quality worse, it is that knowledge that the second check is run made the first or second check less rigorous. 1) inject faults to test the testers 2) test diversity, automated and manual 3) test order inversion The often used alternative is to "be careful" which is not rigorous.
    - jen20 8y ago
      Indeed, that is the root cause, rather than a second check itself.
- keithpeter 8y ago
  As automation moves on I think we will all have to think these issues through. Do I trust the mass of my fellow humans a bit more than an algorithm and its sensors?
  - greglindahl 8y ago
    I'd expect them to have two different sets of strengths and weaknesses.
- userbinator 8y ago
  But we're talking about a plane here... the margin for error must be tiny. They're actually surprisingly resilient, being purposefully designed that way. Look at avherald.com for all the things that happen to planes daily, yet don't make the mainstream news because no massive injury or death resulted.
  - skookumchuck 8y ago
    Another thing about airplane faults is that the pilots almost always have time to troubleshoot and take corrective action.
pytester 8y ago
Perhaps Boeing should take a step back and reconsider their drive for "safety automation" given that it didn't exactly work out so well the last time: https://www.nakedcapitalism.com/2018/11/boeing-crapification-lion-air-crash.html https://www.nakedcapitalism.com/2018/11/boeing-crapification...
- sitkack 8y ago
  I am going to excerpt a section because this needs to be known. > The short version of the story is that Boeing had implemented a new “safety” feature that operated even when its plane was being flown manually, that if it went into a stall, it would lower the nose suddenly to pick airspeed and fly normally again. However, Boeing didn’t tell its buyers or even the FAA about this new goodie. It wasn’t in pilot training or even the manuals. But even worse, this new control could force the nose down so far that it would be impossible not to crash the plane. And no, I am not making this up. From the Wall Street Journal: Boeing Co. withheld information about potential hazards associated with a new flight-control feature suspected of playing a role in last month’s fatal Lion Air jet crash, according to safety experts involved in the investigation, as well as midlevel FAA officials and airline pilots. The automated stall-prevention system on Boeing 737 MAX 8 and MAX 9 models—intended to help cockpit crews avoid mistakenly raising a plane’s nose dangerously high—under unusual conditions can push it down unexpectedly and so strongly that flight crews can’t pull it back up. Such a scenario, Boeing told airlines in a world-wide safety bulletin roughly a week after the accident, can result in a steep dive or crash—even if pilots are manually flying the jetliner and don’t expect flight-control computers to kick in. See also, https://www.aviationtoday.com/2018/11/20/lion-air-boeing-737-investigation-places-flight-controls-focus/ https://www.aviationtoday.com/2018/11/20/lion-air-boeing-737... Indonesian National Transportation Safety Committee (KNKT) officials investigating the Oct. 29 crash initially attributed the most likely cause of the accident to erroneous angle-of-attack (AOA) data. Now those same officials, and several airline pilot unions, claim the flight crew operations manual did not contain critical information for pilots regarding the maneuvering characteristics augmentation system (MCAS) designed to prevent the 737 MAX from stalling.
  - 8y ago
sitkack 8y ago
I am programmer, I don't work in Aerospace, my skin in the game is that I fly in planes and know people that fly in planes. The first thing that comes to mind is, how many flaws do these inspectors find that aren't on the list of checks? Removing the inspectors presupposes that the entire effect they have on the system is codified in measurements they take. Did they run a double blind study with fault injection? > The introduction of sampling instead of comprehensive inspections of every job is a key part of Gonzalez-Beltran’s changes. By the time the flaws arise due to these changes, the folks that instituted them will be retired or dead. I am not disagreeing with holistically transforming manufacturing processes to reduce (and eliminate error), but automated processes are efficient and without constant feedback unobserved process flaws are even harder to detect. Given the importance and criticality of such a move, I am not sure we can trust Boeing with this decision alone. > An internal Boeing document providing “talking points” for managers, obtained by The Seattle Times, points another motivation for the company: faster completion of tasks: “By allowing the mechanic to verify their own work, wait time is eliminated.” > The FAA “endorse and understand this,” Gonzalez-Beltran said. I think what Boeing has done has automated the assembly, but not necessarily the inspections. Of course mechanics should perform the operation correctly, but they shouldn't be verifying their own work. > Gonzalez-Beltran said these were isolated cases of employees not following company procedures. Priorities are not aligned towards safety, if you have people verifying their own work, they are incentivized to cheat. > Gonzalez-Beltran, however, insists he’s getting positive buy-in from mechanics who take pride in performing their work without an inspector’s oversight. So the VP asked the line worker his _feeling_, like how hungry he is and what kind of steak he likes to eat. Airplanes are not cars, while Poke-Yoke should be applied universally to all engineering pursuits, it in and of itself is not a complete solution. https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke
- skookumchuck 8y ago
  > I am not sure we can trust Boeing with this decision alone From the article: "Boeing cannot move forward with such a radical shift in its manufacturing processes without the approval of the FAA."
  - 8y ago
deleted 8y ago
[deleted]
projektfu 8y ago
Looking at the initial comments, I think people need to read up on statistical quality control before commenting with strong language about the value of inspection. Some references. Shewart, Walter. Economic Control of Quality of Manufactured Product (1931) Deming, W Edwards. Out of the Crisis (1982) Taguchi, Genichi. Taguchi’s Quality Engineering Handbook (2005)\ Essentially, inspection is too late in the process to affect the overall quality. You do need to sample and study your products but your process should have the capability to produce good products without defects and you should not be relying on inspection to produce your output. There are exceptions, such as semiconductor chip lithography, where you are always working at the limits of the process. Nonetheless, there are many parts of the process that should not require 100% inspection. For example, wire bonding and packaging. In your aircraft, would you prefer to know that it was made by a process that produces parts and assemblies that are well within their specification, or that it had been made by a defective process but inspected 100%?
- baybal2 8y ago
  > There are exceptions, such as semiconductor chip lithography, where you are always working at the limits of the process. You know, a modern semiconductor fab can have more metrologists and inspection engineers, than actual process people. The more advanced the node, the more it is a black box, where you can not intervene in the process in any way other than initial inputs.
- sokoloff 8y ago
  Light airplanes in the 60s and 70s shared a lot of automotive parts, literally made on the same production line. The interesting part is something like a voltage regulator, Cessna's specification was for 100% testing. Ford would accept a statistical sampling for QC. So, using the same line, a certain number would be inspected, stamped with a Cessna part number and sold there. The remaining assemblies would be sold as Ford parts. https://www.aircraftspruce.com/catalog/eppages/r15v00a.php https://www.aircraftspruce.com/catalog/eppages/r15v00a.php
baybal2 8y ago
From 12 years in electronics, I can say that: Humans are bad at doing tasks that are very simple, but repetitive and requiring 100% correctness. QC is one of them. Computers on other hand, are very good at doing exactly same thing, finding out common irregularities that can be preprogrammed.

46 points by wallflower 8y ago

I felt compelled to comment on this as I am somewhat conflicted in my feelings towards the automation of the checking process. I am all for progress and automation where possible but I also like to put my faith in a human (or multiple humans) then going over it again "just in case"... in this case it would be someone following behind with another torque wrench to make sure the first guy hadn't missed something or was maybe using a defective wrench. I understand that in many industries, checking samples or batches is the norm. It can make sense. But we're talking about a plane here... the margin for error must be tiny. If my car develops a fault, hell, even if a wheel fell off while driving, I have a good chance of recovering safely but with a plane, something as catastrophic as my car wheel scenario could be massive. Plus, you can't pull over to the side of the road. Also, with my security head on, I am questioning the smart wrenches (I confess to knowing nothing about them!). If the torque is electronically programmed, whats to stop hackers tweaking it a bit. Kind of like in Die Hard 2 where the bad guys reset ground level to be -200ft. How would we know?. Could someone change 300nm to be actually 200nm? I assume the inspector will look at an output to verify these things since they won't be going round with their own torque wrench now. Hackers could have a field day with this as it could be years before the plane develops a massive fault and then you'd never be able to track it. I've read enough about aircraft computer systems to know that they aren't perfect as it is! I dunno. This makes me feel a bit uneasy. I always like to think that planes are over-engineered and triple-inspected by many humans for a reason. Maybe I am looking a bit too deep here!

jen20 8y ago
> I also like to put my faith in a human (or multiple humans) then going over it again "just in case"... in this case it would be someone following behind with another torque wrench to make sure the first guy hadn't missed something or was maybe using a defective wrench. A second pass with a torque wrench is probably not such a bad idea, but multiple _checks_ by humans is not necessarily beneficial. A few years ago I worked for a company making small-batch pharmaceuticals (almost exclusively with ingredients drawn and mixed by hand, since the batches were so small), and the consensus (backed by research) there was that adding a second human check made quality worse rather than better - the second checker assumes that the first did the job correctly, and the person doing the work assumes that mistakes will be caught further down the line.
- keithpeter 8y ago
  Very interesting, thanks for posting. No sampling at all? No samples retained for analysis in the event of a problem later?
  - jen20 8y ago
    Depending on the criticality of the actual substances for the given use (both of which were always known), assay of samples may have been required as an in-process control rather than an out-of-band one, too. By "small-batch" I often mean batches of one-offs, reducing the effectiveness of sampling for quality control!
    - keithpeter 8y ago
      "By "small-batch" I often mean batches of one-offs, reducing the effectiveness of sampling for quality control!" But increasing the importance of a sample retained for later analysis I imagine.
- sitkack 8y ago
  The solution is for the checkers to be blind, in that they should have no knowledge of ordering or checks. Everything needs to be independent. It isn't that the second check made quality worse, it is that knowledge that the second check is run made the first or second check less rigorous. 1) inject faults to test the testers 2) test diversity, automated and manual 3) test order inversion The often used alternative is to "be careful" which is not rigorous.
  - jen20 8y ago
    Indeed, that is the root cause, rather than a second check itself.
keithpeter 8y ago
As automation moves on I think we will all have to think these issues through. Do I trust the mass of my fellow humans a bit more than an algorithm and its sensors?
- greglindahl 8y ago
  I'd expect them to have two different sets of strengths and weaknesses.
userbinator 8y ago
But we're talking about a plane here... the margin for error must be tiny. They're actually surprisingly resilient, being purposefully designed that way. Look at avherald.com for all the things that happen to planes daily, yet don't make the mainstream news because no massive injury or death resulted.
- skookumchuck 8y ago
  Another thing about airplane faults is that the pilots almost always have time to troubleshoot and take corrective action.

pytester 8y ago

Perhaps Boeing should take a step back and reconsider their drive for "safety automation" given that it didn't exactly work out so well the last time: https://www.nakedcapitalism.com/2018/11/boeing-crapification-lion-air-crash.html https://www.nakedcapitalism.com/2018/11/boeing-crapification...

sitkack 8y ago
I am going to excerpt a section because this needs to be known. > The short version of the story is that Boeing had implemented a new “safety” feature that operated even when its plane was being flown manually, that if it went into a stall, it would lower the nose suddenly to pick airspeed and fly normally again. However, Boeing didn’t tell its buyers or even the FAA about this new goodie. It wasn’t in pilot training or even the manuals. But even worse, this new control could force the nose down so far that it would be impossible not to crash the plane. And no, I am not making this up. From the Wall Street Journal: Boeing Co. withheld information about potential hazards associated with a new flight-control feature suspected of playing a role in last month’s fatal Lion Air jet crash, according to safety experts involved in the investigation, as well as midlevel FAA officials and airline pilots. The automated stall-prevention system on Boeing 737 MAX 8 and MAX 9 models—intended to help cockpit crews avoid mistakenly raising a plane’s nose dangerously high—under unusual conditions can push it down unexpectedly and so strongly that flight crews can’t pull it back up. Such a scenario, Boeing told airlines in a world-wide safety bulletin roughly a week after the accident, can result in a steep dive or crash—even if pilots are manually flying the jetliner and don’t expect flight-control computers to kick in. See also, https://www.aviationtoday.com/2018/11/20/lion-air-boeing-737-investigation-places-flight-controls-focus/ https://www.aviationtoday.com/2018/11/20/lion-air-boeing-737... Indonesian National Transportation Safety Committee (KNKT) officials investigating the Oct. 29 crash initially attributed the most likely cause of the accident to erroneous angle-of-attack (AOA) data. Now those same officials, and several airline pilot unions, claim the flight crew operations manual did not contain critical information for pilots regarding the maneuvering characteristics augmentation system (MCAS) designed to prevent the 737 MAX from stalling.
- 8y ago

sitkack 8y ago

I am programmer, I don't work in Aerospace, my skin in the game is that I fly in planes and know people that fly in planes. The first thing that comes to mind is, how many flaws do these inspectors find that aren't on the list of checks? Removing the inspectors presupposes that the entire effect they have on the system is codified in measurements they take. Did they run a double blind study with fault injection? > The introduction of sampling instead of comprehensive inspections of every job is a key part of Gonzalez-Beltran’s changes. By the time the flaws arise due to these changes, the folks that instituted them will be retired or dead. I am not disagreeing with holistically transforming manufacturing processes to reduce (and eliminate error), but automated processes are efficient and without constant feedback unobserved process flaws are even harder to detect. Given the importance and criticality of such a move, I am not sure we can trust Boeing with this decision alone. > An internal Boeing document providing “talking points” for managers, obtained by The Seattle Times, points another motivation for the company: faster completion of tasks: “By allowing the mechanic to verify their own work, wait time is eliminated.” > The FAA “endorse and understand this,” Gonzalez-Beltran said. I think what Boeing has done has automated the assembly, but not necessarily the inspections. Of course mechanics should perform the operation correctly, but they shouldn't be verifying their own work. > Gonzalez-Beltran said these were isolated cases of employees not following company procedures. Priorities are not aligned towards safety, if you have people verifying their own work, they are incentivized to cheat. > Gonzalez-Beltran, however, insists he’s getting positive buy-in from mechanics who take pride in performing their work without an inspector’s oversight. So the VP asked the line worker his _feeling_, like how hungry he is and what kind of steak he likes to eat. Airplanes are not cars, while Poke-Yoke should be applied universally to all engineering pursuits, it in and of itself is not a complete solution. https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke

skookumchuck 8y ago
> I am not sure we can trust Boeing with this decision alone From the article: "Boeing cannot move forward with such a radical shift in its manufacturing processes without the approval of the FAA."
- 8y ago

deleted 8y ago

[deleted]

projektfu 8y ago

Looking at the initial comments, I think people need to read up on statistical quality control before commenting with strong language about the value of inspection. Some references. Shewart, Walter. Economic Control of Quality of Manufactured Product (1931) Deming, W Edwards. Out of the Crisis (1982) Taguchi, Genichi. Taguchi’s Quality Engineering Handbook (2005)\ Essentially, inspection is too late in the process to affect the overall quality. You do need to sample and study your products but your process should have the capability to produce good products without defects and you should not be relying on inspection to produce your output. There are exceptions, such as semiconductor chip lithography, where you are always working at the limits of the process. Nonetheless, there are many parts of the process that should not require 100% inspection. For example, wire bonding and packaging. In your aircraft, would you prefer to know that it was made by a process that produces parts and assemblies that are well within their specification, or that it had been made by a defective process but inspected 100%?

baybal2 8y ago
> There are exceptions, such as semiconductor chip lithography, where you are always working at the limits of the process. You know, a modern semiconductor fab can have more metrologists and inspection engineers, than actual process people. The more advanced the node, the more it is a black box, where you can not intervene in the process in any way other than initial inputs.
sokoloff 8y ago
Light airplanes in the 60s and 70s shared a lot of automotive parts, literally made on the same production line. The interesting part is something like a voltage regulator, Cessna's specification was for 100% testing. Ford would accept a statistical sampling for QC. So, using the same line, a certain number would be inspected, stamped with a Cessna part number and sold there. The remaining assemblies would be sold as Ford parts. https://www.aircraftspruce.com/catalog/eppages/r15v00a.php https://www.aircraftspruce.com/catalog/eppages/r15v00a.php

baybal2 8y ago

From 12 years in electronics, I can say that: Humans are bad at doing tasks that are very simple, but repetitive and requiring 100% correctness. QC is one of them. Computers on other hand, are very good at doing exactly same thing, finding out common irregularities that can be preprogrammed.