5 ms·

FAA faulted for outsourcing 787 safety checks to Boeing

47 points by Create 14y ago

tjr 14y ago
I don't know the details of this specific situation (beyond the article here), but the scenario in general is not uncommon. Employees at avionics companies can receive FAA training to act as internal FAA delegates, reviewing work in line with FAA guidelines, just as an employee of the FAA would do.
- itsmeduncan 14y ago
  Came here to agree with you. Most of the repair stations have in house employees that are designated by the FAA to approve any changes. DER is what it is called specifically. This happens because the FAA does not have the resources to look at every single thing on an aircraft.
  - sageikosa 14y ago
    If they did, airline ticket prices would be astronomical.
tlb 14y ago
I disagree with the premise that the FAA might understand modern aircraft design better than Boeing. Or that Boeing doesn't have plenty of incentive to make the aircraft very safe.
- fnordfnordfnord 14y ago
  But you can't disagree that this is a classic example of conflicting interests, and exactly the sort of thing that should be avoided when so many lives are at stake.
  - tlb 14y ago
    Whose interest do you think was served by inadequate battery testing? I'm pretty sure Boeing isn't laughing all the way to the bank right now.
    - justinsteele 14y ago
      Boeing's calculation of a battery reliability threshold would be related to cost of manufacturing vs cost of potential lawsuits related to failures. If they are highly unlikely to fail, and the cost of manufacturing them to an even higher spec is more than the accepted risk, they will not do it. Since the FAA cares less about Boeing's profits and more about the public's safety (hopefully), they would likely have a higher threshold.
      - HeyLaughingBoy 14y ago
        Safety (ability to cause harm) and reliability (potential to not meet requirements) thresholds are different animals. I work in a regulated industry (medical) and I can assure you that I have never seen a Safety threshold that was in any way related to manufacturing or any other cost. The thresholds are always set according to how much harm can be caused and the likelihood of a harmful event. Reliability, on the other hand, has tradeoffs relating to costs. However, in this case, an unreliable subsystem can have a clear safety impact, so its reliability directly feeds into the Hazard calculations. In other words, Safety concerns would dominate. I hope that actually conveys what I'm trying to say. A bit rushed right now :-)
    - ceejayoz 14y ago
      Maybe they thought they could get away with it, and the magnitude of the problem comes as a surprise?
    - pfortuny 14y ago
      Short-term PR and benefits: these are much more important than they may seem to be. And also do not forget "keeping my job for these last two years". After all, "we know we do the battery thing right at Boeing, so there can be nothing wrong with them." Imagine employee A delaying the trials of the "most modern aircraft in the world" because some batteries are "malfunctioning", HE SAYS SO AND IS STILL WORKING FOR US? It is not just Boeing having a conflict of interests, it is its employees as well.
- archangel_one 14y ago
  But it rather defeats the point of having a regulatory body performing safety checks if the manufacturer does those itself, no?
  - rayiner 14y ago
    It doesn't defeat the point necessarily, it's just a design point for an allocation of labor that involves various trade-offs. Boeing doesn't just give the FAA a single sheet of paper saying "we've signed off on it, it's all good!" At the same time, the FAA doesn't take delivery of a 787 and do all the tests from scratch by itself. Instead, the FAA specifies process, and Boeing does tests and gives the results to the FAA, and the two sides iterate until the FAA is happy.
  - tjr 14y ago
    In theory, no, because the Boeing employees should be performing the reviews in accordance with FAA guidelines. If they aren't, that's a problem. But there are so many checks and balances in reviewing aviation engineering work that it seems unlikely that FAA guidelines would get blown off, even if someone at Boeing wanted to.
  - protomyth 14y ago
    It actually happens quite a lot in many industries in the US. Not that the government inspectors have a much better track record than the companies themselves. The penalty for screwing it up is particularly high in some areas. I know of one case (food) where the penalties (government onsite workers & other measures) to the operation of a facility applied even after it is was sold to a competitor.
- SeanDav 14y ago
  To think that a company will police itself adequately in the face of losing money if rules are enforced correctly is naive in the extreme. Since when has big business ever cared more about safety than the bottom line?
  - larrik 14y ago
    Normally this would be totally true, but given that every airliner crash is worldwide headlines, safety IS their bottom line.
    - SeanDav 14y ago
      Safety is always a compromise. The more you spend on safety, the less money you will make. Having the company itself decide when something is "safe enough" is a recipe for disaster, quite literally.
    - MichaelApproved 14y ago
      You'd think that would make sense but when deadlines, profit margins and bonuses are on the line, people tend to take shortcuts and sacrifice safty. Easy example is BP oil spill. You'd think it would be in their best interest to double up on safty because a spill would cost massively more than the safty precautions to prevent it. You'd think. But we see that they take risks and cut corners to meet their deadline while disregarding safty.
    - acdha 14y ago
      What about Challenger? Thiokol engineers wanted to scrub the launch but were overridden by their management over business concerns: http://www.npr.org/blogs/thetwo-way/2012/02/06/146490064/remembering-roger-boisjoly-he-tried-to-stop-shuttle-challenger-launch http://www.npr.org/blogs/thetwo-way/2012/02/06/146490064/rem... It's not exactly the same situation but I think it illustrates the need to have independent oversight from someone outside the same management hierarchy – otherwise it's just too easy to cut corners, particularly since the feedback usually isn't immediate.
      - 14y ago
- CamperBob2 14y ago
  Simple. Airbus gets to do the design reviews at Boeing, and vice versa. The FAA is just the referee. :-P
  - SilasX 14y ago
    Hey, that's a good idea! We could have a system where two interested parties that want opposite outcomes could present their evidence to a neutral committee, with a government agent determine which arguments can be admitted! We shall call it "the adversarial system" and test-run it in the legal world!
- alexqgb 14y ago
  This is wrong. And that's because safety deals with probabilities, not absolutes. In the absence of absolutes, people who are also responding to contrary pressures (time, money, etc.) can find themselves shifting the lines between likely, unlikely, highly unlikely, and statistically impossible. This is especially true when dealing with complex systems that defy attempts to model interacting forces perfectly. [EDIT] The 787 is conspicuously (and very expensively) behind schedule. It also represents an especially large investment of ego on the part of senior management, who were basically at war with their unions, and developed the globally distributed production line for the 787 in response to their conflicts with workers in Seattle. This included the construction of a fleet of supersized jets that could move fuselage size parts-in-progress around the world, meaning parts of the 787 assembly line are, quite literally, airborne. In other words, Boeing is no longer operating in the realm of tried-and-true. To the contrary, they're pushing aircraft manufacturing into uncharted territory. The premise that this was an intelligent response to labor issues has already come under serious fire, in that the distributed assembly line has been credited with an inordinate number of the delays they're suffering from (relying on sub-contracted manufacturers in Italy, for instance, proved especially costly). So there's already a bit of a siege mentality happening at the highest levels of Boeing HQ, and a lot of pressure to pull off what is, in retrospect, looking like a major miscalculation. Put simply, if they were going to err on the side of caution, they wouldn't have pioneered outsourcing on this scale using their flagship product in the first place. When the "appetite for risk" comes from the top down, it's much harder for subordinates to draw the right lines.
fnordfnordfnord 14y ago
The sad/pathetic part of this is that Boeing are infinitely more qualified to make these assessments. The scary/pathetic part is that when faced with such a problem, the FAA's choices were: 1. Punt the responsibility to Boeing and hope it works out. Or 2. Just tell them "no" as we've been telling everyone else who wants to use whatever post-60's technology they've chosen for their aircraft.
- ims 14y ago
  I don't necessarily disagree about those things being sad/scary/pathetic. But these things cost money. Taxpayers are generally averse to spending money on long term prevention. Something that people consider an "expensive boondoggle" and refuse to pay for before an incident tends to become "shame on you for not doing your job" right after. Do you remember after the BP spill how people were shocked -- shocked! -- that the government did not have deep sea oil rig submersibles and that we had to rely on BP's resources? This type of thinking blew my mind. Who on earth would have supported paying for deep sea utility submersibles for some "black swan" oil spill before this incident happened? And the funny part is, now that it's over and everybody has moved on, they probably still wouldn't. So back to the FAA... how are we going to pay for top tier engineering talent in the kinds of numbers required to check these components at the level of detail we might wish? TL;DR: proper oversight costs money.
- patmcguire 14y ago
  This is a growing problem in a lot of industries, sort of similar to what Joel Spolsky calls the "Lord Palmerston" principle http://www.joelonsoftware.com/articles/LordPalmerston.html http://www.joelonsoftware.com/articles/LordPalmerston.html Everything's so big, but you can't learn any faster than people before you, when everything was smaller, so you focus on a tiny area. In the 60's an amateur mechanic could understand and tweak a car, now they're so complex a profession mechanical engineer would have a hard time. How many people could have actually understand the issue with the 787 or with mortgage CDO's before anything went wrong? Only a handful, and they're generally the ones making the things.
- neurotech1 14y ago
  Checking if something has a 1 in 10m probability of failure, is the wrong idea. The question should be 'What if it does?' and that wasn't adequately answered here. There didn't seem to be any thermal isolation between cells, and a questionably risky type of Lithium-Ion chemistry.
millerc 14y ago
I don't see how the FAA could do differently, given the budget cuts and the rule that prevents them from spending more than 40 hours on a project. Until they can justify splitting up a battery into a multitude of projects (N projects on the anode, N projects on the electrolytes, N projects on the casing, ...), or they are given the means to show their expertise, the checks and balances are engineered for failure from the administration level.
jstalin 14y ago
It's a bit of a tradeoff. If the FAA fails at some regulatory function, you can't sue the government. If Boeing fails, at least you can sue them.
- eli 14y ago
  Who said you can't sue the government?
  - CamperBob2 14y ago
    The government. You can sue them only if they give you permission to do so.
  - showerst 14y ago
    It's called sovereign immunity. Basically you can only sue the US government over things where it has agreed to let you, like things covered by The Federal Tort Claims Act. http://en.wikipedia.org/wiki/Sovereign_immunity_in_the_United_States http://en.wikipedia.org/wiki/Sovereign_immunity_in_the_Unite... http://en.wikipedia.org/wiki/Federal_Tort_Claims_Act http://en.wikipedia.org/wiki/Federal_Tort_Claims_Act
    - eli 14y ago
      If you got hurt because the FAA failed to do its job wouldn't that be a perfect example of a case where you could successfully sue the government?
      - jstalin 14y ago
        Not unless Congress gave you permission to, which they haven't.
        eli 14y ago
        IANAL, but I'm pretty sure you're wrong. The Federal Tort Act explains the procedures (and puts some limits on) how you can sue the FAA, but it certainly doesn't prohibit it. Here's 3 seconds of googling: http://www.firstcoastnews.com/news/local/story.aspx?storyid=149194 http://www.firstcoastnews.com/news/local/story.aspx?storyid=... http://www.10news.com/news/faa-settles-lawsuit-with-family-of-deceased-pilot http://www.10news.com/news/faa-settles-lawsuit-with-family-o...
protomyth 14y ago
I'm actual surprised its not the NTSB that would do the checks. Its seems like, given the funding, that would be a natural fit and help later in the airplane's life.
Metapony 14y ago
I can see outsourcing a few of them, but that many? Wow. Just wow.

47 points by Create 14y ago

tjr 14y ago

I don't know the details of this specific situation (beyond the article here), but the scenario in general is not uncommon. Employees at avionics companies can receive FAA training to act as internal FAA delegates, reviewing work in line with FAA guidelines, just as an employee of the FAA would do.

itsmeduncan 14y ago
Came here to agree with you. Most of the repair stations have in house employees that are designated by the FAA to approve any changes. DER is what it is called specifically. This happens because the FAA does not have the resources to look at every single thing on an aircraft.
- sageikosa 14y ago
  If they did, airline ticket prices would be astronomical.

tlb 14y ago

I disagree with the premise that the FAA might understand modern aircraft design better than Boeing. Or that Boeing doesn't have plenty of incentive to make the aircraft very safe.

fnordfnordfnord 14y ago
But you can't disagree that this is a classic example of conflicting interests, and exactly the sort of thing that should be avoided when so many lives are at stake.
- tlb 14y ago
  Whose interest do you think was served by inadequate battery testing? I'm pretty sure Boeing isn't laughing all the way to the bank right now.
  - justinsteele 14y ago
    Boeing's calculation of a battery reliability threshold would be related to cost of manufacturing vs cost of potential lawsuits related to failures. If they are highly unlikely to fail, and the cost of manufacturing them to an even higher spec is more than the accepted risk, they will not do it. Since the FAA cares less about Boeing's profits and more about the public's safety (hopefully), they would likely have a higher threshold.
    - HeyLaughingBoy 14y ago
      Safety (ability to cause harm) and reliability (potential to not meet requirements) thresholds are different animals. I work in a regulated industry (medical) and I can assure you that I have never seen a Safety threshold that was in any way related to manufacturing or any other cost. The thresholds are always set according to how much harm can be caused and the likelihood of a harmful event. Reliability, on the other hand, has tradeoffs relating to costs. However, in this case, an unreliable subsystem can have a clear safety impact, so its reliability directly feeds into the Hazard calculations. In other words, Safety concerns would dominate. I hope that actually conveys what I'm trying to say. A bit rushed right now :-)
  - ceejayoz 14y ago
    Maybe they thought they could get away with it, and the magnitude of the problem comes as a surprise?
  - pfortuny 14y ago
    Short-term PR and benefits: these are much more important than they may seem to be. And also do not forget "keeping my job for these last two years". After all, "we know we do the battery thing right at Boeing, so there can be nothing wrong with them." Imagine employee A delaying the trials of the "most modern aircraft in the world" because some batteries are "malfunctioning", HE SAYS SO AND IS STILL WORKING FOR US? It is not just Boeing having a conflict of interests, it is its employees as well.
archangel_one 14y ago
But it rather defeats the point of having a regulatory body performing safety checks if the manufacturer does those itself, no?
- rayiner 14y ago
  It doesn't defeat the point necessarily, it's just a design point for an allocation of labor that involves various trade-offs. Boeing doesn't just give the FAA a single sheet of paper saying "we've signed off on it, it's all good!" At the same time, the FAA doesn't take delivery of a 787 and do all the tests from scratch by itself. Instead, the FAA specifies process, and Boeing does tests and gives the results to the FAA, and the two sides iterate until the FAA is happy.
- tjr 14y ago
  In theory, no, because the Boeing employees should be performing the reviews in accordance with FAA guidelines. If they aren't, that's a problem. But there are so many checks and balances in reviewing aviation engineering work that it seems unlikely that FAA guidelines would get blown off, even if someone at Boeing wanted to.
- protomyth 14y ago
  It actually happens quite a lot in many industries in the US. Not that the government inspectors have a much better track record than the companies themselves. The penalty for screwing it up is particularly high in some areas. I know of one case (food) where the penalties (government onsite workers & other measures) to the operation of a facility applied even after it is was sold to a competitor.
SeanDav 14y ago
To think that a company will police itself adequately in the face of losing money if rules are enforced correctly is naive in the extreme. Since when has big business ever cared more about safety than the bottom line?
- larrik 14y ago
  Normally this would be totally true, but given that every airliner crash is worldwide headlines, safety IS their bottom line.
  - SeanDav 14y ago
    Safety is always a compromise. The more you spend on safety, the less money you will make. Having the company itself decide when something is "safe enough" is a recipe for disaster, quite literally.
  - MichaelApproved 14y ago
    You'd think that would make sense but when deadlines, profit margins and bonuses are on the line, people tend to take shortcuts and sacrifice safty. Easy example is BP oil spill. You'd think it would be in their best interest to double up on safty because a spill would cost massively more than the safty precautions to prevent it. You'd think. But we see that they take risks and cut corners to meet their deadline while disregarding safty.
  - acdha 14y ago
    What about Challenger? Thiokol engineers wanted to scrub the launch but were overridden by their management over business concerns: http://www.npr.org/blogs/thetwo-way/2012/02/06/146490064/remembering-roger-boisjoly-he-tried-to-stop-shuttle-challenger-launch http://www.npr.org/blogs/thetwo-way/2012/02/06/146490064/rem... It's not exactly the same situation but I think it illustrates the need to have independent oversight from someone outside the same management hierarchy – otherwise it's just too easy to cut corners, particularly since the feedback usually isn't immediate.
    - 14y ago
CamperBob2 14y ago
Simple. Airbus gets to do the design reviews at Boeing, and vice versa. The FAA is just the referee. :-P
- SilasX 14y ago
  Hey, that's a good idea! We could have a system where two interested parties that want opposite outcomes could present their evidence to a neutral committee, with a government agent determine which arguments can be admitted! We shall call it "the adversarial system" and test-run it in the legal world!
alexqgb 14y ago
This is wrong. And that's because safety deals with probabilities, not absolutes. In the absence of absolutes, people who are also responding to contrary pressures (time, money, etc.) can find themselves shifting the lines between likely, unlikely, highly unlikely, and statistically impossible. This is especially true when dealing with complex systems that defy attempts to model interacting forces perfectly. [EDIT] The 787 is conspicuously (and very expensively) behind schedule. It also represents an especially large investment of ego on the part of senior management, who were basically at war with their unions, and developed the globally distributed production line for the 787 in response to their conflicts with workers in Seattle. This included the construction of a fleet of supersized jets that could move fuselage size parts-in-progress around the world, meaning parts of the 787 assembly line are, quite literally, airborne. In other words, Boeing is no longer operating in the realm of tried-and-true. To the contrary, they're pushing aircraft manufacturing into uncharted territory. The premise that this was an intelligent response to labor issues has already come under serious fire, in that the distributed assembly line has been credited with an inordinate number of the delays they're suffering from (relying on sub-contracted manufacturers in Italy, for instance, proved especially costly). So there's already a bit of a siege mentality happening at the highest levels of Boeing HQ, and a lot of pressure to pull off what is, in retrospect, looking like a major miscalculation. Put simply, if they were going to err on the side of caution, they wouldn't have pioneered outsourcing on this scale using their flagship product in the first place. When the "appetite for risk" comes from the top down, it's much harder for subordinates to draw the right lines.

fnordfnordfnord 14y ago

The sad/pathetic part of this is that Boeing are infinitely more qualified to make these assessments. The scary/pathetic part is that when faced with such a problem, the FAA's choices were: 1. Punt the responsibility to Boeing and hope it works out. Or 2. Just tell them "no" as we've been telling everyone else who wants to use whatever post-60's technology they've chosen for their aircraft.

ims 14y ago
I don't necessarily disagree about those things being sad/scary/pathetic. But these things cost money. Taxpayers are generally averse to spending money on long term prevention. Something that people consider an "expensive boondoggle" and refuse to pay for before an incident tends to become "shame on you for not doing your job" right after. Do you remember after the BP spill how people were shocked -- shocked! -- that the government did not have deep sea oil rig submersibles and that we had to rely on BP's resources? This type of thinking blew my mind. Who on earth would have supported paying for deep sea utility submersibles for some "black swan" oil spill before this incident happened? And the funny part is, now that it's over and everybody has moved on, they probably still wouldn't. So back to the FAA... how are we going to pay for top tier engineering talent in the kinds of numbers required to check these components at the level of detail we might wish? TL;DR: proper oversight costs money.
patmcguire 14y ago
This is a growing problem in a lot of industries, sort of similar to what Joel Spolsky calls the "Lord Palmerston" principle http://www.joelonsoftware.com/articles/LordPalmerston.html http://www.joelonsoftware.com/articles/LordPalmerston.html Everything's so big, but you can't learn any faster than people before you, when everything was smaller, so you focus on a tiny area. In the 60's an amateur mechanic could understand and tweak a car, now they're so complex a profession mechanical engineer would have a hard time. How many people could have actually understand the issue with the 787 or with mortgage CDO's before anything went wrong? Only a handful, and they're generally the ones making the things.
neurotech1 14y ago
Checking if something has a 1 in 10m probability of failure, is the wrong idea. The question should be 'What if it does?' and that wasn't adequately answered here. There didn't seem to be any thermal isolation between cells, and a questionably risky type of Lithium-Ion chemistry.

millerc 14y ago

I don't see how the FAA could do differently, given the budget cuts and the rule that prevents them from spending more than 40 hours on a project. Until they can justify splitting up a battery into a multitude of projects (N projects on the anode, N projects on the electrolytes, N projects on the casing, ...), or they are given the means to show their expertise, the checks and balances are engineered for failure from the administration level.

jstalin 14y ago

It's a bit of a tradeoff. If the FAA fails at some regulatory function, you can't sue the government. If Boeing fails, at least you can sue them.

eli 14y ago
Who said you can't sue the government?
- CamperBob2 14y ago
  The government. You can sue them only if they give you permission to do so.
- showerst 14y ago
  It's called sovereign immunity. Basically you can only sue the US government over things where it has agreed to let you, like things covered by The Federal Tort Claims Act. http://en.wikipedia.org/wiki/Sovereign_immunity_in_the_United_States http://en.wikipedia.org/wiki/Sovereign_immunity_in_the_Unite... http://en.wikipedia.org/wiki/Federal_Tort_Claims_Act http://en.wikipedia.org/wiki/Federal_Tort_Claims_Act
  - eli 14y ago
    If you got hurt because the FAA failed to do its job wouldn't that be a perfect example of a case where you could successfully sue the government?
    - jstalin 14y ago
      Not unless Congress gave you permission to, which they haven't.
      - eli 14y ago
        IANAL, but I'm pretty sure you're wrong. The Federal Tort Act explains the procedures (and puts some limits on) how you can sue the FAA, but it certainly doesn't prohibit it. Here's 3 seconds of googling: http://www.firstcoastnews.com/news/local/story.aspx?storyid=149194 http://www.firstcoastnews.com/news/local/story.aspx?storyid=... http://www.10news.com/news/faa-settles-lawsuit-with-family-of-deceased-pilot http://www.10news.com/news/faa-settles-lawsuit-with-family-o...

protomyth 14y ago

I'm actual surprised its not the NTSB that would do the checks. Its seems like, given the funding, that would be a natural fit and help later in the airplane's life.

Metapony 14y ago

I can see outsourcing a few of them, but that many? Wow. Just wow.