7 ms·

The 787's Problems Run Deeper Than Outsourcing

92 points by hype7 14y ago

andrewcooke 14y ago
from the linked boeing paper: if outsourcing is to be employed, it is understood to be absolutely necessary that detailed parts and subassemblies be designed with that process in mind. this requires considerable additional upfront effort in planning to avoid the situation whereby major subassemblies do not fit together at final assembly, increasing the cost by orders of magnitude more than was saved by designing in isolation from the work allocation activities. forgive typos - cut + paste from pdf not working. there's more good stuff about the timescales over which outsourcing for low labour is appropriate. https://s3.amazonaws.com/s3.documentcloud.org/documents/69746/hart-smith-on-outsourcing.pdf https://s3.amazonaws.com/s3.documentcloud.org/documents/6974...
graeham 14y ago
Seems a bit pre-mature for the conclusions of this article. The article is motivated by the grounding of the plane due to battery issues, but its not yet clear what is causing these batteries problems. In otherwords, the article's argument would still be true if there were no battery issues, but I think readers are more interested in the context of these problems. Generally I agree with the thesis of the article - aircraft are incredibly complex and require both modularization and top-level design management. And the article does make a good point that this kind of framework is needed for this scale of projects - whether module are worked on "insourced" or "outsourced". I think a potential issue with outsourcing is that geographic and organizational isolation probably leads itself to tougher integration.
- willvarfar 14y ago
  The article cites a long list of delays and problems, does it not?
  - joezydeco 14y ago
    I get the feeling the market had forgotten about the delays until this current situation. Planes were shipping in a highly visible fashion.
    - jessaustin 14y ago
      Delays in the introduction of new models are extremely common in aerospace. If these battery issues had been noticed by the right people sooner the 787 would have been delayed more than it was. Very few airlines would have cancelled orders over that.
- cpleppert 14y ago
  I'm not sure the author hasn't run too far with his thesis here. Every modern airliner is outsourced to a major and entire systems must be designed by the supplier that provides them. If the requirements are understood this usually isn't an issue. Major components of an airliner are often designed and manufactured in geographically separate locations. In the case of Airbus each national entity has different processes in some cases and this has caused issues in the past. The 787 is no different in this respect where entire systems are manufactured and built by different entities and then transported for finally assembly. In the case of the 787, if the author is correct that modularization has caused the issues then one would expect that the largest issues would occur with the major elements such as wings as they are built entirely by a subcontractor in japan. Of course, there have been no such issues. The electrical problems are probably driven more by the extreme demands placed on the power supply than anything else. It wasn't as if Boeing wasn't aware of the risks involved in choosing Li-ion batteries and was just blindsided by a supplier. It made the decision and then sold it to regulatory agencies. If it didn't understand how to make the batteries safe or it wasn't comfortable with the design Boeing should have rejected it. Yes, we still don't know the root cause of the issues. But suddenly leaping from that to placing blame on McDonnell executives keen a little too keen on outsourcing just based on a hunch is a little strange to say the least.
acomjean 14y ago
I think the article gets it. I worked as a Boeing subcontractor on a radar system. Boeing really seemed to want to get into the government contract game, a different beast than commercial aviation. They were making more on defense contracts than commercial aviation (although that seems to have changed back the past couple years). quote: The thing about these Government contracts is that they are paid as development proceeds. This is entirely different — and a lot less risk.
- hammock 14y ago
  I wonder about SpaceX. They are also trying to get in the defense contracts game (already have as of Dec 2012).
  - pyre 14y ago
    They also have strong leadership. I have a feeling that as long as Elon Musk is at the helm, he won't let that kind of stuff seep into their operation. After he's gone? It could be anyone's guess. Pulling in defence contracts could affect the corporate culture long-term, but I have a feeling the Elon will try to keep it at bay while he's there. I mean, he has aspirations to be on a Mars mission himself. I can't imagine he would want the equipment to be skimped on.
- jordanb 14y ago
  Boeing is in the government contracting game. They're the second biggest military contractor after Lockheed. Military contracts are low risk for the contractor because the government absorbs all the overruns, but relatively low profit as well since the contracts specify the amount of profit the company is allowed to earn. Boeing's huge military contract business is good, steady money, but the commercial aviation business is where they take the lion's share of their profits. That being said, it seems like both Airbus (EADS) and Boeing are profoundly dysfunctional organizations that are really incapable of doing anything without consuming enormous resources. I wonder if both of them have the "military contractor" mindset built into their corporate culture, and that's why their commercial projects are a serial set of disasters. I think A&B are lucky to have each other as competitors. If a lean, hungry company ever entered the market it'd wipe the floor with both of them.
hencq 14y ago
As a counter point, outsourcing has served the car industry quite well. At one point in its lifetime Ford had it's own rubber tree farms. Nowadays car companies rely a lot on suppliers and heavily involve them in design as well. It's just very unlikely that one company has the skills and the focus to be good at everything. That said (and the article makes this point as well) suppliers have to be closely involved for this model to work. When Toyota was consistently beating American car manufacturers in terms of quality one of the biggest differences was in how they managed their suppliers. American companies tended to have multiple suppliers for the same parts and determine (sub) contracts mainly on a cost basis. Toyota on the other hand had very long term relationships with its suppliers and worked much more intensely with them.
- nickpinkston 14y ago
  I don't think this is a counterpoint. They said that it makes sense to be vertical early on. Look at the Ford Rouge Works where iron ore and rubber came in one side and Model A's came off the other. Now, it's a very mature category where it's all about refinement, not major innovation - so it's good to develop an interchangeable supplier base because you increase competition between them and you don't need to do anything radical. If you look at SpaceX for example, they make >90% of their parts because there's full-stack innovation going on constantly.
  - hencq 14y ago
    > Now, it's a very mature category where it's all about refinement, not major innovation I disagree. You are completely right that in that situation it makes sense to develop an interchangeable supplier base. as you mention it allows you to increase competition and put pressure on them to reduce costs. However, in a situation where you're still looking for innovation, like developing a new car (or plane) it makes sense to work with highly integrated suppliers. These won't be interchangeable and because you're very dependent on them, the opportunities for cost saving are probably smaller. On the other hand, because they have specialized knowledge and skills they can actually help you innovate much faster. At least that's the theory; you're right that SpaceX seems to focus on doing as much as possible in-house and it seems to work for them. A reason could be however, that the type of suppliers necessary to make this model work are just not there (yet) in their business. I guess in practice it's about finding the right balance. Insisting on doing everything yourself can lead to bad cases of NIH syndrome, while outsourcing everything leads to a loss of control.
michael_miller 14y ago
I really wish people would stop speculating about the 787s problems. Yes, it has had serious issues. But why is the author qualified to talk about them? From his bio, it's not clear that he has any aviation experience, and he doesn't appear to have any inside sources. The article is littered with phrases like "My hypothesis is that McDonnell's mindset from its defense work — minimizing the amount of capital put at risk during R&D — was applied to the 787" and "But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon". How can the author make such brazen claims without having talked to a Boeing engineer or visited the factory? I assume if he had done either, he would have noted it in the article. Only one type of person is qualified to comment on the problems: a person working at Boeing, or someone talking to him/her. Otherwise, we're just playing the Apple rumor game, piecing together tidbits of supply chain data to come up with the conclusion we want to hear.
- noiv 14y ago
  I'd like to add a second type of persons having certified its airworthiness and as a third type everybody else wondering whether the tons of flying metal labeled aircraft will stay over their head.
- caycep 14y ago
  it's a bit of a gray line - it's being published on HBR which is a "Big Name Publication" to a certain extent. that being said, it IS a blog, and blogs are for random musings such as this...
  - michael_miller 14y ago
    "Big Name Publications" do not get a free pass for speculation, in print or in blog. They gain credibility, so they can say things like "From talking to senior Boeing engineers, we uncovered..." but making up hypotheses out of thin air is shoddy journalism that should be criticized, regardless of the source.
  - AJ007 14y ago
    When I read enough poor quality articles from a publication, I never read anything from that publication again. I haven't read HBR in about 4 years.
redwood 14y ago
Odd thesis: essentially that not outsourcing, but outsourcing too early in design is the problem. Well I don't exactly think you can divorce the two...Once you make a choice to outsource larger and larger components, you're inherently losing control of the design and perhaps more importantly exact construction of those components. In so doing you open yourself to design risk. The way the author describes the primary risk of outsourcing as "business profit risk" is odd to me. Still the article is ridiculous considering the 787's battery problem, if indeed limited only to the battery, seems like it will be a fairly simple one to fix, and an isolated incident. I like how he lists three 787 problems, two of which are "smoke in cabin" and "fire" as if the latter two aren't likely the same root cause. Sure fuel leaking was an issue too. So we work around it. This is typical of the internet era where we can nitpick everything and then suddenly a multi-decade project comes along and we don't know how to handle its optimization/fix process.
- hype7 14y ago
  You might think it's odd, but it's exactly what Boeing had done in the past. And if you think the 787's problems are confined to its batteries, please read this: http://www.bbc.co.uk/news/business-21230940 http://www.bbc.co.uk/news/business-21230940 "Keith Hayward, head of research at the Royal Aeronautical Society, said that if the issue is no longer about replacing a faulty battery, it raised the prospect of Boeing having to do a major re-design. "I think people had their fingers crossed that it was a battery fault... it looks more systemic and serious to me. I suspect it could be difficult to identify the cause," he said." (Disclosure: I'm the article's author) Cheers -- james
kdsudac 14y ago
Went to see one of the Boeing leaders talk specifically about the dreamliner's development a few years ago at a Georgia Tech symposium. He made a point to mention how many different international suppliers were involved and how that was a huge challenge. e.g. They had to custom modify several 747s to increase the cargo hold to fit some of the sub-assemblies. He gave the impression that part of why there were so many different sub-contractors is to appease foreign governments--especially since many airlines are nationalized. The corporate equivalent of pork-barrel spending.
robomartin 14y ago
Oh, please folks, temper your enthusiasm to hang these guys. For all we know it might very well be grotesquely misplaced. As I stated in another thread, I'd prefer to read commentary from someone who has actually had first-level access to the relevant design and test documentation. Everything else is nothing more than conjecture. What are some of the failure modes of battery systems, including Li-based systems? OK, here's an incomplete list that has nothing whatsoever to do with the 787 case because, well, no design and test data has been released. Again, this has nothing to do with the 787. Think of it as the start of a list of things to watch out for if you were going to set out to design a battery pack and related systems. Sourcing - Latent defects due to manufacturing QA issues - With Li-based cells such things as humidity content can be critical as they might lead to the generation of methane gas during the service life of the cell. Assembly - Handling and storage problems - Mechanical damage, from dropping a cell to any number of imaginable issues. - Assembly defects that can lead to (I**2)*R heating issues Testing (both at by supplier and OEM) - Failure to detect defects (sometimes impossible if internal) - A faulty testing process that actually causes or accelerates damage to cells, perhaps only cells that have specific problems - Inadequate issue escalation workflow Electrical - (I**2)*R heating of faulty interconnects (inter-cell and elsewhere) - Component defects that can lead to (I**2)*R heating issues - Oxide formation that can lead to (I**2)*R heating issues - Interconnect crimp quality problems that can lead to (I**2)*R heating issues - Interconnect crimp quality problems that can lead to other issues - Main pack electronics problems This can include chargers and monitoring electronics - Charge termination failure or inadequacy - Thermal sensing failure or inadequacy - Pressure sensing failure or inadequacy - Fluid sensing failure or inadequacy - Inadequate semiconductor rating (for example: operating temperature) - Inadequate power semiconductor heat dissipation This doesn't necessarily mean bath thermal design. It could be caused by inadequate or faulty mounting of TIL (thermal interface materials). It could also be caused by mechanical and environmental issues affecting the TIL or the mating of the semiconductor to the heat dissipation mechanism. For example, vibration. - Inadequate or faulty failover modes - Static damage to semiconductors during electronics assembly, testing or general handing - PCB via or trace fractures, contamination or over-etching - BGA ball soldering problems (mostly only visible via x-ray inspection) - Bad, cold, fractured or chemically compromised solder joints - Dendrite buildup in lead-free solder chemistry leading to shorts in small pitch components - Unaccounted race conditions Example: pull-pull technologies without adequate dead-band - Thermally-induced failures due to changes in operating characteristics of components as temperature fluctuates. Example: The aforementioned push-pull topology where the deadband might have been adequate within a certain temperature range but causing shot-through shorts outside of this range. - Unaccounted for effects of such environmental issues as humidity. Example: Many VCO chips do not deal with humidity very well, particularly if it can condensate. - Faulty or non-existing conformal coating - Inadequate static damage counter measures (inputs, anything expose to handling) - Sub-pack electronics problems This is applicable if a large pack is made from a multitude of smaller packs, each with it's own control and monitoring electronics. Sub-pack electronics can suffer from the same issues as the main pack electronics above. Software - Myriad of code-based issues (lock-ups, endless loops, pointer problems, uncaught exceptions, etc.) - Software problems caused by hardware issues Example: Static damage to a chip causing a bit to unexpectedly flip. If not caught in software this can lead to catastrophic failure Mechanical - Faulty inspection and certification process - Inadequate design materials - Adequate materials but supplier quality issues resulted in unexpected materials issues - Inadequate design tolerances - Adequate design tolerances but faulty assembly, subcontractor or testing process caused them to be violated - Faulty fasteners - Faulty fastener application - Unexpected interference between components - Mechanical pinching or chafing of wires - Mechanical puncture damage to cells during installation or operation - Mechanical abrasion to protective cell wall or packaging - Insufficient mechanical rigidity - Insufficient venting mechanism - Insufficient containment or venting of gasses (methane), flames or heat - Mechanical issues once pack is installed - Thermal - Mechanical (pinching, squeezing, twisting, cutting) - Resonance Thermal - Inadequate thermal management - Lack of consideration of thermal runaway problem - Lack of thermal isolation between individual cells or sub-packs - Insufficient thermal sensing - Insufficient thermal overload coping mechanism (both mechanical and electronic) - A design that might cause severe differential heating of cells deep inside the pack - Inadequate electronic and mechanical fail-over measures in the case of thermal runaway - Inadequate thermal runaway detection - Cell supplier issues (quality control, testing, etc.) - Thermal issues once installed - Problematic heat exchange - Heat input from adjoining assemblies - Accidental plugging of filters, inlets, etc. Final release - Electrical testing - Mechanical testing - Environmental testing - Random destructive testing at the cell level Operational - User fails to follow published procedures - Alarms are ignored - Mechanical issues are ignored - Inadequate maintenance - Handling issues Anyhow, this is not, by a long-shot, an exhaustive list. There's a lot I am ignorant about and, of course, that stuff is missing. Feel free to expand. The point is that a proper investigation would use --at a bare minimum-- something like this and, more than likely, a book full of specs and procedures before reaching a verifiable and reproducible conclusion. Maybe it's just me. I don't know. I just don't like it when people do some of what is being done to Boeing. Particularly when they don't have access to the reams of FAA, Boeing and airline data that is required to really understand the problem. EDIT: Formatting
jamesseattle 14y ago
I worked on two airplane programs at boeing. The first was a derivative of the 747, the 747-400. The main features of this version that differed from previous 747s were going from analog to digital avionics, a slightly longer body and new (larger) wings. The schedule from project launch to first delivery was about 4 years. We were two weeks late on first delivery. I also worked on the 787. The difference between the 787 and every previous Boeing plane was almost everything. The schedule from project launch to first delivery was about 4 years. We were two years late. Moral: don't bite off more than you can chew.
Osiris 14y ago
All this speculation about what's happening from people that aren't aware of the facts reminds me of something that happened to me the other day. A top manager in my company came to give our office an all-hands meeting about management changes. During that, he claimed that my product (Online Calendar) doesn't support timezones and bashed it in front of the whole group. After the meeting my manager and I confronted him and told him that the problem was with the iOS Calendar application since we're only providing data via CalDav. He was insistent that because we have control of the data that it is, in the end, our fault and a problem with our software. My boss's boss gets involved and immediately recognized that the problem is that the phone had a setting enabled that removed automatic timezone switching for the Calendar app. A quick change to that setting and the iOS calendar was showing events in the current (Mountain) timezone rather than his home (Pacific) timezone. So, even someone who's worked in tech for a long time and has a pretty good understanding of things like protocols can still have no idea what they are talking about without all the relevant information.

92 points by hype7 14y ago

from the linked boeing paper: if outsourcing is to be employed, it is understood to be absolutely necessary that detailed parts and subassemblies be designed with that process in mind. this requires considerable additional upfront effort in planning to avoid the situation whereby major subassemblies do not fit together at final assembly, increasing the cost by orders of magnitude more than was saved by designing in isolation from the work allocation activities. forgive typos - cut + paste from pdf not working. there's more good stuff about the timescales over which outsourcing for low labour is appropriate. https://s3.amazonaws.com/s3.documentcloud.org/documents/69746/hart-smith-on-outsourcing.pdf https://s3.amazonaws.com/s3.documentcloud.org/documents/6974...

graeham 14y ago

Seems a bit pre-mature for the conclusions of this article. The article is motivated by the grounding of the plane due to battery issues, but its not yet clear what is causing these batteries problems. In otherwords, the article's argument would still be true if there were no battery issues, but I think readers are more interested in the context of these problems. Generally I agree with the thesis of the article - aircraft are incredibly complex and require both modularization and top-level design management. And the article does make a good point that this kind of framework is needed for this scale of projects - whether module are worked on "insourced" or "outsourced". I think a potential issue with outsourcing is that geographic and organizational isolation probably leads itself to tougher integration.

willvarfar 14y ago
The article cites a long list of delays and problems, does it not?
- joezydeco 14y ago
  I get the feeling the market had forgotten about the delays until this current situation. Planes were shipping in a highly visible fashion.
  - jessaustin 14y ago
    Delays in the introduction of new models are extremely common in aerospace. If these battery issues had been noticed by the right people sooner the 787 would have been delayed more than it was. Very few airlines would have cancelled orders over that.
cpleppert 14y ago
I'm not sure the author hasn't run too far with his thesis here. Every modern airliner is outsourced to a major and entire systems must be designed by the supplier that provides them. If the requirements are understood this usually isn't an issue. Major components of an airliner are often designed and manufactured in geographically separate locations. In the case of Airbus each national entity has different processes in some cases and this has caused issues in the past. The 787 is no different in this respect where entire systems are manufactured and built by different entities and then transported for finally assembly. In the case of the 787, if the author is correct that modularization has caused the issues then one would expect that the largest issues would occur with the major elements such as wings as they are built entirely by a subcontractor in japan. Of course, there have been no such issues. The electrical problems are probably driven more by the extreme demands placed on the power supply than anything else. It wasn't as if Boeing wasn't aware of the risks involved in choosing Li-ion batteries and was just blindsided by a supplier. It made the decision and then sold it to regulatory agencies. If it didn't understand how to make the batteries safe or it wasn't comfortable with the design Boeing should have rejected it. Yes, we still don't know the root cause of the issues. But suddenly leaping from that to placing blame on McDonnell executives keen a little too keen on outsourcing just based on a hunch is a little strange to say the least.

acomjean 14y ago

I think the article gets it. I worked as a Boeing subcontractor on a radar system. Boeing really seemed to want to get into the government contract game, a different beast than commercial aviation. They were making more on defense contracts than commercial aviation (although that seems to have changed back the past couple years). quote: The thing about these Government contracts is that they are paid as development proceeds. This is entirely different — and a lot less risk.

hammock 14y ago
I wonder about SpaceX. They are also trying to get in the defense contracts game (already have as of Dec 2012).
- pyre 14y ago
  They also have strong leadership. I have a feeling that as long as Elon Musk is at the helm, he won't let that kind of stuff seep into their operation. After he's gone? It could be anyone's guess. Pulling in defence contracts could affect the corporate culture long-term, but I have a feeling the Elon will try to keep it at bay while he's there. I mean, he has aspirations to be on a Mars mission himself. I can't imagine he would want the equipment to be skimped on.
jordanb 14y ago
Boeing is in the government contracting game. They're the second biggest military contractor after Lockheed. Military contracts are low risk for the contractor because the government absorbs all the overruns, but relatively low profit as well since the contracts specify the amount of profit the company is allowed to earn. Boeing's huge military contract business is good, steady money, but the commercial aviation business is where they take the lion's share of their profits. That being said, it seems like both Airbus (EADS) and Boeing are profoundly dysfunctional organizations that are really incapable of doing anything without consuming enormous resources. I wonder if both of them have the "military contractor" mindset built into their corporate culture, and that's why their commercial projects are a serial set of disasters. I think A&B are lucky to have each other as competitors. If a lean, hungry company ever entered the market it'd wipe the floor with both of them.

hencq 14y ago

As a counter point, outsourcing has served the car industry quite well. At one point in its lifetime Ford had it's own rubber tree farms. Nowadays car companies rely a lot on suppliers and heavily involve them in design as well. It's just very unlikely that one company has the skills and the focus to be good at everything. That said (and the article makes this point as well) suppliers have to be closely involved for this model to work. When Toyota was consistently beating American car manufacturers in terms of quality one of the biggest differences was in how they managed their suppliers. American companies tended to have multiple suppliers for the same parts and determine (sub) contracts mainly on a cost basis. Toyota on the other hand had very long term relationships with its suppliers and worked much more intensely with them.

nickpinkston 14y ago
I don't think this is a counterpoint. They said that it makes sense to be vertical early on. Look at the Ford Rouge Works where iron ore and rubber came in one side and Model A's came off the other. Now, it's a very mature category where it's all about refinement, not major innovation - so it's good to develop an interchangeable supplier base because you increase competition between them and you don't need to do anything radical. If you look at SpaceX for example, they make >90% of their parts because there's full-stack innovation going on constantly.
- hencq 14y ago
  > Now, it's a very mature category where it's all about refinement, not major innovation I disagree. You are completely right that in that situation it makes sense to develop an interchangeable supplier base. as you mention it allows you to increase competition and put pressure on them to reduce costs. However, in a situation where you're still looking for innovation, like developing a new car (or plane) it makes sense to work with highly integrated suppliers. These won't be interchangeable and because you're very dependent on them, the opportunities for cost saving are probably smaller. On the other hand, because they have specialized knowledge and skills they can actually help you innovate much faster. At least that's the theory; you're right that SpaceX seems to focus on doing as much as possible in-house and it seems to work for them. A reason could be however, that the type of suppliers necessary to make this model work are just not there (yet) in their business. I guess in practice it's about finding the right balance. Insisting on doing everything yourself can lead to bad cases of NIH syndrome, while outsourcing everything leads to a loss of control.

michael_miller 14y ago

I really wish people would stop speculating about the 787s problems. Yes, it has had serious issues. But why is the author qualified to talk about them? From his bio, it's not clear that he has any aviation experience, and he doesn't appear to have any inside sources. The article is littered with phrases like "My hypothesis is that McDonnell's mindset from its defense work — minimizing the amount of capital put at risk during R&D — was applied to the 787" and "But, in this instance, it wasn't so much the outsourcing, as it was the decision to modularize a complicated problem too soon". How can the author make such brazen claims without having talked to a Boeing engineer or visited the factory? I assume if he had done either, he would have noted it in the article. Only one type of person is qualified to comment on the problems: a person working at Boeing, or someone talking to him/her. Otherwise, we're just playing the Apple rumor game, piecing together tidbits of supply chain data to come up with the conclusion we want to hear.

noiv 14y ago
I'd like to add a second type of persons having certified its airworthiness and as a third type everybody else wondering whether the tons of flying metal labeled aircraft will stay over their head.
caycep 14y ago
it's a bit of a gray line - it's being published on HBR which is a "Big Name Publication" to a certain extent. that being said, it IS a blog, and blogs are for random musings such as this...
- michael_miller 14y ago
  "Big Name Publications" do not get a free pass for speculation, in print or in blog. They gain credibility, so they can say things like "From talking to senior Boeing engineers, we uncovered..." but making up hypotheses out of thin air is shoddy journalism that should be criticized, regardless of the source.
- AJ007 14y ago
  When I read enough poor quality articles from a publication, I never read anything from that publication again. I haven't read HBR in about 4 years.

redwood 14y ago

Odd thesis: essentially that not outsourcing, but outsourcing too early in design is the problem. Well I don't exactly think you can divorce the two...Once you make a choice to outsource larger and larger components, you're inherently losing control of the design and perhaps more importantly exact construction of those components. In so doing you open yourself to design risk. The way the author describes the primary risk of outsourcing as "business profit risk" is odd to me. Still the article is ridiculous considering the 787's battery problem, if indeed limited only to the battery, seems like it will be a fairly simple one to fix, and an isolated incident. I like how he lists three 787 problems, two of which are "smoke in cabin" and "fire" as if the latter two aren't likely the same root cause. Sure fuel leaking was an issue too. So we work around it. This is typical of the internet era where we can nitpick everything and then suddenly a multi-decade project comes along and we don't know how to handle its optimization/fix process.

hype7 14y ago
You might think it's odd, but it's exactly what Boeing had done in the past. And if you think the 787's problems are confined to its batteries, please read this: http://www.bbc.co.uk/news/business-21230940 http://www.bbc.co.uk/news/business-21230940 "Keith Hayward, head of research at the Royal Aeronautical Society, said that if the issue is no longer about replacing a faulty battery, it raised the prospect of Boeing having to do a major re-design. "I think people had their fingers crossed that it was a battery fault... it looks more systemic and serious to me. I suspect it could be difficult to identify the cause," he said." (Disclosure: I'm the article's author) Cheers -- james

kdsudac 14y ago

Went to see one of the Boeing leaders talk specifically about the dreamliner's development a few years ago at a Georgia Tech symposium. He made a point to mention how many different international suppliers were involved and how that was a huge challenge. e.g. They had to custom modify several 747s to increase the cargo hold to fit some of the sub-assemblies. He gave the impression that part of why there were so many different sub-contractors is to appease foreign governments--especially since many airlines are nationalized. The corporate equivalent of pork-barrel spending.

robomartin 14y ago

Oh, please folks, temper your enthusiasm to hang these guys. For all we know it might very well be grotesquely misplaced. As I stated in another thread, I'd prefer to read commentary from someone who has actually had first-level access to the relevant design and test documentation. Everything else is nothing more than conjecture. What are some of the failure modes of battery systems, including Li-based systems? OK, here's an incomplete list that has nothing whatsoever to do with the 787 case because, well, no design and test data has been released. Again, this has nothing to do with the 787. Think of it as the start of a list of things to watch out for if you were going to set out to design a battery pack and related systems. Sourcing - Latent defects due to manufacturing QA issues - With Li-based cells such things as humidity content can be critical as they might lead to the generation of methane gas during the service life of the cell. Assembly - Handling and storage problems - Mechanical damage, from dropping a cell to any number of imaginable issues. - Assembly defects that can lead to (I**2)*R heating issues Testing (both at by supplier and OEM) - Failure to detect defects (sometimes impossible if internal) - A faulty testing process that actually causes or accelerates damage to cells, perhaps only cells that have specific problems - Inadequate issue escalation workflow Electrical - (I**2)*R heating of faulty interconnects (inter-cell and elsewhere) - Component defects that can lead to (I**2)*R heating issues - Oxide formation that can lead to (I**2)*R heating issues - Interconnect crimp quality problems that can lead to (I**2)*R heating issues - Interconnect crimp quality problems that can lead to other issues - Main pack electronics problems This can include chargers and monitoring electronics - Charge termination failure or inadequacy - Thermal sensing failure or inadequacy - Pressure sensing failure or inadequacy - Fluid sensing failure or inadequacy - Inadequate semiconductor rating (for example: operating temperature) - Inadequate power semiconductor heat dissipation This doesn't necessarily mean bath thermal design. It could be caused by inadequate or faulty mounting of TIL (thermal interface materials). It could also be caused by mechanical and environmental issues affecting the TIL or the mating of the semiconductor to the heat dissipation mechanism. For example, vibration. - Inadequate or faulty failover modes - Static damage to semiconductors during electronics assembly, testing or general handing - PCB via or trace fractures, contamination or over-etching - BGA ball soldering problems (mostly only visible via x-ray inspection) - Bad, cold, fractured or chemically compromised solder joints - Dendrite buildup in lead-free solder chemistry leading to shorts in small pitch components - Unaccounted race conditions Example: pull-pull technologies without adequate dead-band - Thermally-induced failures due to changes in operating characteristics of components as temperature fluctuates. Example: The aforementioned push-pull topology where the deadband might have been adequate within a certain temperature range but causing shot-through shorts outside of this range. - Unaccounted for effects of such environmental issues as humidity. Example: Many VCO chips do not deal with humidity very well, particularly if it can condensate. - Faulty or non-existing conformal coating - Inadequate static damage counter measures (inputs, anything expose to handling) - Sub-pack electronics problems This is applicable if a large pack is made from a multitude of smaller packs, each with it's own control and monitoring electronics. Sub-pack electronics can suffer from the same issues as the main pack electronics above. Software - Myriad of code-based issues (lock-ups, endless loops, pointer problems, uncaught exceptions, etc.) - Software problems caused by hardware issues Example: Static damage to a chip causing a bit to unexpectedly flip. If not caught in software this can lead to catastrophic failure Mechanical - Faulty inspection and certification process - Inadequate design materials - Adequate materials but supplier quality issues resulted in unexpected materials issues - Inadequate design tolerances - Adequate design tolerances but faulty assembly, subcontractor or testing process caused them to be violated - Faulty fasteners - Faulty fastener application - Unexpected interference between components - Mechanical pinching or chafing of wires - Mechanical puncture damage to cells during installation or operation - Mechanical abrasion to protective cell wall or packaging - Insufficient mechanical rigidity - Insufficient venting mechanism - Insufficient containment or venting of gasses (methane), flames or heat - Mechanical issues once pack is installed - Thermal - Mechanical (pinching, squeezing, twisting, cutting) - Resonance Thermal - Inadequate thermal management - Lack of consideration of thermal runaway problem - Lack of thermal isolation between individual cells or sub-packs - Insufficient thermal sensing - Insufficient thermal overload coping mechanism (both mechanical and electronic) - A design that might cause severe differential heating of cells deep inside the pack - Inadequate electronic and mechanical fail-over measures in the case of thermal runaway - Inadequate thermal runaway detection - Cell supplier issues (quality control, testing, etc.) - Thermal issues once installed - Problematic heat exchange - Heat input from adjoining assemblies - Accidental plugging of filters, inlets, etc. Final release - Electrical testing - Mechanical testing - Environmental testing - Random destructive testing at the cell level Operational - User fails to follow published procedures - Alarms are ignored - Mechanical issues are ignored - Inadequate maintenance - Handling issues Anyhow, this is not, by a long-shot, an exhaustive list. There's a lot I am ignorant about and, of course, that stuff is missing. Feel free to expand. The point is that a proper investigation would use --at a bare minimum-- something like this and, more than likely, a book full of specs and procedures before reaching a verifiable and reproducible conclusion. Maybe it's just me. I don't know. I just don't like it when people do some of what is being done to Boeing. Particularly when they don't have access to the reams of FAA, Boeing and airline data that is required to really understand the problem. EDIT: Formatting

jamesseattle 14y ago

I worked on two airplane programs at boeing. The first was a derivative of the 747, the 747-400. The main features of this version that differed from previous 747s were going from analog to digital avionics, a slightly longer body and new (larger) wings. The schedule from project launch to first delivery was about 4 years. We were two weeks late on first delivery. I also worked on the 787. The difference between the 787 and every previous Boeing plane was almost everything. The schedule from project launch to first delivery was about 4 years. We were two years late. Moral: don't bite off more than you can chew.

Osiris 14y ago

All this speculation about what's happening from people that aren't aware of the facts reminds me of something that happened to me the other day. A top manager in my company came to give our office an all-hands meeting about management changes. During that, he claimed that my product (Online Calendar) doesn't support timezones and bashed it in front of the whole group. After the meeting my manager and I confronted him and told him that the problem was with the iOS Calendar application since we're only providing data via CalDav. He was insistent that because we have control of the data that it is, in the end, our fault and a problem with our software. My boss's boss gets involved and immediately recognized that the problem is that the phone had a setting enabled that removed automatic timezone switching for the Calendar app. A quick change to that setting and the iOS calendar was showing events in the current (Mountain) timezone rather than his home (Pacific) timezone. So, even someone who's worked in tech for a long time and has a pretty good understanding of things like protocols can still have no idea what they are talking about without all the relevant information.