6 ms·

Money saved by firing experienced engs and outsourcing the development << money lost because they can't sell planes. Why is that so hard to understand? Massive

Money saved by firing experienced engs and outsourcing the development << money lost because they can't sell planes. Why is that so hard to understand? Massive fail at risk management maybe?

cronix 7y ago
Can't sell planes, or fly the ones that exist, and pay out huge untold sums in death lawsuits.
- crorella 7y ago
  Yeah, that too, after the fact seems so stupidly obvious.
meowface 7y ago
For one, in part because many managers don't really understand software or software projects and think it's mostly just people * man-hours. "Through this cost-saving measure, we've tripled the amount of people working on the project and expect to reduce the amount of time it'll take to complete by two-thirds without incurring any additional costs." In their minds, it's an easy win. And they were probably regularly reassured by various internal and external entities that the code was high-quality and thoroughly tested and very safe. Obviously they're not going to outsource if they think it's likely to cause a safety hazard; they just didn't place the odds of this outcome anywhere near what the actual odds ended up being.
- weaksauce 7y ago
  someone needs to throw the mythical man month at these managers.
  - durnygbur 7y ago
    There would be no englightment happening.
- nobodyandproud 7y ago
  Which means they have no business managing. It’s to tiresome to repeat because it should be obvious , but engineering is not a commodity. The end product may be, but not the design problem process nor the manufacturing process.
  - xiphias2 7y ago
    For me the solution is quite easy: I don't invest in companies where I don't see technical excellence happening. The world is changing fast enough that old style management won't get investment after some time.
- deleted 7y ago
  [deleted]
pjc50 7y ago
The risk management is being TBTF. In a nationalist era, is the US going to buy planes from Airbus, or are they going to bail out Boeing?
- broahmed 7y ago
  For others wondering: TBTF = too big to fail
dongping 7y ago
The managers who saved money would have been promoted well before any accident happens.
bsder 7y ago
> Why is that so hard to understand? Massive fail at risk management maybe? Managerial bonus or promotion for saving money this quarter >> money lost more quarters out than I'll be around
- gscott 7y ago
  This is really it. Board of directors set goals, say reducing costs, they offer a big bonus, you meet the goal, get the big bonus. Problems become someone else's proble. You buy a big car, house, go to fancy non-profit's $500 dinners.
  - gotdevops 7y ago
    This is definitely one major issue. At the top. The challenge at the top is that these outsourcing contracts are for many millions of dollars. They are often written in such a way that the parent cannot dictate how the outsourcing firm conducts business. The parent fills out a ticket and the outsourcer assigns the ticket to a resource and marks the ticket complete. The issue here is that this is not a bug related to email or Login credentials. This is an airplane with folks relying on it for safety of flight.
rossdavidh 7y ago
I have seen this kind of problem before (never in a case where lives were on the line), and I called it "pathological optimism". If you are in a corporate culture of the sort that lionizes "can-do" thinking, then saying "that won't work" is going to look defeatist. I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. But more likely, it's just that pathological optimism works, really well, right up until that moment when it doesn't. Taking risks in order to lower costs makes the bottom line better, which seems to call for more of the same, right up until the moment when the risk explodes in your face. It is not unlike the attitude towards software security, in many industries. People harping about security risks seems like just all cost and no benefit, right up until the point where you have a big problem.
- CydeWeys 7y ago
  > I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. If so, they're learning the wrong lessons. SV unicorns become valuable because of explosive growth, not because they underpay their engineers. Just the opposite; a lot of engineers at SV startups are paid quite handsomely (especially the early ones at successful unicorns). SV startups don't win on cost-cutting.
  - gotdevops 7y ago
    I think you are right on with this. Most startups also leveraging an organizational culture that goes against silos or fiefdoms. In the startup culture I’ve been a part of fast feedback and early identification of flaws in the process is a cornerstone of the development process. Troubleshooting issues in production is the worse case scenario. In this case production is literally a flying airplane.
ganeshkrishnan 7y ago
The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little. The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed. The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall. For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunities-overseas/vacancies-in-india.html https://www.airbus.com/careers/search-and-apply/opportunitie... Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.
- pxue 7y ago
  Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?
  - ganeshkrishnan 7y ago
    https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidents https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs
    - gotdevops 7y ago
      It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22
- gotdevops 7y ago
  You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22
RachelF 7y ago
Most managers only think short-term. In the short term it saved money. In the long term is cost Boeing their reputation. But in the long term the MBA manager has been likely been promoted and is working elsewhere, killing another company, while his or her CV boasts of "reducing inefficiencies".
- ensiferum 7y ago
  As long as they can get a RSU package and get a promotion it's s win.
tarsinge 7y ago
Unfortunately it seems other critical industries are affected, reminded me of the EPR nuclear reactor fiasco https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluoto_3_(Finland) https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluot... Olkiluoto (Finland) > In May 2006, construction delays of about one year were announced, following quality control problems across the construction. In part the delays were due to the lack of oversight of subcontractors inexperienced in nuclear construction. > At the end of June 2007, it was reported that Säteilyturvakeskus (STUK), the Finnish Radiation and Nuclear Safety Authority, had found a number of safety-related design and manufacturing 'deficiencies'. > Initial cost estimates were about €3.7 billion, but the project has since seen several severe cost increments and delays, with latest cost estimates (from 2012) of more than €8 billion. Flamanville (France) > In April 2008, the French nuclear safety agency (Autorité de sûreté nucléaire, ASN) reported that a quarter of the welds inspected in the secondary containment steel liner are not in accordance with norms, and that cracks have been found in the concrete base. > In April 2015, Areva informed the French nuclear regulator ASN that anomalies had been detected in the reactor vessel steel, causing "lower than expected mechanical toughness values". Further tests are underway.[64] In July 2015 The Daily Telegraph reported that Areva had been aware of this problem since 2006.[65] In June 2015, multiple faults in cooling system safety valves were discovered by ASN.

Money saved by firing experienced engs and outsourcing the development << money lost because they can't sell planes. Why is that so hard to understand? Massive fail at risk management maybe?

cronix 7y ago

Can't sell planes, or fly the ones that exist, and pay out huge untold sums in death lawsuits.

crorella 7y ago
Yeah, that too, after the fact seems so stupidly obvious.

meowface 7y ago

For one, in part because many managers don't really understand software or software projects and think it's mostly just people * man-hours. "Through this cost-saving measure, we've tripled the amount of people working on the project and expect to reduce the amount of time it'll take to complete by two-thirds without incurring any additional costs." In their minds, it's an easy win. And they were probably regularly reassured by various internal and external entities that the code was high-quality and thoroughly tested and very safe. Obviously they're not going to outsource if they think it's likely to cause a safety hazard; they just didn't place the odds of this outcome anywhere near what the actual odds ended up being.

weaksauce 7y ago
someone needs to throw the mythical man month at these managers.
- durnygbur 7y ago
  There would be no englightment happening.
nobodyandproud 7y ago
Which means they have no business managing. It’s to tiresome to repeat because it should be obvious , but engineering is not a commodity. The end product may be, but not the design problem process nor the manufacturing process.
- xiphias2 7y ago
  For me the solution is quite easy: I don't invest in companies where I don't see technical excellence happening. The world is changing fast enough that old style management won't get investment after some time.
deleted 7y ago
[deleted]

pjc50 7y ago

The risk management is being TBTF. In a nationalist era, is the US going to buy planes from Airbus, or are they going to bail out Boeing?

broahmed 7y ago
For others wondering: TBTF = too big to fail

dongping 7y ago

The managers who saved money would have been promoted well before any accident happens.

bsder 7y ago

> Why is that so hard to understand? Massive fail at risk management maybe? Managerial bonus or promotion for saving money this quarter >> money lost more quarters out than I'll be around

gscott 7y ago
This is really it. Board of directors set goals, say reducing costs, they offer a big bonus, you meet the goal, get the big bonus. Problems become someone else's proble. You buy a big car, house, go to fancy non-profit's $500 dinners.
- gotdevops 7y ago
  This is definitely one major issue. At the top. The challenge at the top is that these outsourcing contracts are for many millions of dollars. They are often written in such a way that the parent cannot dictate how the outsourcing firm conducts business. The parent fills out a ticket and the outsourcer assigns the ticket to a resource and marks the ticket complete. The issue here is that this is not a bug related to email or Login credentials. This is an airplane with folks relying on it for safety of flight.

rossdavidh 7y ago

I have seen this kind of problem before (never in a case where lives were on the line), and I called it "pathological optimism". If you are in a corporate culture of the sort that lionizes "can-do" thinking, then saying "that won't work" is going to look defeatist. I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. But more likely, it's just that pathological optimism works, really well, right up until that moment when it doesn't. Taking risks in order to lower costs makes the bottom line better, which seems to call for more of the same, right up until the moment when the risk explodes in your face. It is not unlike the attitude towards software security, in many industries. People harping about security risks seems like just all cost and no benefit, right up until the point where you have a big problem.

CydeWeys 7y ago
> I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. If so, they're learning the wrong lessons. SV unicorns become valuable because of explosive growth, not because they underpay their engineers. Just the opposite; a lot of engineers at SV startups are paid quite handsomely (especially the early ones at successful unicorns). SV startups don't win on cost-cutting.
- gotdevops 7y ago
  I think you are right on with this. Most startups also leveraging an organizational culture that goes against silos or fiefdoms. In the startup culture I’ve been a part of fast feedback and early identification of flaws in the process is a cornerstone of the development process. Troubleshooting issues in production is the worse case scenario. In this case production is literally a flying airplane.

ganeshkrishnan 7y ago

The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little. The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed. The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall. For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunities-overseas/vacancies-in-india.html https://www.airbus.com/careers/search-and-apply/opportunitie... Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.

pxue 7y ago
Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?
- ganeshkrishnan 7y ago
  https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidents https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs
  - gotdevops 7y ago
    It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22
gotdevops 7y ago
You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done. https://qualitysafety.bmj.com/content/13/suppl_2/ii22 https://qualitysafety.bmj.com/content/13/suppl_2/ii22

RachelF 7y ago

Most managers only think short-term. In the short term it saved money. In the long term is cost Boeing their reputation. But in the long term the MBA manager has been likely been promoted and is working elsewhere, killing another company, while his or her CV boasts of "reducing inefficiencies".

ensiferum 7y ago
As long as they can get a RSU package and get a promotion it's s win.

tarsinge 7y ago

Unfortunately it seems other critical industries are affected, reminded me of the EPR nuclear reactor fiasco https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluoto_3_(Finland) https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluot... Olkiluoto (Finland) > In May 2006, construction delays of about one year were announced, following quality control problems across the construction. In part the delays were due to the lack of oversight of subcontractors inexperienced in nuclear construction. > At the end of June 2007, it was reported that Säteilyturvakeskus (STUK), the Finnish Radiation and Nuclear Safety Authority, had found a number of safety-related design and manufacturing 'deficiencies'. > Initial cost estimates were about €3.7 billion, but the project has since seen several severe cost increments and delays, with latest cost estimates (from 2012) of more than €8 billion. Flamanville (France) > In April 2008, the French nuclear safety agency (Autorité de sûreté nucléaire, ASN) reported that a quarter of the welds inspected in the secondary containment steel liner are not in accordance with norms, and that cracks have been found in the concrete base. > In April 2015, Areva informed the French nuclear regulator ASN that anomalies had been detected in the reactor vessel steel, causing "lower than expected mechanical toughness values". Further tests are underway.[64] In July 2015 The Daily Telegraph reported that Areva had been aware of this problem since 2006.[65] In June 2015, multiple faults in cooling system safety valves were discovered by ASN.