6 ms·

> Almost all problems occurred at interfaces between companies (prime vs. sub, customer vs. prime) or between different groups within the same company, where on

> Almost all problems occurred at interfaces between companies (prime vs. sub, customer vs. prime) or between different groups within the same company, where one group misunderstood what another group was doing, or at actual mechanical and electrical interfaces between components designed and built by different groups. This is obviously a well-known phenomenon in software engineering and I don't think anyone here is going to be be particularly surprised that it occurs in the aerospace setting. What is a little more surprising, to me at least, is that the systems people over there don't have procedures in place to minimise risks stemming from lack of communication. It isn't realistic for any sub-team to be fully familiar with the overall system but surely, for instance, if a team is working on component X which interfaces with components Y and Z, then it should be standard practice for the X team to spend at least some time with the Y and Z teams during development?

whoitwas 2y ago
Yeah. Where are the integration tests? As a total outsider reading headlines, I would assume there's minimal testing framework and sparse QA.
sgt101 2y ago
PRINCE-2 and other methodologies used in these kinds of programs make ample provision for doing this - but like all methodologies the benefits only come from proper application. If the program manager is subjected to political pressure from different stakeholders then the processes and approaches that should catch division and misapprehensions may simply not run.
ClumsyPilot 2y ago
Maybe there should not be 300 subcontractors involved in delivery and contracts should stipulated that work cannot be outsourced? The outsourcing of everything is part of the reason no one is ever held accountable.
- chgs 2y ago
  The point of outsourcing is because people don’t want to be held accountable. Stopping outsourcing doesn’t change the fundamental problem of accountability, people just find different ways to avoid it.
  - stackghost 2y ago
    >The point of outsourcing is because people don’t want to be held accountable. Having worked on the public sector (Air Force), there's enormous pressure on groups like NASA to outsource because voters perceive government work as wasteful and expensive, and contracted work as efficient because free market.
    - soperj 2y ago
      > there's enormous pressure on groups like NASA to outsource because voters perceive government work as wasteful and expensive, and contracted work as efficient because free market. And, those contracts end up being the most wasteful and expensive of all.
limit499karma 2y ago
The issue is design and development methodology. > "Spend some time with the other team" How about design documents? Is that truly a lost art among the latter-day geeks?
- hanniabu 2y ago
  Everybody hates reading documentation
  - seb1204 2y ago
    I read it but there is less and less available. Presentation and video recordings are more common but useless for the self study and search of information. Writing good documentation and instructions is hard. I try it a lot.
- gnfargbl 2y ago
  I'm sure that these groups are producing specifications, and I'm sure those specifications are being followed to the letter (and perhaps even being validated as such). The problem is that the spec only ever contains about 80% of reality, with the rest being lost either to implicit assumptions made by the writer, or to requirements that the implementer couldn't possibly hit and can't know (unilaterally) how to trade into something more realistic. This is why you have to get the humans to talk to the other humans. If that communication happens via a collaborative design document then yes, that's a process, and it's one that can work.
  - SoftTalker 2y ago
    It's also why we could not recreate a Saturn-V today. We have the specs, but we don't have the knowledge and skills of the people who actually built them.
    - buildsjets 2y ago
      Even if you have the specs, you do not know if there was some important variable that was not referenced in the specs, and then you need a billion dollar research project to figure out what was missing from the original spec. Reference: FOGBANK. https://www.twz.com/32867/fogbank-is-mysterious-material-used-in-nukes-thats-so-secret-nobody-can-say-what-it-is https://www.twz.com/32867/fogbank-is-mysterious-material-use...
amelius 2y ago
They should double the teams. For every interface, they should add a team who's sole job it is to design and test that interface.
- datadrivenangel 2y ago
  I had a relative who spent years as a systems integration engineer at Boeing. This was his job basically.
- galangalalgol 2y ago
  As labor is a driving cost, wouldn't that almost double the price? They are already uncompetitive in price with spaceX right? I'm not suggesting that profits be valued over lives, but they are clearly doing something wrong beyond having too few employees.
  - pixl97 2y ago
    Stock market goes up when we get rid of more employees, that's all that matters... Until the moment it doesn't.
    - GolfPopper 2y ago
      But by that time, the management whose quarterly or yearly bonuses drove the decision have moved on to bring their skills at increasing stock value to some other company (or retired).
    - mensetmanusman 2y ago
      Maybe you are right. Being a public company is a net negative for space.
- fn-mote 2y ago
  I am pretty sure you should have marked this comment /s. The Mythical Man Month surely applies.
  - 2y ago
nordsieck 2y ago
What's amazing to me is that it doesn't seem like Boeing did tests with a fully integrated capsule until after the CFT test was in progress. They did test firings of individual thrusters, and even did some with multiple thrusters, but with many of the systems in the doghouse missing and the insulation taken off. Having read a good amount about their methods, it really seems like Boeing has relied heavily on component level tests and analysis rather than integrated tests. And it has bitten them many times so far.
- bumby 2y ago
  >with many of the systems in the doghouse missing and the insulation taken off. I'm curious where you're getting this? I've read speculation, but I've never seen any authoritative source claim the test hardware configuration was different than the flight configuration. The better sources I've seen tend to indicate it was an inadequate thruster profile in the tests, rather than a configuration issue.
acomjean 2y ago
Back when I worked on this hardware/software integration, we often didn't have the hardware to test. So we coded to the specs. I spent a lot of time reading those and trying to figure out what they meant. It was a little challenging but usually all the information was there. It worked (mostly) and we tested alot. Some stuff was strange, I still remember seeing angles in BAMs (Binary Angle Mesurements) https://en.wikipedia.org/wiki/Binary_angular_measurement https://en.wikipedia.org/wiki/Binary_angular_measurement
- azernik 2y ago
  This jibes with the way people compare SpaceX and "old space" development in industry sources I follow: SpaceX works "hardware rich", building lots of prototypes early in the development process. When Boeing and ULA launch their first "production" launch, the previous test articles generally haven't been anywhere near complete.
micromacrofoot 2y ago
I wonder if it's possible to avoid sub-teams of a project at this scale, could everyone working on it have a general understanding of the entire system? even with imperfect understanding, individual contributors would cover the gaps for each other. Are there full-stack engineers? or are the individual domains too complex compared to coding?
- _moof 2y ago
  This isn't really possible on a project like this. There are just too many specialties, and you need folks who have deep expertise in each one. Just off the top of my head there's structures, mechanisms, fluids, propulsion, avionics, dynamics, software, integration, systems, instrumentation, test, operations, human factors, and manufacturing, and each one of those has sub-specialties. In avionics for example you've got RF and power (among others); in software there's embedded, flight, ground, and interfaces (again, among others). There's a chief engineer whose job it is to oversee the project but they will be relying on the expertise of the individual teams, and each team has to work closely with and lean on their partner teams. Sometimes you'll have people who are cross-trained - I have experience in avionics, software, and ops - but that's not typical, and it doesn't take much to feel spread thin (I certainly do).
BiteCode_dev 2y ago
It's like people expect NASA to be infallible.
- pdonis 2y ago
  No, people look at what NASA was able to accomplish during the 1960s and compare it to now, and wonder how the level of competence can be so drastically lower now vs. then. NASA was not infallible during the 1960s, but the level of engineering competence was much higher.
  - dml2135 2y ago
    Well my first thought to answer that question is -- how do the funding levels compare between now and then?
    - pdonis 2y ago
      The combined Mercury, Gemini, and Apollo programs cost around $30 billion in then year dollars, which equates to about $300 billion in today's dollars. That's an average of about $25 billion a year over 12 years. That's about the same as what is being spent per year on NASA now.
      - bumby 2y ago
        But it’s now spread across many, many more missions. People sometimes forget that human exploration is just one of multiple directorates within NASA.
        pdonis 2y ago
        NASA was doing plenty of unmanned missions during the Mercury-Gemini-Apollo years: the Explorer, Pioneer, Echo, Ranger, Telstar, Mariner, Lunar Explorer, and Surveyor programs all had multiple missions in that time period. So no, I don't buy the argument that NASA has to spread its budget over many more missions now as compared to then.
throwaway984393 2y ago
[dead]

whoitwas 2y ago

Yeah. Where are the integration tests? As a total outsider reading headlines, I would assume there's minimal testing framework and sparse QA.

sgt101 2y ago

PRINCE-2 and other methodologies used in these kinds of programs make ample provision for doing this - but like all methodologies the benefits only come from proper application. If the program manager is subjected to political pressure from different stakeholders then the processes and approaches that should catch division and misapprehensions may simply not run.

ClumsyPilot 2y ago

Maybe there should not be 300 subcontractors involved in delivery and contracts should stipulated that work cannot be outsourced? The outsourcing of everything is part of the reason no one is ever held accountable.

chgs 2y ago
The point of outsourcing is because people don’t want to be held accountable. Stopping outsourcing doesn’t change the fundamental problem of accountability, people just find different ways to avoid it.
- stackghost 2y ago
  >The point of outsourcing is because people don’t want to be held accountable. Having worked on the public sector (Air Force), there's enormous pressure on groups like NASA to outsource because voters perceive government work as wasteful and expensive, and contracted work as efficient because free market.
  - soperj 2y ago
    > there's enormous pressure on groups like NASA to outsource because voters perceive government work as wasteful and expensive, and contracted work as efficient because free market. And, those contracts end up being the most wasteful and expensive of all.

limit499karma 2y ago

The issue is design and development methodology. > "Spend some time with the other team" How about design documents? Is that truly a lost art among the latter-day geeks?

hanniabu 2y ago
Everybody hates reading documentation
- seb1204 2y ago
  I read it but there is less and less available. Presentation and video recordings are more common but useless for the self study and search of information. Writing good documentation and instructions is hard. I try it a lot.
gnfargbl 2y ago
I'm sure that these groups are producing specifications, and I'm sure those specifications are being followed to the letter (and perhaps even being validated as such). The problem is that the spec only ever contains about 80% of reality, with the rest being lost either to implicit assumptions made by the writer, or to requirements that the implementer couldn't possibly hit and can't know (unilaterally) how to trade into something more realistic. This is why you have to get the humans to talk to the other humans. If that communication happens via a collaborative design document then yes, that's a process, and it's one that can work.
- SoftTalker 2y ago
  It's also why we could not recreate a Saturn-V today. We have the specs, but we don't have the knowledge and skills of the people who actually built them.
  - buildsjets 2y ago
    Even if you have the specs, you do not know if there was some important variable that was not referenced in the specs, and then you need a billion dollar research project to figure out what was missing from the original spec. Reference: FOGBANK. https://www.twz.com/32867/fogbank-is-mysterious-material-used-in-nukes-thats-so-secret-nobody-can-say-what-it-is https://www.twz.com/32867/fogbank-is-mysterious-material-use...

amelius 2y ago

They should double the teams. For every interface, they should add a team who's sole job it is to design and test that interface.

datadrivenangel 2y ago
I had a relative who spent years as a systems integration engineer at Boeing. This was his job basically.
galangalalgol 2y ago
As labor is a driving cost, wouldn't that almost double the price? They are already uncompetitive in price with spaceX right? I'm not suggesting that profits be valued over lives, but they are clearly doing something wrong beyond having too few employees.
- pixl97 2y ago
  Stock market goes up when we get rid of more employees, that's all that matters... Until the moment it doesn't.
  - GolfPopper 2y ago
    But by that time, the management whose quarterly or yearly bonuses drove the decision have moved on to bring their skills at increasing stock value to some other company (or retired).
  - mensetmanusman 2y ago
    Maybe you are right. Being a public company is a net negative for space.
fn-mote 2y ago
I am pretty sure you should have marked this comment /s. The Mythical Man Month surely applies.
- 2y ago

nordsieck 2y ago

What's amazing to me is that it doesn't seem like Boeing did tests with a fully integrated capsule until after the CFT test was in progress. They did test firings of individual thrusters, and even did some with multiple thrusters, but with many of the systems in the doghouse missing and the insulation taken off. Having read a good amount about their methods, it really seems like Boeing has relied heavily on component level tests and analysis rather than integrated tests. And it has bitten them many times so far.

bumby 2y ago
>with many of the systems in the doghouse missing and the insulation taken off. I'm curious where you're getting this? I've read speculation, but I've never seen any authoritative source claim the test hardware configuration was different than the flight configuration. The better sources I've seen tend to indicate it was an inadequate thruster profile in the tests, rather than a configuration issue.

acomjean 2y ago

Back when I worked on this hardware/software integration, we often didn't have the hardware to test. So we coded to the specs. I spent a lot of time reading those and trying to figure out what they meant. It was a little challenging but usually all the information was there. It worked (mostly) and we tested alot. Some stuff was strange, I still remember seeing angles in BAMs (Binary Angle Mesurements) https://en.wikipedia.org/wiki/Binary_angular_measurement https://en.wikipedia.org/wiki/Binary_angular_measurement

azernik 2y ago
This jibes with the way people compare SpaceX and "old space" development in industry sources I follow: SpaceX works "hardware rich", building lots of prototypes early in the development process. When Boeing and ULA launch their first "production" launch, the previous test articles generally haven't been anywhere near complete.

micromacrofoot 2y ago

I wonder if it's possible to avoid sub-teams of a project at this scale, could everyone working on it have a general understanding of the entire system? even with imperfect understanding, individual contributors would cover the gaps for each other. Are there full-stack engineers? or are the individual domains too complex compared to coding?

_moof 2y ago
This isn't really possible on a project like this. There are just too many specialties, and you need folks who have deep expertise in each one. Just off the top of my head there's structures, mechanisms, fluids, propulsion, avionics, dynamics, software, integration, systems, instrumentation, test, operations, human factors, and manufacturing, and each one of those has sub-specialties. In avionics for example you've got RF and power (among others); in software there's embedded, flight, ground, and interfaces (again, among others). There's a chief engineer whose job it is to oversee the project but they will be relying on the expertise of the individual teams, and each team has to work closely with and lean on their partner teams. Sometimes you'll have people who are cross-trained - I have experience in avionics, software, and ops - but that's not typical, and it doesn't take much to feel spread thin (I certainly do).

BiteCode_dev 2y ago

It's like people expect NASA to be infallible.

pdonis 2y ago
No, people look at what NASA was able to accomplish during the 1960s and compare it to now, and wonder how the level of competence can be so drastically lower now vs. then. NASA was not infallible during the 1960s, but the level of engineering competence was much higher.
- dml2135 2y ago
  Well my first thought to answer that question is -- how do the funding levels compare between now and then?
  - pdonis 2y ago
    The combined Mercury, Gemini, and Apollo programs cost around $30 billion in then year dollars, which equates to about $300 billion in today's dollars. That's an average of about $25 billion a year over 12 years. That's about the same as what is being spent per year on NASA now.
    - bumby 2y ago
      But it’s now spread across many, many more missions. People sometimes forget that human exploration is just one of multiple directorates within NASA.
      - pdonis 2y ago
        NASA was doing plenty of unmanned missions during the Mercury-Gemini-Apollo years: the Explorer, Pioneer, Echo, Ranger, Telstar, Mariner, Lunar Explorer, and Surveyor programs all had multiple missions in that time period. So no, I don't buy the argument that NASA has to spread its budget over many more missions now as compared to then.

throwaway984393 2y ago

[dead]