3 ms·

I think this is a fair take when you're talking about failures of nascent designs because of unknown failure modes. But it's frustrating when there are failures

by bumby 5y ago

I think this is a fair take when you're talking about failures of nascent designs because of unknown failure modes. But it's frustrating when there are failures from known failure modes that just had process escapes. A couple examples: - CST-100 was sent into orbit with software mapped to valves incorrectly. There is no credible reason this shouldn't have been tested thoroughly on the ground first. - Falcon 9 had a failure due to a tank support strut that had strength characteristics a fraction of its design spec. Good supplier quality control should have vetted the material prior to its use. (SpaceX has since added material tests). Finding unknown causes of failures help us learn more about the science of the industry. But not catching known failure modes due to lack of process control is a different animal.

hn_throwaway_99 5y ago
> But not catching known failure modes due to lack of process control is a different animal. I agree, but for any complicated product like a rocket launch there are probably many many tens of thousands/hundreds of thousands/millions of things like this where one small uncaught error can cause mission failure. At some point it's just statistics, you'd expect to see some catastrophic failures across some percentage of products early in their lifecycle, regardless of how good your process control is.
- bumby 5y ago
  Yes, there is reliability analysis involved which invokes probability. However, that's not the case with the items I mentioned above. It would be one thing if SpaceX tested the material coupon and found it within specs, but by sheer probability, the strut had a portion of its material structure that was anomalous. That's what you're talking about. What I'm talking about is there are specific process steps that would have caught those mistakes above, and those processes were just not performed. Neither of the cases mentioned above are the result of "We did everything right, but the probability gods were just not on our side."
  - hn_throwaway_99 5y ago
    > It would be one thing if SpaceX tested the material coupon and found it within specs, but by sheer probability, the strut had a portion of its material structure that was anomalous. That's what you're talking about. Actually, that's not what I'm talking about. What I'm talking about is lessons from QA processes and the chance that any particular bug escapes from one QA process to the next. The idea being, suppose you have: * one million different "components" in a system * each of those components have, say, 10 possible failure modes * There are 100 of these new complex products a year. Obviously the numbers above are all made up, but the idea would be in that example you'd have a billion (1,000,000 X 10 X 100) possible failure modes. Any QA filter, might be expected to catch, say, 95% of remaining bugs. So you run as many filters as you can, but at some point you're still left with the small probability for new bugs, and they only way to remediate those bugs is to figure out what went wrong after the fact.
    - bumby 5y ago
      Not every bug creates a failure, so there's not necessarily a need to be completely bug-free. The point is to focus efforts on the bugs that can cause failure and design mitigations for them. If your system has a billion potential software failure modes, my hope is that a project manager will say the software is too complex and needs to be redesigned. I think this is where some in the software crowd get it wrong in their understanding of how software reliability in safety-critical applications is performed. There are a lot of project managers who deliberately avoid using software as a mitigation because of the complexity issue (see the 737Max use of software as a mitigation for a good example of where this practice can go wrong). So take the software example from above on CST-100. The valve mapping error should have been a known failure mode ("We command valve A and it doesn't respond." or "We command valve X and valve A responds"). Those failures are very straightforward to test for with simulations on the ground, yet somehow those errors made it into a flight configuration. Arguably, the software timer issue on that flight falls into a similar category, but the mitigations there are a bit more complex.
  - buran77 5y ago
    > there are specific process steps that would have caught those mistakes above Overestimating the risks can bury a project just as quickly as underestimating them. If you test for every possible failure mode you may never get off the ground. There's always a test or check that can catch something before the risk is realized. Every nut and bolt can be X-rayed, every line of code and possible combination can be tested, it's just a balancing act. Sometimes the risk doesn't justify all the tests, sometimes a test is accidentally omitted, or poorly defined because someone made a mistake or misunderstood the explanations behind it, or someone misread the result, or the risk was considered avoided because the code was tried and tested tech on Ariane 4. Rockets are so complex that there are millions of situations where a mistake can be made and later amplified.
deleted 5y ago
[deleted]

by bumby 5y ago

hn_throwaway_99 5y ago

> But not catching known failure modes due to lack of process control is a different animal. I agree, but for any complicated product like a rocket launch there are probably many many tens of thousands/hundreds of thousands/millions of things like this where one small uncaught error can cause mission failure. At some point it's just statistics, you'd expect to see some catastrophic failures across some percentage of products early in their lifecycle, regardless of how good your process control is.

bumby 5y ago
Yes, there is reliability analysis involved which invokes probability. However, that's not the case with the items I mentioned above. It would be one thing if SpaceX tested the material coupon and found it within specs, but by sheer probability, the strut had a portion of its material structure that was anomalous. That's what you're talking about. What I'm talking about is there are specific process steps that would have caught those mistakes above, and those processes were just not performed. Neither of the cases mentioned above are the result of "We did everything right, but the probability gods were just not on our side."
- hn_throwaway_99 5y ago
  > It would be one thing if SpaceX tested the material coupon and found it within specs, but by sheer probability, the strut had a portion of its material structure that was anomalous. That's what you're talking about. Actually, that's not what I'm talking about. What I'm talking about is lessons from QA processes and the chance that any particular bug escapes from one QA process to the next. The idea being, suppose you have: * one million different "components" in a system * each of those components have, say, 10 possible failure modes * There are 100 of these new complex products a year. Obviously the numbers above are all made up, but the idea would be in that example you'd have a billion (1,000,000 X 10 X 100) possible failure modes. Any QA filter, might be expected to catch, say, 95% of remaining bugs. So you run as many filters as you can, but at some point you're still left with the small probability for new bugs, and they only way to remediate those bugs is to figure out what went wrong after the fact.
  - bumby 5y ago
    Not every bug creates a failure, so there's not necessarily a need to be completely bug-free. The point is to focus efforts on the bugs that can cause failure and design mitigations for them. If your system has a billion potential software failure modes, my hope is that a project manager will say the software is too complex and needs to be redesigned. I think this is where some in the software crowd get it wrong in their understanding of how software reliability in safety-critical applications is performed. There are a lot of project managers who deliberately avoid using software as a mitigation because of the complexity issue (see the 737Max use of software as a mitigation for a good example of where this practice can go wrong). So take the software example from above on CST-100. The valve mapping error should have been a known failure mode ("We command valve A and it doesn't respond." or "We command valve X and valve A responds"). Those failures are very straightforward to test for with simulations on the ground, yet somehow those errors made it into a flight configuration. Arguably, the software timer issue on that flight falls into a similar category, but the mitigations there are a bit more complex.
- buran77 5y ago
  > there are specific process steps that would have caught those mistakes above Overestimating the risks can bury a project just as quickly as underestimating them. If you test for every possible failure mode you may never get off the ground. There's always a test or check that can catch something before the risk is realized. Every nut and bolt can be X-rayed, every line of code and possible combination can be tested, it's just a balancing act. Sometimes the risk doesn't justify all the tests, sometimes a test is accidentally omitted, or poorly defined because someone made a mistake or misunderstood the explanations behind it, or someone misread the result, or the risk was considered avoided because the code was tried and tested tech on Ariane 4. Rockets are so complex that there are millions of situations where a mistake can be made and later amplified.

deleted 5y ago

[deleted]