4 ms·

> I can think of very little reason that valve command mappings on a human-rated rocket were not adequately tested The procedure and checks were probably repea

> I can think of very little reason that valve command mappings on a human-rated rocket were not adequately tested The procedure and checks were probably repeated multiple times during earlier preparations with no issues. God only knows how many potentially critical problems exist in every system on a code path never taken, or with a component that's never used. It's one thing to test everything for the design of the system and call it complete, and another to test everything for operations. Human error caused an operational misconfiguration on a system that was otherwise validated. Think of aviation where the entire plane design and the technology used are fully tested and known to meet all the requirements. But every time you operate that plane you have a shorter pre-flight check to confirm operational worthiness. Some things that can cause a disaster won't be covered, and some items on the the checklist will be mistakenly marked as OK, which may or may not cause a problem. I'm not saying it's normal just that statistically speaking at some point you'll miss something for one reason or another, and eventually the proper conditions will be met for a disastrous outcome. Might as ell be a valve control check.

bumby 5y ago
I understand your point and agree with you philosophically. But I disagree fundamentally that it’s always a just a matter of fact that test coverage can never be 100%. There’s more to it in many cases. And human factors in design is a huge focus in aerospace (I wouldn’t be surprised if the formal discipline originated there). Aerospace is highly process driven to avoid these issues. Those issues were caught by luck, due to the way follow on testing was conducted after the timer failure. The result indicated they were not adequately tested previously. I get that not everything can be tested. But those unknown errors that go untested should be relegated to those low probability events. Neither of the examples I gave fall in that category. Hell, NASA even requires testing for random bit flips due to radiation, there’s no excuse to let a valve command mapping error slip through the cracks.
- airstrike 5y ago
  You have the benefit of hindsight. If it is so easy to know ex ante, what other currently untested parts of rocket development do you suggest SpaceX spend more time testing today?
  - bumby 5y ago
    Your claim is reasonable for the extremely low probability events, but the fact that you brought it up here makes me think you're missing the larger point. The last few levels have been dealing with CST-100, not SpaceX so I'll try to continue that thread to be more coherent and then touch on the SpaceX example. Here's all that's really needed to capture the CST-100 issue: 1) Valve A is listed a "must" requirement. E.g., "Valve A must work when commanded". 2) As a "must" requirement, there is a test plan that covers that operation (again, another requirement). Safety-critical development is driven by formal requirements more than many other types of software. Requirement #2 can happen any number of ways. If they decide to meet that requirement via simulator, they need to ensure the simulator is of sufficient fidelity to ensure #1 is met. This fidelity would mean that the simulator valves are mapped to the correct simulator commands. To answer your question directly, I would expect every "must" and "must not" requirement to be tested; any that aren't would be suspect for potential failure. If you can get me a list of requirements (ideally, with a failure modes effects analysis or hazard analysis) and associated test plans, then I can give you a list of potential failures. So the question becomes: would you be okay with a test plan that doesn't check that a software command is received to meet a simple but critical high level requirement? Do you think a test plan that misses such a simple test is within an acceptable risk envelope? This isn't some low-probability event, but a test error that can be traced to a very simple requirement that wasn't tested. With SpaceX, supplier quality has been a major point in aerospace for decades. I've worked in organizations that have rigorous procurement processes that check for such things, including audits of manufacturing facilities for critical items. Could the supplier forge the certifications and the system fail because of it? Sure, but that's not the case here. The checks just didn't happen. Materials checks are extremely common, even on small items like bolts let alone large structural components. Likewise, I would expect any non-checked critical procurement to have a higher level of risk in producing a failure. This isn't about predicting some random, unheard of failure or ex ante stock prediction. It's about due-diligence on well-understood risks.

bumby 5y ago

I understand your point and agree with you philosophically. But I disagree fundamentally that it’s always a just a matter of fact that test coverage can never be 100%. There’s more to it in many cases. And human factors in design is a huge focus in aerospace (I wouldn’t be surprised if the formal discipline originated there). Aerospace is highly process driven to avoid these issues. Those issues were caught by luck, due to the way follow on testing was conducted after the timer failure. The result indicated they were not adequately tested previously. I get that not everything can be tested. But those unknown errors that go untested should be relegated to those low probability events. Neither of the examples I gave fall in that category. Hell, NASA even requires testing for random bit flips due to radiation, there’s no excuse to let a valve command mapping error slip through the cracks.

airstrike 5y ago
You have the benefit of hindsight. If it is so easy to know ex ante, what other currently untested parts of rocket development do you suggest SpaceX spend more time testing today?
- bumby 5y ago
  Your claim is reasonable for the extremely low probability events, but the fact that you brought it up here makes me think you're missing the larger point. The last few levels have been dealing with CST-100, not SpaceX so I'll try to continue that thread to be more coherent and then touch on the SpaceX example. Here's all that's really needed to capture the CST-100 issue: 1) Valve A is listed a "must" requirement. E.g., "Valve A must work when commanded". 2) As a "must" requirement, there is a test plan that covers that operation (again, another requirement). Safety-critical development is driven by formal requirements more than many other types of software. Requirement #2 can happen any number of ways. If they decide to meet that requirement via simulator, they need to ensure the simulator is of sufficient fidelity to ensure #1 is met. This fidelity would mean that the simulator valves are mapped to the correct simulator commands. To answer your question directly, I would expect every "must" and "must not" requirement to be tested; any that aren't would be suspect for potential failure. If you can get me a list of requirements (ideally, with a failure modes effects analysis or hazard analysis) and associated test plans, then I can give you a list of potential failures. So the question becomes: would you be okay with a test plan that doesn't check that a software command is received to meet a simple but critical high level requirement? Do you think a test plan that misses such a simple test is within an acceptable risk envelope? This isn't some low-probability event, but a test error that can be traced to a very simple requirement that wasn't tested. With SpaceX, supplier quality has been a major point in aerospace for decades. I've worked in organizations that have rigorous procurement processes that check for such things, including audits of manufacturing facilities for critical items. Could the supplier forge the certifications and the system fail because of it? Sure, but that's not the case here. The checks just didn't happen. Materials checks are extremely common, even on small items like bolts let alone large structural components. Likewise, I would expect any non-checked critical procurement to have a higher level of risk in producing a failure. This isn't about predicting some random, unheard of failure or ex ante stock prediction. It's about due-diligence on well-understood risks.