4 ms·

I'm sorry but I think you’re mixing things up in the traditional sense of risk management. The probability is covered separately in risk management. A brake fai

by bumby 2y ago

I'm sorry but I think you’re mixing things up in the traditional sense of risk management. The probability is covered separately in risk management. A brake failure doesn't guarantee a particular consequence, but it does bound it and general practice is to assign the worst consequence. To use it on the Max scenario, MCAS can fault and not cause a plane to crash, but it still should be characterized as a "catastrophic" fault because it can cause loss of life in some scenarios. The probability is determined separately. Then post-mitigation consequence/likelihood/detectability are assessed. Take a look at how FMEAs are conducted if you still disagree; it’s fairly standardized in other safety critical domains. I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold.

Retric 2y ago
FMEA/FMECA are only the first step in a system reliability study, it isn’t an analysis of mitigation strategies. Fault tree analysis (etc) concerns itself with things like how long an aircraft can fly with an engine failure. Doing that involves an assessment of various risks. > I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold. Individual failures should never result in such scenarios, but ditching in the ocean may be the best option after a major fuel leak, loss of all engine power, etc.
- bumby 2y ago
  FMEA is not a “reliability study”. Reliability studies are inputs to FMEAs. Where do you think the likelihood data comes from? If you’re doing an FMEA before a reliability study, your probabilities are just guesses. I don’t think any FMEA is going to list “ditch into the ocean” as an acceptable mitigation. Ie it will never be a way to buy risk down to an acceptable level. >it isn’t an analysis of mitigation strategies. Take a look at NASAs FMEA guidebook. It clearly lists identifying mitigations as part of the FMEA process. You’ll see similar in other organizations but possibly with different names (“control” instead of “mitigation”) https://standards.nasa.gov/sites/default/files/standards/GSFC/Baseline/0/GSFC-HDBK-8004_Approved_1.pdf https://standards.nasa.gov/sites/default/files/standards/GSF...
  - Retric 2y ago
    Semantics and recursion aside. That separation where “ditch into the ocean” is not a listed mitigation for FMEA, but is eventually considered and added to manuals is why I’m saying it’s incomplete. 100% AI systems can be safer than human in the loop systems avoiding suicide by pilot etc, but conversely that means the AI must also deal with extreme edge cases. It’s a subtle but critical nuance.
    - bumby 2y ago
      Based on your answers, I'm guessing you haven't been involved in this process. (Not saying it as a bad thing, but just as a rationale for clarifying with further discussion). An FMEA (and other safety-critical/quality products) go through an approval process. So if "ditch into the ocean" is not on the FMEA, it means they should have other mitigations/controls that bought the risk down to an acceptable level. They can't/shouldn't just push forward with a risk that exceeds their acceptable tolerance. If implemented correctly, the FMEA is complete insomuch as it ensured each hazard was brought to an acceptable risk level. And certainly, a safety officer isn't going to say the system doesn't need further controls because they put "ditch into the ocean" in the manuals. If that's the rationale, it begs the question "Why wasn't the risk mitigated in the FMEA and hazard analysis?" Usually it's because they're trying to move fast due to cost/schedule pressure, not because they managed the risk. There are edge cases, but even something like a double bird strike can be considered an acceptable risk because the probability is so low. Not impossible, but low enough. That’s what “ditch in the ocean” operations are for. I agree that software system can improve safety but we shouldn't assume so without the relevant rigor that includes formal risk mitigation. Software tends to elicit interfacing faults; the implication being as the number of interfaces increases the potential number of fault modes can increase geometrically. This means it is much harder to test/mitigate software faults, especially when implementing a black-box AI model. My hunch is that many of those trying to implement AI in safety-critical applications are not rigorously mitigating risk like more mature domains. Because, you know, move fast and break things.
      - deleted 2y ago
        [deleted]

by bumby 2y ago

Retric 2y ago

FMEA/FMECA are only the first step in a system reliability study, it isn’t an analysis of mitigation strategies. Fault tree analysis (etc) concerns itself with things like how long an aircraft can fly with an engine failure. Doing that involves an assessment of various risks. > I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold. Individual failures should never result in such scenarios, but ditching in the ocean may be the best option after a major fuel leak, loss of all engine power, etc.

bumby 2y ago
FMEA is not a “reliability study”. Reliability studies are inputs to FMEAs. Where do you think the likelihood data comes from? If you’re doing an FMEA before a reliability study, your probabilities are just guesses. I don’t think any FMEA is going to list “ditch into the ocean” as an acceptable mitigation. Ie it will never be a way to buy risk down to an acceptable level. >it isn’t an analysis of mitigation strategies. Take a look at NASAs FMEA guidebook. It clearly lists identifying mitigations as part of the FMEA process. You’ll see similar in other organizations but possibly with different names (“control” instead of “mitigation”) https://standards.nasa.gov/sites/default/files/standards/GSFC/Baseline/0/GSFC-HDBK-8004_Approved_1.pdf https://standards.nasa.gov/sites/default/files/standards/GSF...
- Retric 2y ago
  Semantics and recursion aside. That separation where “ditch into the ocean” is not a listed mitigation for FMEA, but is eventually considered and added to manuals is why I’m saying it’s incomplete. 100% AI systems can be safer than human in the loop systems avoiding suicide by pilot etc, but conversely that means the AI must also deal with extreme edge cases. It’s a subtle but critical nuance.
  - bumby 2y ago
    Based on your answers, I'm guessing you haven't been involved in this process. (Not saying it as a bad thing, but just as a rationale for clarifying with further discussion). An FMEA (and other safety-critical/quality products) go through an approval process. So if "ditch into the ocean" is not on the FMEA, it means they should have other mitigations/controls that bought the risk down to an acceptable level. They can't/shouldn't just push forward with a risk that exceeds their acceptable tolerance. If implemented correctly, the FMEA is complete insomuch as it ensured each hazard was brought to an acceptable risk level. And certainly, a safety officer isn't going to say the system doesn't need further controls because they put "ditch into the ocean" in the manuals. If that's the rationale, it begs the question "Why wasn't the risk mitigated in the FMEA and hazard analysis?" Usually it's because they're trying to move fast due to cost/schedule pressure, not because they managed the risk. There are edge cases, but even something like a double bird strike can be considered an acceptable risk because the probability is so low. Not impossible, but low enough. That’s what “ditch in the ocean” operations are for. I agree that software system can improve safety but we shouldn't assume so without the relevant rigor that includes formal risk mitigation. Software tends to elicit interfacing faults; the implication being as the number of interfaces increases the potential number of fault modes can increase geometrically. This means it is much harder to test/mitigate software faults, especially when implementing a black-box AI model. My hunch is that many of those trying to implement AI in safety-critical applications are not rigorously mitigating risk like more mature domains. Because, you know, move fast and break things.
    - deleted 2y ago
      [deleted]