3 ms·

We absolutely should expect safety critical software to mitigate risk. The idea of “welp, the software is confused so just toss it back to a human” is a cop-out

by bumby 2y ago

We absolutely should expect safety critical software to mitigate risk. The idea of “welp, the software is confused so just toss it back to a human” is a cop-out in risk mitigation. At the most generous, it constitutes an “administrative” control, which is the least preferred option, tantamount to giving directions in the users manual. The general control hierarchy is: remove the hazard, engineering control, PPE, administrative control.

Retric 2y ago
I agreed, but thing about risk mitigation is there’s tradeoffs. A self driving Taxi with a tire pressure sensor error vs total break failure are simply wildly different situations and each should get different responses. Further, designing for a possibility sleeping but valid human driver is very different than designing for a likely empty vehicle.
- bumby 2y ago
  I don’t think anyone who work in risk mitigation would conflate those scenarios. Risk = probability x consequence. What you’re pointing to is the difference in consequence. It’s already acknowledged in mature risk mitigation programs. Eg, FMEAs list fault consequence, severity, likelihood, and detectability and don’t assume all faults are the same. The tradeoff is an engineering/business decision to assign appropriate mitigations to land in an acceptable risk posture. If you aren’t mitigating it with the appropriate controls, you aren’t managing risk. My point is just passing the buck to the human is not an appropriate control in many critical scenarios.
  - Retric 2y ago
    > What you’re pointing to is the difference in consequence. No. A break failure doesn’t guarantee a specific negative outcome, it dramatically raises the probability of various negative consequences. My point is risk mitigation is about lowering risk but there may be no reasonably safe options. A car stopped on a freeway is still a high risk situation, but it beats traveling at 70 MPH without working cameras.
    - bumby 2y ago
      I'm sorry but I think you’re mixing things up in the traditional sense of risk management. The probability is covered separately in risk management. A brake failure doesn't guarantee a particular consequence, but it does bound it and general practice is to assign the worst consequence. To use it on the Max scenario, MCAS can fault and not cause a plane to crash, but it still should be characterized as a "catastrophic" fault because it can cause loss of life in some scenarios. The probability is determined separately. Then post-mitigation consequence/likelihood/detectability are assessed. Take a look at how FMEAs are conducted if you still disagree; it’s fairly standardized in other safety critical domains. I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold.
      - Retric 2y ago
        FMEA/FMECA are only the first step in a system reliability study, it isn’t an analysis of mitigation strategies. Fault tree analysis (etc) concerns itself with things like how long an aircraft can fly with an engine failure. Doing that involves an assessment of various risks. > I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold. Individual failures should never result in such scenarios, but ditching in the ocean may be the best option after a major fuel leak, loss of all engine power, etc.
        bumby 2y ago
        FMEA is not a “reliability study”. Reliability studies are inputs to FMEAs. Where do you think the likelihood data comes from? If you’re doing an FMEA before a reliability study, your probabilities are just guesses. I don’t think any FMEA is going to list “ditch into the ocean” as an acceptable mitigation. Ie it will never be a way to buy risk down to an acceptable level. >it isn’t an analysis of mitigation strategies. Take a look at NASAs FMEA guidebook. It clearly lists identifying mitigations as part of the FMEA process. You’ll see similar in other organizations but possibly with different names (“control” instead of “mitigation”) https://standards.nasa.gov/sites/default/files/standards/GSFC/Baseline/0/GSFC-HDBK-8004_Approved_1.pdf https://standards.nasa.gov/sites/default/files/standards/GSF...

by bumby 2y ago

Retric 2y ago

I agreed, but thing about risk mitigation is there’s tradeoffs. A self driving Taxi with a tire pressure sensor error vs total break failure are simply wildly different situations and each should get different responses. Further, designing for a possibility sleeping but valid human driver is very different than designing for a likely empty vehicle.

bumby 2y ago
I don’t think anyone who work in risk mitigation would conflate those scenarios. Risk = probability x consequence. What you’re pointing to is the difference in consequence. It’s already acknowledged in mature risk mitigation programs. Eg, FMEAs list fault consequence, severity, likelihood, and detectability and don’t assume all faults are the same. The tradeoff is an engineering/business decision to assign appropriate mitigations to land in an acceptable risk posture. If you aren’t mitigating it with the appropriate controls, you aren’t managing risk. My point is just passing the buck to the human is not an appropriate control in many critical scenarios.
- Retric 2y ago
  > What you’re pointing to is the difference in consequence. No. A break failure doesn’t guarantee a specific negative outcome, it dramatically raises the probability of various negative consequences. My point is risk mitigation is about lowering risk but there may be no reasonably safe options. A car stopped on a freeway is still a high risk situation, but it beats traveling at 70 MPH without working cameras.
  - bumby 2y ago
    I'm sorry but I think you’re mixing things up in the traditional sense of risk management. The probability is covered separately in risk management. A brake failure doesn't guarantee a particular consequence, but it does bound it and general practice is to assign the worst consequence. To use it on the Max scenario, MCAS can fault and not cause a plane to crash, but it still should be characterized as a "catastrophic" fault because it can cause loss of life in some scenarios. The probability is determined separately. Then post-mitigation consequence/likelihood/detectability are assessed. Take a look at how FMEAs are conducted if you still disagree; it’s fairly standardized in other safety critical domains. I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold.
    - Retric 2y ago
      FMEA/FMECA are only the first step in a system reliability study, it isn’t an analysis of mitigation strategies. Fault tree analysis (etc) concerns itself with things like how long an aircraft can fly with an engine failure. Doing that involves an assessment of various risks. > I agree there may be cases where there are no reasonably safe options. That means your engineered system (especially in a public-facing product) is not ready for production because you haven't met a reasonable risk threshold. Individual failures should never result in such scenarios, but ditching in the ocean may be the best option after a major fuel leak, loss of all engine power, etc.
      - bumby 2y ago
        FMEA is not a “reliability study”. Reliability studies are inputs to FMEAs. Where do you think the likelihood data comes from? If you’re doing an FMEA before a reliability study, your probabilities are just guesses. I don’t think any FMEA is going to list “ditch into the ocean” as an acceptable mitigation. Ie it will never be a way to buy risk down to an acceptable level. >it isn’t an analysis of mitigation strategies. Take a look at NASAs FMEA guidebook. It clearly lists identifying mitigations as part of the FMEA process. You’ll see similar in other organizations but possibly with different names (“control” instead of “mitigation”) https://standards.nasa.gov/sites/default/files/standards/GSFC/Baseline/0/GSFC-HDBK-8004_Approved_1.pdf https://standards.nasa.gov/sites/default/files/standards/GSF...