4 ms·

It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through

It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model

Ringz 7mo ago
I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.
- anonymars 7mo ago
  Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
  - jacquesm 7mo ago
    I've had that multiple times. As well as the closely related 'that can't possibly have ever worked' and sure enough it never did. Forensics in old codebases with modern tools is always fun.
    - magicalhippo 7mo ago
      > As well as the closely related 'that can't possibly have ever worked' and sure enough it never did. I had one of those, customer is adamant latest version broke some function, I check related code and it hasn't been touched for 7 years, and as written couldn't possibly work. I try and indeed, doesn't work. Yet customer persisted. Long story short, an unrelated bug in a different module caused the old, non-functioning code to do something entirely different if you had that other module open as well, and the user had disciverdd this and started relying on this emergent functionality. I had made a change to that other module in the new release and in the process returned the first module to its non-functioning state. The reason they interacted was of course some global variables. Good times...
      - jacquesm 7mo ago
        Global variables... the original sin if you ask me. Forget that apple.
      - anonymars 7mo ago
        By the way, a corollary I encountered, I think with one of the recent AWS meltdowns, is that a paradoxical consequence of designing for "reliability" is that it guarantees that when something does happen, it's going to be bad, because the reliability engineering has done a good job of masking all the smaller faults. Which means 1. anything that gets through, almost by definition, is going to be bad enough to escape the safeguards, and 2. when things do get bad enough to escape the safeguards, it will likely expose the avalanche of things that were already in a failure state but were being mitigated The takeaway, which I'm not really sure how to practically make use of, was that if a system isn't observably failing occasionally in small ways, one day it's going to instead fail in a big way I don't think that's necessarily something rigorously proven but I do think of it sometimes in the face of some mess
        jacquesm 7mo ago
        That's a fairly common pattern. As frequency of incidents goes down the severity of the average incident goes up. There has to be some underlying mechanism for this (maybe the one you describe but I'm not so sure that's the whole story).

Ringz 7mo ago

I use this model all the time. It's very helpful for explaining the multifactorial genesis of catastrophes to ordinary people.

anonymars 7mo ago
Also perhaps worth a read: https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22723#:~:text=You%E2%80%99ve%20all%20experienced,errors%20are%20exposed. https://devblogs.microsoft.com/oldnewthing/20080416-00/?p=22... "You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed."
- jacquesm 7mo ago
  I've had that multiple times. As well as the closely related 'that can't possibly have ever worked' and sure enough it never did. Forensics in old codebases with modern tools is always fun.
  - magicalhippo 7mo ago
    > As well as the closely related 'that can't possibly have ever worked' and sure enough it never did. I had one of those, customer is adamant latest version broke some function, I check related code and it hasn't been touched for 7 years, and as written couldn't possibly work. I try and indeed, doesn't work. Yet customer persisted. Long story short, an unrelated bug in a different module caused the old, non-functioning code to do something entirely different if you had that other module open as well, and the user had disciverdd this and started relying on this emergent functionality. I had made a change to that other module in the new release and in the process returned the first module to its non-functioning state. The reason they interacted was of course some global variables. Good times...
    - jacquesm 7mo ago
      Global variables... the original sin if you ask me. Forget that apple.
    - anonymars 7mo ago
      By the way, a corollary I encountered, I think with one of the recent AWS meltdowns, is that a paradoxical consequence of designing for "reliability" is that it guarantees that when something does happen, it's going to be bad, because the reliability engineering has done a good job of masking all the smaller faults. Which means 1. anything that gets through, almost by definition, is going to be bad enough to escape the safeguards, and 2. when things do get bad enough to escape the safeguards, it will likely expose the avalanche of things that were already in a failure state but were being mitigated The takeaway, which I'm not really sure how to practically make use of, was that if a system isn't observably failing occasionally in small ways, one day it's going to instead fail in a big way I don't think that's necessarily something rigorously proven but I do think of it sometimes in the face of some mess
      - jacquesm 7mo ago
        That's a fairly common pattern. As frequency of incidents goes down the severity of the average incident goes up. There has to be some underlying mechanism for this (maybe the one you describe but I'm not so sure that's the whole story).