3 ms·
I agree with you but I think it’s useful to walk through the traditional model to understand why so many people think this, and why it totally fails in software
by djcapelis 7y ago
I agree with you but I think it’s useful to walk through the traditional model to understand why so many people think this, and why it totally fails in software in practice. In traditional safety engineering a wide ranging fault tree analysis (FTA) or other analysis models detail every potential fault in a system and provides estimated percentages of what types of faults occur over a total system lifespan and the system isn’t something that is considered safe until sufficient mitigations are put in place to allow the whole system to drop below the required threshold for a certain criticality of incident over its lifespan. This work is legitimately awesome and has allowed an amazing amount of reliability engineering that keeps us safe in so many weird ways each day as we do simple things like walk through an automatic door or drive down a freeway or enter any large structure.
Unfortunately none of these models assume the presence of an attack or malicious entity. So they painstaking provide a model for making sure your bridge doesn’t fall down on its own but are generally pretty useless for ensuring someone can’t blow it up. This is considered fine in a lot of safety critical engineering work which is closer to reliability engineering than security engineering.
Malicious behavior in the “digital realm” (I hate this phrase but here we are) has such a low barrier to entry that every single one of these models fall apart there and a threat model which doesn’t assume potentially malicious actors and behavior is something we’ve come to realize is basically negligent. The pure explosion in complexity and states possible in the FTA when analyzing software security with actual adversaries makes them... not particularly useful. In addition the sharply binary outcomes (if you change the bit you just outright win) in so much of software design make it really hard to apply anything resembling traditional engineering controls or mitigations. Where with a bridge you can beef up some concrete and the numbers which affect the outcome change, in software you can’t really beef up security in a real sense by just removing some but not all of the software weaknesses.
We could decide that none of these things work well together, but I actually think the futility of trying to construct an FTA model for software leads to some very supportable and radical conclusions about how to actually build secure systems. Many of which involve just distrusting most software pieces entirely (or carefully and selectively trusting which tasks) and looking at high level FTAs to understand how to compose larger systems that maintain some key properties or state without ever trusting say, the entire codebase of an operating system kernel.