5 ms·

This is an article written by someone who’s never actually worked on a safety critical system. Such systems have an exhaustive threat model with likelihood esti

This is an article written by someone who’s never actually worked on a safety critical system. Such systems have an exhaustive threat model with likelihood estimates developed and verified well before a single line of code is written.

tptacek 7y ago
As someone who has (perhaps like the author) done security assessment work for safety critical systems, I sure would like to hear more about these exhaustive threat models with likelihood estimates, because they sure didn't seem to have much to do with preventing me from popping a shell.
- deleted 7y ago
  [deleted]
- tedunangst 7y ago
  You popped a shell, therefore it wasn't a true safety critical system.
  - NovemberWhiskey 7y ago
    Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the parent post. Safety-critical systems (at the time: DO-178B design assurance level A in airborne systems) were about custom processor boards based on mature technologies (PowerPC was popular; caches often disabled for maximally predictable peformance), quadruplex redundancy, software with 100% MC/DC test coverage, verification of traceability between source code and object code, zero-runtimes, no dynamic memory allocation, extensive time budgeting analysis for control loops, worst-case stack utilization analyses, hardware watchdogs, extensive built-in-test and multiple independence objectives (verification team had to be independent of implementation team etc) as well as formal methods like data and information flow analysis. These systems are virtually nothing like conventional "information technology" systems.
    - tptacek 7y ago
      This is approximately what people used to say about EAL4+ certified products, until everyone realized you can just look at the list of EAL4+ products and the CVE database and compare. Look, I don't doubt that there are avionics systems that are secure by design (largely because their inputs and their functionality are extremely constrained). But that's a question-begging definition of "safety critical", because --- my expertise isn't in avionics but I've done a fair bit of utilities and medical work, for example --- those "safety critical" systems tend to be embedded in larger distributed compute systems that are riddled with vulnerabilities, with an ultimate systemic security result of "some doofus on the Internet can blow up a transformer". So if the argument is "you can in fact built safety-critical systems by massively expanding the cost in money and time to built drastically simplified and less useful components", sure, you win. But if your argument is "the safety critical industries all know how to field software-based infrastructure that is safe from attack", then, based simply on personal experience, I'm going to call "shenanigans", because no, they cannot.
      - pnako 7y ago
        Safety-critical systems and secure systems are completely orthogonal. Although the article makes a good point, I don't think it was very sensible to use the expression "safety-critical software", for this reason. In a safety-critical system (which might include software), you are trying to protect yourself against mother nature. Which is tricky but doable, because we can predict it. Generally, the security assessment is minimal because the system is assumed to be "behind the firewall": in a secure cabinet in a plane, in an access-controlled building or factory, etc. Of course, all hell breaks loose the day someone decides to connect that system to the Internet, where thousands of smart humans with bad intentions try to break it. I think the author would have helped his point better if he simply talked about secure software, not safety-critical software.
        killjoywashere 7y ago
        > Safety-critical systems and secure systems are completely orthogonal This strikes me as the most true statement in this conversation. I feel like I need this as a bumper sticker. I think a significant issue in this space is figuring out how to get the security people to put as much skin in the game as the safety-critical people have. The security people often get to continue paying their mortgage as long as they say "no". And as a physician leading a development team, I really struggle with this. If the budgets need to go up to solve the compliance problems, that's fine, but the compliance people seem to always have one more tissue-thin layer of requirements to add. It's like trying to drill your way out of a growing onion.
- pkaye 7y ago
  What kind of safety critical system was it?
- Tomte 7y ago
  Security is still dismal in typical industrial applications that require safety. Typically, safety field buses handle the threat of sabotage with „the field bus is inside the factory, so nobody has access“. Of course, people hook the production net up to some other network, so that managers can view process information from their office. You can easily impersonate network participants, just by sniffing the unencrypted telegrams and generating a new telegram with the correct sender id, consecutive number and CRC. Suddenly your emergency stop doesn’t work anymore. Security gets more important, though. Customers demand certifications and pen tests. So manufacturers „harden“ their systems. This often means not much more than updating used open source software to reasonably new versions and getting rid of MD5 or SHA1 password hashing. It‘s a total shitshow, and a very juicy target for pentesters and other security professionals. I‘ve often thought about entering that field myself, but at least with my current employer, I have zero opportunity to do so.
- djcapelis 7y ago
  I agree with you but I think it’s useful to walk through the traditional model to understand why so many people think this, and why it totally fails in software in practice. In traditional safety engineering a wide ranging fault tree analysis (FTA) or other analysis models detail every potential fault in a system and provides estimated percentages of what types of faults occur over a total system lifespan and the system isn’t something that is considered safe until sufficient mitigations are put in place to allow the whole system to drop below the required threshold for a certain criticality of incident over its lifespan. This work is legitimately awesome and has allowed an amazing amount of reliability engineering that keeps us safe in so many weird ways each day as we do simple things like walk through an automatic door or drive down a freeway or enter any large structure. Unfortunately none of these models assume the presence of an attack or malicious entity. So they painstaking provide a model for making sure your bridge doesn’t fall down on its own but are generally pretty useless for ensuring someone can’t blow it up. This is considered fine in a lot of safety critical engineering work which is closer to reliability engineering than security engineering. Malicious behavior in the “digital realm” (I hate this phrase but here we are) has such a low barrier to entry that every single one of these models fall apart there and a threat model which doesn’t assume potentially malicious actors and behavior is something we’ve come to realize is basically negligent. The pure explosion in complexity and states possible in the FTA when analyzing software security with actual adversaries makes them... not particularly useful. In addition the sharply binary outcomes (if you change the bit you just outright win) in so much of software design make it really hard to apply anything resembling traditional engineering controls or mitigations. Where with a bridge you can beef up some concrete and the numbers which affect the outcome change, in software you can’t really beef up security in a real sense by just removing some but not all of the software weaknesses. We could decide that none of these things work well together, but I actually think the futility of trying to construct an FTA model for software leads to some very supportable and radical conclusions about how to actually build secure systems. Many of which involve just distrusting most software pieces entirely (or carefully and selectively trusting which tasks) and looking at high level FTAs to understand how to compose larger systems that maintain some key properties or state without ever trusting say, the entire codebase of an operating system kernel.

tptacek 7y ago

As someone who has (perhaps like the author) done security assessment work for safety critical systems, I sure would like to hear more about these exhaustive threat models with likelihood estimates, because they sure didn't seem to have much to do with preventing me from popping a shell.

deleted 7y ago
[deleted]
tedunangst 7y ago
You popped a shell, therefore it wasn't a true safety critical system.
- NovemberWhiskey 7y ago
  Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the parent post. Safety-critical systems (at the time: DO-178B design assurance level A in airborne systems) were about custom processor boards based on mature technologies (PowerPC was popular; caches often disabled for maximally predictable peformance), quadruplex redundancy, software with 100% MC/DC test coverage, verification of traceability between source code and object code, zero-runtimes, no dynamic memory allocation, extensive time budgeting analysis for control loops, worst-case stack utilization analyses, hardware watchdogs, extensive built-in-test and multiple independence objectives (verification team had to be independent of implementation team etc) as well as formal methods like data and information flow analysis. These systems are virtually nothing like conventional "information technology" systems.
  - tptacek 7y ago
    This is approximately what people used to say about EAL4+ certified products, until everyone realized you can just look at the list of EAL4+ products and the CVE database and compare. Look, I don't doubt that there are avionics systems that are secure by design (largely because their inputs and their functionality are extremely constrained). But that's a question-begging definition of "safety critical", because --- my expertise isn't in avionics but I've done a fair bit of utilities and medical work, for example --- those "safety critical" systems tend to be embedded in larger distributed compute systems that are riddled with vulnerabilities, with an ultimate systemic security result of "some doofus on the Internet can blow up a transformer". So if the argument is "you can in fact built safety-critical systems by massively expanding the cost in money and time to built drastically simplified and less useful components", sure, you win. But if your argument is "the safety critical industries all know how to field software-based infrastructure that is safe from attack", then, based simply on personal experience, I'm going to call "shenanigans", because no, they cannot.
    - pnako 7y ago
      Safety-critical systems and secure systems are completely orthogonal. Although the article makes a good point, I don't think it was very sensible to use the expression "safety-critical software", for this reason. In a safety-critical system (which might include software), you are trying to protect yourself against mother nature. Which is tricky but doable, because we can predict it. Generally, the security assessment is minimal because the system is assumed to be "behind the firewall": in a secure cabinet in a plane, in an access-controlled building or factory, etc. Of course, all hell breaks loose the day someone decides to connect that system to the Internet, where thousands of smart humans with bad intentions try to break it. I think the author would have helped his point better if he simply talked about secure software, not safety-critical software.
      - killjoywashere 7y ago
        > Safety-critical systems and secure systems are completely orthogonal This strikes me as the most true statement in this conversation. I feel like I need this as a bumper sticker. I think a significant issue in this space is figuring out how to get the security people to put as much skin in the game as the safety-critical people have. The security people often get to continue paying their mortgage as long as they say "no". And as a physician leading a development team, I really struggle with this. If the budgets need to go up to solve the compliance problems, that's fine, but the compliance people seem to always have one more tissue-thin layer of requirements to add. It's like trying to drill your way out of a growing onion.
pkaye 7y ago
What kind of safety critical system was it?
Tomte 7y ago
Security is still dismal in typical industrial applications that require safety. Typically, safety field buses handle the threat of sabotage with „the field bus is inside the factory, so nobody has access“. Of course, people hook the production net up to some other network, so that managers can view process information from their office. You can easily impersonate network participants, just by sniffing the unencrypted telegrams and generating a new telegram with the correct sender id, consecutive number and CRC. Suddenly your emergency stop doesn’t work anymore. Security gets more important, though. Customers demand certifications and pen tests. So manufacturers „harden“ their systems. This often means not much more than updating used open source software to reasonably new versions and getting rid of MD5 or SHA1 password hashing. It‘s a total shitshow, and a very juicy target for pentesters and other security professionals. I‘ve often thought about entering that field myself, but at least with my current employer, I have zero opportunity to do so.
djcapelis 7y ago
I agree with you but I think it’s useful to walk through the traditional model to understand why so many people think this, and why it totally fails in software in practice. In traditional safety engineering a wide ranging fault tree analysis (FTA) or other analysis models detail every potential fault in a system and provides estimated percentages of what types of faults occur over a total system lifespan and the system isn’t something that is considered safe until sufficient mitigations are put in place to allow the whole system to drop below the required threshold for a certain criticality of incident over its lifespan. This work is legitimately awesome and has allowed an amazing amount of reliability engineering that keeps us safe in so many weird ways each day as we do simple things like walk through an automatic door or drive down a freeway or enter any large structure. Unfortunately none of these models assume the presence of an attack or malicious entity. So they painstaking provide a model for making sure your bridge doesn’t fall down on its own but are generally pretty useless for ensuring someone can’t blow it up. This is considered fine in a lot of safety critical engineering work which is closer to reliability engineering than security engineering. Malicious behavior in the “digital realm” (I hate this phrase but here we are) has such a low barrier to entry that every single one of these models fall apart there and a threat model which doesn’t assume potentially malicious actors and behavior is something we’ve come to realize is basically negligent. The pure explosion in complexity and states possible in the FTA when analyzing software security with actual adversaries makes them... not particularly useful. In addition the sharply binary outcomes (if you change the bit you just outright win) in so much of software design make it really hard to apply anything resembling traditional engineering controls or mitigations. Where with a bridge you can beef up some concrete and the numbers which affect the outcome change, in software you can’t really beef up security in a real sense by just removing some but not all of the software weaknesses. We could decide that none of these things work well together, but I actually think the futility of trying to construct an FTA model for software leads to some very supportable and radical conclusions about how to actually build secure systems. Many of which involve just distrusting most software pieces entirely (or carefully and selectively trusting which tasks) and looking at high level FTAs to understand how to compose larger systems that maintain some key properties or state without ever trusting say, the entire codebase of an operating system kernel.