4 ms·

Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the p

Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the parent post. Safety-critical systems (at the time: DO-178B design assurance level A in airborne systems) were about custom processor boards based on mature technologies (PowerPC was popular; caches often disabled for maximally predictable peformance), quadruplex redundancy, software with 100% MC/DC test coverage, verification of traceability between source code and object code, zero-runtimes, no dynamic memory allocation, extensive time budgeting analysis for control loops, worst-case stack utilization analyses, hardware watchdogs, extensive built-in-test and multiple independence objectives (verification team had to be independent of implementation team etc) as well as formal methods like data and information flow analysis. These systems are virtually nothing like conventional "information technology" systems.

tptacek 7y ago
This is approximately what people used to say about EAL4+ certified products, until everyone realized you can just look at the list of EAL4+ products and the CVE database and compare. Look, I don't doubt that there are avionics systems that are secure by design (largely because their inputs and their functionality are extremely constrained). But that's a question-begging definition of "safety critical", because --- my expertise isn't in avionics but I've done a fair bit of utilities and medical work, for example --- those "safety critical" systems tend to be embedded in larger distributed compute systems that are riddled with vulnerabilities, with an ultimate systemic security result of "some doofus on the Internet can blow up a transformer". So if the argument is "you can in fact built safety-critical systems by massively expanding the cost in money and time to built drastically simplified and less useful components", sure, you win. But if your argument is "the safety critical industries all know how to field software-based infrastructure that is safe from attack", then, based simply on personal experience, I'm going to call "shenanigans", because no, they cannot.
- pnako 7y ago
  Safety-critical systems and secure systems are completely orthogonal. Although the article makes a good point, I don't think it was very sensible to use the expression "safety-critical software", for this reason. In a safety-critical system (which might include software), you are trying to protect yourself against mother nature. Which is tricky but doable, because we can predict it. Generally, the security assessment is minimal because the system is assumed to be "behind the firewall": in a secure cabinet in a plane, in an access-controlled building or factory, etc. Of course, all hell breaks loose the day someone decides to connect that system to the Internet, where thousands of smart humans with bad intentions try to break it. I think the author would have helped his point better if he simply talked about secure software, not safety-critical software.
  - killjoywashere 7y ago
    > Safety-critical systems and secure systems are completely orthogonal This strikes me as the most true statement in this conversation. I feel like I need this as a bumper sticker. I think a significant issue in this space is figuring out how to get the security people to put as much skin in the game as the safety-critical people have. The security people often get to continue paying their mortgage as long as they say "no". And as a physician leading a development team, I really struggle with this. If the budgets need to go up to solve the compliance problems, that's fine, but the compliance people seem to always have one more tissue-thin layer of requirements to add. It's like trying to drill your way out of a growing onion.
    - tptacek 7y ago
      Compliance people and software security people aren't the same people. Software security people aren't generally in the business of saying "no"; they start with the requirements of the system, established elsewhere, and try to ensure that the implementation of those requirements doesn't cough up calc.exe.
      - killjoywashere 7y ago
        Poor control of nouns on my part, fair enough. I'm happy to hire SWEs with an interest in secure systems. But how do I move the compliance piece away from static pre-defined standards and toward something that supports a devops development paradigm? What's the mutually agreed trust-but-verify layer? Testing? And how do you get that "MVP" out the door in that case?
- NovemberWhiskey 7y ago
  I absolutely concede that last point - we (the software engineering profession) do not know how to make software that is robust in extremis at prices that are considered reasonable, and we do not know how make some types of software robust at all.
- Veserv 7y ago
  Anybody who said that about EAL4+ systems mischaracterized the standard. The standard only specifies resistance to casual and inadvertent attack at that level which is consistent with the observed outcomes. It is only at EAL5 that the standard specifies resistance against moderately skilled attackers. Note that this is not disagreement with your later point. Most safety critical industries deploy software that is recklessly inadequate with respect to security and have no idea how to achieve the required level of security.

tptacek 7y ago

This is approximately what people used to say about EAL4+ certified products, until everyone realized you can just look at the list of EAL4+ products and the CVE database and compare. Look, I don't doubt that there are avionics systems that are secure by design (largely because their inputs and their functionality are extremely constrained). But that's a question-begging definition of "safety critical", because --- my expertise isn't in avionics but I've done a fair bit of utilities and medical work, for example --- those "safety critical" systems tend to be embedded in larger distributed compute systems that are riddled with vulnerabilities, with an ultimate systemic security result of "some doofus on the Internet can blow up a transformer". So if the argument is "you can in fact built safety-critical systems by massively expanding the cost in money and time to built drastically simplified and less useful components", sure, you win. But if your argument is "the safety critical industries all know how to field software-based infrastructure that is safe from attack", then, based simply on personal experience, I'm going to call "shenanigans", because no, they cannot.

pnako 7y ago
Safety-critical systems and secure systems are completely orthogonal. Although the article makes a good point, I don't think it was very sensible to use the expression "safety-critical software", for this reason. In a safety-critical system (which might include software), you are trying to protect yourself against mother nature. Which is tricky but doable, because we can predict it. Generally, the security assessment is minimal because the system is assumed to be "behind the firewall": in a secure cabinet in a plane, in an access-controlled building or factory, etc. Of course, all hell breaks loose the day someone decides to connect that system to the Internet, where thousands of smart humans with bad intentions try to break it. I think the author would have helped his point better if he simply talked about secure software, not safety-critical software.
- killjoywashere 7y ago
  > Safety-critical systems and secure systems are completely orthogonal This strikes me as the most true statement in this conversation. I feel like I need this as a bumper sticker. I think a significant issue in this space is figuring out how to get the security people to put as much skin in the game as the safety-critical people have. The security people often get to continue paying their mortgage as long as they say "no". And as a physician leading a development team, I really struggle with this. If the budgets need to go up to solve the compliance problems, that's fine, but the compliance people seem to always have one more tissue-thin layer of requirements to add. It's like trying to drill your way out of a growing onion.
  - tptacek 7y ago
    Compliance people and software security people aren't the same people. Software security people aren't generally in the business of saying "no"; they start with the requirements of the system, established elsewhere, and try to ensure that the implementation of those requirements doesn't cough up calc.exe.
    - killjoywashere 7y ago
      Poor control of nouns on my part, fair enough. I'm happy to hire SWEs with an interest in secure systems. But how do I move the compliance piece away from static pre-defined standards and toward something that supports a devops development paradigm? What's the mutually agreed trust-but-verify layer? Testing? And how do you get that "MVP" out the door in that case?
NovemberWhiskey 7y ago
I absolutely concede that last point - we (the software engineering profession) do not know how to make software that is robust in extremis at prices that are considered reasonable, and we do not know how make some types of software robust at all.
Veserv 7y ago
Anybody who said that about EAL4+ systems mischaracterized the standard. The standard only specifies resistance to casual and inadvertent attack at that level which is consistent with the observed outcomes. It is only at EAL5 that the standard specifies resistance against moderately skilled attackers. Note that this is not disagreement with your later point. Most safety critical industries deploy software that is recklessly inadequate with respect to security and have no idea how to achieve the required level of security.