9 ms·

On Safety Critical Software

71 points by di 7y ago

roymurdock 7y ago
Bait and switch. Author starts with one sentence on safety critical systems then transitions entirely to discussing challenges with insecure, non safety-critical software such as mobile phone OSs, email services, and web browsers for important people (e.g. politicians and policy makers). Yes safety and security are intertwined, no they're not the same and it's just confusing the matter to talk about functional safety-criticality in the context of secure communications. People who need secure communication channels often do their research or have teams to recommend or build systems for them, I don't think it's necessary for Google Chrome to tell the general public what tradeoffs they are making between usability and security (nice to have somewhere in the documentation? sure. but not necessary).
- frenchyatwork 7y ago
  > People who need secure communication channels often do their research That only applies to the fraction of people who know they need secure communication channels or that insecure communication channels are even a thing. The article lists references to several cases where people didn't have adequate security and it caused them real harm. To be fair though, I'm not sure Google or Mozilla are capable of communicating the security trade-offs they're making without putting a huge amount of marketing spin on it, but we should still hold them accountable.
  - maqp 7y ago
    I couldn't agree more. It's a rare sight to see a secure communication tool be completely open about the complex threat model of modern world. When I started working on TFC (a messaging system that utilizes high assurance architecture like hardware-enforced TCB splitting and isolation), I wanted to be completely transparent about the threat model: https://github.com/maqp/tfc/wiki/Threat-model https://github.com/maqp/tfc/wiki/Threat-model I knew not everyone would read it but not including it could do more damage to the user. I could've just reasoned "well, it's the most secure alternative out there, so why bother", but I assumed the system would be surpassed at some point, so I wanted people to know when that would happen. Perhaps, in a way I realized in order to distinguish the project from others, and to help people see the benefits, I would have to teach people about the threat model. In a way even completely unencrypted apps like Palringo could be safe if they would communicate the threat model 100% transparently, i.e. they would teach about the different attacks, and with every step, they would tell you that the app does not protect from such or such an attacker. One problem here is of course, people don't really use the app for its security features, but to ease their lives. It's not that they don't care about security, they just don't evaluate it proactively, but switch to something else, reactively. So I no longer think transparent threat model is enough: the application should do what it can to protect its users. A reasonable limitation isn't "group chats can't have end-to-end encryption" or "multi-device app can't have end-to-end encryption", like Telegram shills/fanboys tout, this is just lack of good design. A reasonable limitation is "networked TCB can use E2EE for everything, but can't protect from remote key exfiltration with zero days". One responsible thing to do, would be to include recommendations to alternative products with security-convenience trade-offs that can't be solved with intelligent design. A kind of "If you're concerned with repeated code delivery problem our Protonmail web client suffers from, please consider using the native mobile apps we have. If OTOH you're concerned with endpoint security, look into traditional PGP with airgapped computer. Refer e.g. to what hak5 did with QR-code based ciphertext transfer". This alone would set protonmail among the best applications, but instead they choose to lie by omission that the JS web-client has equivalent security to their native clients. The threat model between the two apps is large enough to necessitate some kind of notification, but unfortunately they not only haven't addressed the issue, they refused to address the issue when it was pointed out to them. IMO we should consider such companies selfish and greedy.
- tptacek 7y ago
  Your "bait and switch" is the author's whole point: that we intuitively understand the safety criticality of embedded infrastructure components, and have expectations about how they're designed, implemented, and maintained, but don't recognize when those expectations are implicitly imputed to commodity software that happens to serve those roles for some users. If every once in a while a nuclear power plant wound up using an Internet-connected Mac Mini for its control systems, we'd immediately see the problem. But we don't see that problem when browsers and email systems get embedded deep into our political system, thus becoming (perhaps unintentionally) critical infrastructure.
  - mantap 7y ago
    You can say the same thing about electricity. Sometimes there are outages. People manage without it until it gets back up. Electricity is critical infrastructure, yes, but is not safety critical - if someone relies on a constant supply of electricity to keep people alive then they install a generator. If someone relies on a constant supply of Internet to keep people alive they need a slap in the face.
- Spooky23 7y ago
  The point is, people lives are often in the hands of fragile software that doesn’t meet the standard of a safety critical system.
glitchc 7y ago
This is an article written by someone who’s never actually worked on a safety critical system. Such systems have an exhaustive threat model with likelihood estimates developed and verified well before a single line of code is written.
- tptacek 7y ago
  As someone who has (perhaps like the author) done security assessment work for safety critical systems, I sure would like to hear more about these exhaustive threat models with likelihood estimates, because they sure didn't seem to have much to do with preventing me from popping a shell.
  - deleted 7y ago
    [deleted]
  - tedunangst 7y ago
    You popped a shell, therefore it wasn't a true safety critical system.
    - NovemberWhiskey 7y ago
      Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the parent post. Safety-critical systems (at the time: DO-178B design assurance level A in airborne systems) were about custom processor boards based on mature technologies (PowerPC was popular; caches often disabled for maximally predictable peformance), quadruplex redundancy, software with 100% MC/DC test coverage, verification of traceability between source code and object code, zero-runtimes, no dynamic memory allocation, extensive time budgeting analysis for control loops, worst-case stack utilization analyses, hardware watchdogs, extensive built-in-test and multiple independence objectives (verification team had to be independent of implementation team etc) as well as formal methods like data and information flow analysis. These systems are virtually nothing like conventional "information technology" systems.
kahlonel 7y ago
It’s not “safety critical” unless there’s a chance of someone dying as a direct result of the system’s failure. Stop trying to mix two totally different ideas together.
- Tomte 7y ago
  Injury or environmental damage also counts. Mnemonic: Safety is concerned with whether the system can harm its environment. Security is concerned with whether the environment can harm the system.
WalterBright 7y ago
The article is about security, not safety critical. A couple articles I wrote on safety critical software: Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html https://www.digitalmars.com/articles/b39.html Designing Safe Software Systems Part 2 https://www.digitalmars.com/articles/b40.html https://www.digitalmars.com/articles/b40.html
CivBase 7y ago
The article mostly focuses on software security. "Safety critical" software standards generally focus more on stability than security. Security is part of it. However, "safety critical" standards are overkill if you just want to improve security. Regardless, I doubt the most popular consumer-focused software will ever meet the same "safety critical" standards as medical or avionics software. I work on safety critical software and the bureaucracy that comes with it drives innovation to a snails pace. As long as the average user is willing to give up a little stability for cutting-edge features, developers who prioritize "safety critical" standards simply wont be able to compete.

71 points by di 7y ago

Bait and switch. Author starts with one sentence on safety critical systems then transitions entirely to discussing challenges with insecure, non safety-critical software such as mobile phone OSs, email services, and web browsers for important people (e.g. politicians and policy makers). Yes safety and security are intertwined, no they're not the same and it's just confusing the matter to talk about functional safety-criticality in the context of secure communications. People who need secure communication channels often do their research or have teams to recommend or build systems for them, I don't think it's necessary for Google Chrome to tell the general public what tradeoffs they are making between usability and security (nice to have somewhere in the documentation? sure. but not necessary).

frenchyatwork 7y ago
> People who need secure communication channels often do their research That only applies to the fraction of people who know they need secure communication channels or that insecure communication channels are even a thing. The article lists references to several cases where people didn't have adequate security and it caused them real harm. To be fair though, I'm not sure Google or Mozilla are capable of communicating the security trade-offs they're making without putting a huge amount of marketing spin on it, but we should still hold them accountable.
- maqp 7y ago
  I couldn't agree more. It's a rare sight to see a secure communication tool be completely open about the complex threat model of modern world. When I started working on TFC (a messaging system that utilizes high assurance architecture like hardware-enforced TCB splitting and isolation), I wanted to be completely transparent about the threat model: https://github.com/maqp/tfc/wiki/Threat-model https://github.com/maqp/tfc/wiki/Threat-model I knew not everyone would read it but not including it could do more damage to the user. I could've just reasoned "well, it's the most secure alternative out there, so why bother", but I assumed the system would be surpassed at some point, so I wanted people to know when that would happen. Perhaps, in a way I realized in order to distinguish the project from others, and to help people see the benefits, I would have to teach people about the threat model. In a way even completely unencrypted apps like Palringo could be safe if they would communicate the threat model 100% transparently, i.e. they would teach about the different attacks, and with every step, they would tell you that the app does not protect from such or such an attacker. One problem here is of course, people don't really use the app for its security features, but to ease their lives. It's not that they don't care about security, they just don't evaluate it proactively, but switch to something else, reactively. So I no longer think transparent threat model is enough: the application should do what it can to protect its users. A reasonable limitation isn't "group chats can't have end-to-end encryption" or "multi-device app can't have end-to-end encryption", like Telegram shills/fanboys tout, this is just lack of good design. A reasonable limitation is "networked TCB can use E2EE for everything, but can't protect from remote key exfiltration with zero days". One responsible thing to do, would be to include recommendations to alternative products with security-convenience trade-offs that can't be solved with intelligent design. A kind of "If you're concerned with repeated code delivery problem our Protonmail web client suffers from, please consider using the native mobile apps we have. If OTOH you're concerned with endpoint security, look into traditional PGP with airgapped computer. Refer e.g. to what hak5 did with QR-code based ciphertext transfer". This alone would set protonmail among the best applications, but instead they choose to lie by omission that the JS web-client has equivalent security to their native clients. The threat model between the two apps is large enough to necessitate some kind of notification, but unfortunately they not only haven't addressed the issue, they refused to address the issue when it was pointed out to them. IMO we should consider such companies selfish and greedy.
tptacek 7y ago
Your "bait and switch" is the author's whole point: that we intuitively understand the safety criticality of embedded infrastructure components, and have expectations about how they're designed, implemented, and maintained, but don't recognize when those expectations are implicitly imputed to commodity software that happens to serve those roles for some users. If every once in a while a nuclear power plant wound up using an Internet-connected Mac Mini for its control systems, we'd immediately see the problem. But we don't see that problem when browsers and email systems get embedded deep into our political system, thus becoming (perhaps unintentionally) critical infrastructure.
- mantap 7y ago
  You can say the same thing about electricity. Sometimes there are outages. People manage without it until it gets back up. Electricity is critical infrastructure, yes, but is not safety critical - if someone relies on a constant supply of electricity to keep people alive then they install a generator. If someone relies on a constant supply of Internet to keep people alive they need a slap in the face.
Spooky23 7y ago
The point is, people lives are often in the hands of fragile software that doesn’t meet the standard of a safety critical system.

glitchc 7y ago

This is an article written by someone who’s never actually worked on a safety critical system. Such systems have an exhaustive threat model with likelihood estimates developed and verified well before a single line of code is written.

tptacek 7y ago
As someone who has (perhaps like the author) done security assessment work for safety critical systems, I sure would like to hear more about these exhaustive threat models with likelihood estimates, because they sure didn't seem to have much to do with preventing me from popping a shell.
- deleted 7y ago
  [deleted]
- tedunangst 7y ago
  You popped a shell, therefore it wasn't a true safety critical system.
  - NovemberWhiskey 7y ago
    Although we cannot preclude the GP's experience, as a former developer and verifier of safety-critical systems, my experience (from the 2000s) agrees with the parent post. Safety-critical systems (at the time: DO-178B design assurance level A in airborne systems) were about custom processor boards based on mature technologies (PowerPC was popular; caches often disabled for maximally predictable peformance), quadruplex redundancy, software with 100% MC/DC test coverage, verification of traceability between source code and object code, zero-runtimes, no dynamic memory allocation, extensive time budgeting analysis for control loops, worst-case stack utilization analyses, hardware watchdogs, extensive built-in-test and multiple independence objectives (verification team had to be independent of implementation team etc) as well as formal methods like data and information flow analysis. These systems are virtually nothing like conventional "information technology" systems.

kahlonel 7y ago

It’s not “safety critical” unless there’s a chance of someone dying as a direct result of the system’s failure. Stop trying to mix two totally different ideas together.

Tomte 7y ago
Injury or environmental damage also counts. Mnemonic: Safety is concerned with whether the system can harm its environment. Security is concerned with whether the environment can harm the system.

WalterBright 7y ago

The article is about security, not safety critical. A couple articles I wrote on safety critical software: Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html https://www.digitalmars.com/articles/b39.html Designing Safe Software Systems Part 2 https://www.digitalmars.com/articles/b40.html https://www.digitalmars.com/articles/b40.html

CivBase 7y ago

The article mostly focuses on software security. "Safety critical" software standards generally focus more on stability than security. Security is part of it. However, "safety critical" standards are overkill if you just want to improve security. Regardless, I doubt the most popular consumer-focused software will ever meet the same "safety critical" standards as medical or avionics software. I work on safety critical software and the bureaucracy that comes with it drives innovation to a snails pace. As long as the average user is willing to give up a little stability for cutting-edge features, developers who prioritize "safety critical" standards simply wont be able to compete.