3 ms·

True, but realistically the line has to be drawn somewhere. A software malfunction in a microwave oven, or some other device with a heating element, could cause

True, but realistically the line has to be drawn somewhere. A software malfunction in a microwave oven, or some other device with a heating element, could cause a house fire, but that software isn't regulated the way avionics software systems are. In terms of overall fatalities, distracted drivers on their phones can be expected to continue to kill far more than vehicle hackers. It's troubling though that hacking vehicles' brakes could allow for targeted attacks with little chance of being caught. The same applies to insecure medical solutions.

flubert 6y ago
>A software malfunction in a microwave oven, or some other device with a heating element, could cause a house fire... Now I'm curious. Does anyone know the manufacturer and model number for a microwave for which this is true? Have we really replaced hardware thermal switches, fuses, etc. with firmware?
- MaxBarraclough 6y ago
  Perhaps microwaves have hardware protections to prevent heating when the door is open, but a software error could presumably still have it spring to life unexpectedly, which could potentially be dangerous. It's a (perversely) interesting creative exercise to think of catastrophic consequences that might follow from the failure of a system we don't normally think of as critical. Let's restrict it to physical systems, excluding concerns like being banned from Gmail, and issues like misinformation and extremism spreading through social media. • A bug in a sound driver could cause hearing damage, especially if someone is using earphones • Perhaps a bug in a graphics driver could be dangerous to people with epilepsy • A bug in a laptop's battery-management system could cause a fire on an airliner • A bug in an overclocking tool, or in motherboard firmware, wouldn't typically cause a fire, but perhaps • We know that bugs in satnav systems can cause inattentive drivers to cause serious accidents • A bug in an app about food intolerances could have severe consequences. A graphics/rendering bug, perhaps in the OS, could cause the app to misrender.
  - userbinator 6y ago
    Perhaps microwaves have hardware protections to prevent heating when the door is open, but a software error could presumably still have it spring to life unexpectedly, which could potentially be dangerous AFAIK, the magnetron power supply is physicaly connected through multiple door switches, so this is an actual unoverridable and redundant disconnect. One switch shorts out the supply when the door is opened, so that if the others fail, it will still blow the fuse and cut power. I'm pretty sure that design is due to regulation, so I don't see it ever being replaced by software.
Veserv 6y ago
You apply different standards of rigor based on the failure modes. If the avionics software catastrophically fails in a commercial airliner hundreds of people could die. If there is a critical software malfunction in a single gas stove that could result in it filling a house in the middle of a night with gas and there are no other safety precautions then maybe four people could die. This is already a 1-2 order of magnitude difference in risk and easy to mitigate by having a hardware safety interlock which reduces dependence on software reliability. The real problem in these cases is internet-connectivity + software-control. Without internet-connectivity, your primary failure mode is random chance. The probability of failure of each system is largely independent from the others as there are few ways to induce simultaneous failure in all such systems. This means that the worst-case failure mode of any individual error is the worst-case of single-system failure which, for most systems, is relatively limited. However, once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure. Since these systems will all be running the same software, an error in one is an error in all, and you can now exploit the error in all simultaneously. This instantly adds a gigantic single point of failure whose failure mode is critical failure in all systems at the same time. Suddenly, the software failure mode of an internet-connected gas stove with remote turn-on [1] is no longer one house blowing up in the middle of the night killing possibly 4 people, it is every one of the houses buying the 3.5 million gas stoves sold this year [1] blowing up killing possibly 14 million people. The introduction of internet-connectivity raises the worst case failure mode from a tragedy to approximately a nuclear first strike and surpassing Hitler as a mass murderer. They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad. [1] https://www.geappliances.com/ge/connected-appliances/ranges-ovens-cooking.htm https://www.geappliances.com/ge/connected-appliances/ranges-... [2] https://www.statista.com/statistics/221447/gas-range-and-oven-shipment-forecast-in-the-us-by-quarter/ https://www.statista.com/statistics/221447/gas-range-and-ove...
- MaxBarraclough 6y ago
  > If there is a critical software malfunction in a single gas stove that could result in it filling a house in the middle of a night with gas and there are no other safety precautions then maybe four people could die. You've made an assumption about the failure pattern. Perhaps thousands of stoves could fail at once, all encountering the same defect. Something like this could happen if the software fails after 10,000 hours of continuous operation, say. (A bug of this sort once resulted in the deaths of 28 US soldiers, when a missile defence system failed. [0]) > The real problem in these cases is internet-connectivity + software-control I agree this is a very obvious and severe problem, and it's rather shocking that it's even legal to do this. Avionics software is famously strictly regulated, but the software in road vehicles isn't. > Without internet-connectivity, your primary failure mode is random chance It depends. There could be a failure after some number of hours of continuous operation. Perhaps a particular stretch of road could be dangerously mishandled by all self-driving cars of a certain model. There's also a risk that deliberately crafted fake road signs could cause the system to drive dangerously, opening the door to targeted attacks. > once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure That's a good point. It's similar to how we place enormous trust in Windows Update, except we're talking about a life-and-death system. > They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad. Well, no, they plainly aren't, but I do agree with your general point. I wonder if we'll see any kind of general software regulation (that is, not specific to a domain such as aviation) to cover this sort of recklessness. Currently, the company behind such a product would only suffer after the fact, were something to go badly wrong. [0] https://www-users.math.umn.edu/~arnold/disasters/patriot.html https://www-users.math.umn.edu/~arnold/disasters/patriot.htm...

flubert 6y ago

>A software malfunction in a microwave oven, or some other device with a heating element, could cause a house fire... Now I'm curious. Does anyone know the manufacturer and model number for a microwave for which this is true? Have we really replaced hardware thermal switches, fuses, etc. with firmware?

MaxBarraclough 6y ago
Perhaps microwaves have hardware protections to prevent heating when the door is open, but a software error could presumably still have it spring to life unexpectedly, which could potentially be dangerous. It's a (perversely) interesting creative exercise to think of catastrophic consequences that might follow from the failure of a system we don't normally think of as critical. Let's restrict it to physical systems, excluding concerns like being banned from Gmail, and issues like misinformation and extremism spreading through social media. • A bug in a sound driver could cause hearing damage, especially if someone is using earphones • Perhaps a bug in a graphics driver could be dangerous to people with epilepsy • A bug in a laptop's battery-management system could cause a fire on an airliner • A bug in an overclocking tool, or in motherboard firmware, wouldn't typically cause a fire, but perhaps • We know that bugs in satnav systems can cause inattentive drivers to cause serious accidents • A bug in an app about food intolerances could have severe consequences. A graphics/rendering bug, perhaps in the OS, could cause the app to misrender.
- userbinator 6y ago
  Perhaps microwaves have hardware protections to prevent heating when the door is open, but a software error could presumably still have it spring to life unexpectedly, which could potentially be dangerous AFAIK, the magnetron power supply is physicaly connected through multiple door switches, so this is an actual unoverridable and redundant disconnect. One switch shorts out the supply when the door is opened, so that if the others fail, it will still blow the fuse and cut power. I'm pretty sure that design is due to regulation, so I don't see it ever being replaced by software.

Veserv 6y ago

You apply different standards of rigor based on the failure modes. If the avionics software catastrophically fails in a commercial airliner hundreds of people could die. If there is a critical software malfunction in a single gas stove that could result in it filling a house in the middle of a night with gas and there are no other safety precautions then maybe four people could die. This is already a 1-2 order of magnitude difference in risk and easy to mitigate by having a hardware safety interlock which reduces dependence on software reliability. The real problem in these cases is internet-connectivity + software-control. Without internet-connectivity, your primary failure mode is random chance. The probability of failure of each system is largely independent from the others as there are few ways to induce simultaneous failure in all such systems. This means that the worst-case failure mode of any individual error is the worst-case of single-system failure which, for most systems, is relatively limited. However, once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure. Since these systems will all be running the same software, an error in one is an error in all, and you can now exploit the error in all simultaneously. This instantly adds a gigantic single point of failure whose failure mode is critical failure in all systems at the same time. Suddenly, the software failure mode of an internet-connected gas stove with remote turn-on [1] is no longer one house blowing up in the middle of the night killing possibly 4 people, it is every one of the houses buying the 3.5 million gas stoves sold this year [1] blowing up killing possibly 14 million people. The introduction of internet-connectivity raises the worst case failure mode from a tragedy to approximately a nuclear first strike and surpassing Hitler as a mass murderer. They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad. [1] https://www.geappliances.com/ge/connected-appliances/ranges-ovens-cooking.htm https://www.geappliances.com/ge/connected-appliances/ranges-... [2] https://www.statista.com/statistics/221447/gas-range-and-oven-shipment-forecast-in-the-us-by-quarter/ https://www.statista.com/statistics/221447/gas-range-and-ove...

MaxBarraclough 6y ago
> If there is a critical software malfunction in a single gas stove that could result in it filling a house in the middle of a night with gas and there are no other safety precautions then maybe four people could die. You've made an assumption about the failure pattern. Perhaps thousands of stoves could fail at once, all encountering the same defect. Something like this could happen if the software fails after 10,000 hours of continuous operation, say. (A bug of this sort once resulted in the deaths of 28 US soldiers, when a missile defence system failed. [0]) > The real problem in these cases is internet-connectivity + software-control I agree this is a very obvious and severe problem, and it's rather shocking that it's even legal to do this. Avionics software is famously strictly regulated, but the software in road vehicles isn't. > Without internet-connectivity, your primary failure mode is random chance It depends. There could be a failure after some number of hours of continuous operation. Perhaps a particular stretch of road could be dangerously mishandled by all self-driving cars of a certain model. There's also a risk that deliberately crafted fake road signs could cause the system to drive dangerously, opening the door to targeted attacks. > once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure That's a good point. It's similar to how we place enormous trust in Windows Update, except we're talking about a life-and-death system. > They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad. Well, no, they plainly aren't, but I do agree with your general point. I wonder if we'll see any kind of general software regulation (that is, not specific to a domain such as aviation) to cover this sort of recklessness. Currently, the company behind such a product would only suffer after the fact, were something to go badly wrong. [0] https://www-users.math.umn.edu/~arnold/disasters/patriot.html https://www-users.math.umn.edu/~arnold/disasters/patriot.htm...