4 ms·

The correct answer to that is very simple, and very expensive: just make correct software. Such malware are only possible because the targeted system has vulne

The correct answer to that is very simple, and very expensive: just make correct software. Such malware are only possible because the targeted system has vulnerabilities to begin with. One just has to ensure the absence of such vulnerabilities, possibly using machine checked proofs. One obvious approach is to properly isolate the driving software and sensors from external input.

kbenson 10y ago
> The correct answer to that is very simple, and very expensive: just make correct software. I used to think that was unlikely. Then I had a few discussions with people about the state of C, and things that could be done to make it default to a slightly more deterministic case by changing how undefined behavior is dealt with in regards to optimization. Now I think it's impossible, because nobody is willing to give up even theoretical unknown performance increases for more security. At least for C. You would have to use something that's much more strict about behavior, like Ada with SPARK, or possibly Rust. I'm willing to bet most the code already written for these projects is C or C++. Good luck getting that changed if so.
- loup-vaillant 10y ago
  Forcing a change away from C/C++ is easy: regulation. If a software error (or vulnerability) kills anyone, the vendor is to stop all sales and recall all cars immediately. Sales can resume pending an evaluation period, and the evaluation is to be made by independent experts, at the expense of the vendor. (We should heavily imply that such an evaluation can take months, killing the company in the process.) Correct software is expensive. But if we make sure incorrect software is even more expensive, we'll get correct software. Once the industry is forced to get serious about correctness, they will move away from the C/C++ minefield real quick —or at least come up with safe ways of using C and C++.
dorfsmay 10y ago
+ a mechanical override.
c22 10y ago
Yeah, let's just ask the engineers who are solving one of the toughest robotics problems ever to rewrite the entire house of cards that is their technology stack without any vulnerabilities. Of course even machine checked proofs will have trouble finding vulnerabilities that creep in during the specifications stage or exist in the realm of hardware. How do you isolate your sensors from external input and expect to do anything?
- loup-vaillant 10y ago
  > Yeah, let's just ask the engineers who are solving one of the toughest robotics problems ever to rewrite the entire house of cards that is their technology stack without any vulnerabilities. I did say "very expensive". But yes, if we're serious about correctness, safety, and security, I don't see any other choice. We should scrap the crap and spend the $billions necessary to rebuild it right. Ideally. We don't really need to go that far. Isolation is possible and not that hard, see Qmail: http://hillside.net/plop/2004/papers/mhafiz1/PLoP2004_mhafiz1_0.pdf http://hillside.net/plop/2004/papers/mhafiz1/PLoP2004_mhafiz... > Of course even machine checked proofs will have trouble finding vulnerabilities that creep in during the specifications stage or exist in the realm of hardware. I'm no hardware specialist, but you can still prove the correctness of the design of such and such hardware. The actual chips can still fail the specifications, but that's easier to test once you know the design itself is correct. Also, specifications themselves can be checked. Only the high-level properties must ultimately be decided and reviewed by hand. Not that they won't be complicated or numerous, but that's still much smaller and easier to deal with than the entire implementation. > How do you isolate your sensors from external input and expect to do anything? I wasn't talking about the normal sensor input, which of course can be tricked the same way human sensory input can (for instance with a big flash or something). I was thinking about sensor command, such as which way they should be oriented or something. Though I expect most sensors will have no such input, and will only output to the system.
- ThrowawayR2 10y ago
  It's not as if embedded systems work the same way as the jumbled mess of technology that web servers use. It's perfectly possible to firewall off core system functions into a separate network and separate individual processors for each function. Existing vehicles today have already a strict hardware firewall between the CAN bus[1] and the IVI[2] system to prevent bugs in the more vulnerable and less strictly designed IVI software from sending data that disrupts safety-critical systems. [1] https://en.wikipedia.org/wiki/CAN_bus https://en.wikipedia.org/wiki/CAN_bus [2] https://en.wikipedia.org/wiki/In-car_entertainment https://en.wikipedia.org/wiki/In-car_entertainment
awqrre 10y ago
There are no perfectly safe computers... even when air-gapped, they can get compromised. But one problem with modern cars is that they are connected whether you like it or not.
sangnoir 10y ago
> The correct answer to that is very simple, and very expensive: just make correct software Correct software can be compromised via the update process, or social engineering. It's necessary, but not sufficient to ensure integrity. A cheaper, correct option would be a mechanical lever/breaker labelled "manual override" for licensed drivers or "emergency stop" like they have on every industrial robot or heavy machinery since the 60's. Sometimes the answer to a software problem isn't more/better software.
Noos 10y ago
It's not ransomware i worry about. What about targeting the lidar/radar directly? Some people still have radar/laser jammers, and only ten states ban them. What on earth will your programming do to counter that? Or what if an actor uses something like this: https://techcrunch.com/2016/06/29/now-you-see-me-now-you-dont/ https://techcrunch.com/2016/06/29/now-you-see-me-now-you-don... that confounds the car's sensors. Self-driving cars have a whole spectrum of attack vectors that human beings don't have.
- loup-vaillant 10y ago
  It is likewise quite easy to throw a rock on the highway from a bridge. What you are really worrying about is new creative ways to kill and destroy. Yes, hostile input will have to be taken into account. Fortunately, this is easily detectable (just look at the logs). And if people die as the result, it will count as murder. Finally, one does not simply jam sensors from across the planet. You need a physical presence on site, and that's riskier than a remote hack.
  - Noos 10y ago
    Self-driving cars can make it easier to do this systemically, and not even purposefully. The jamming sensors I meant that another car passes by with a radar jammer on, for its intended purpose (jamming police radar sensors for speeding) and it hits the self-driving car too. A lot of the current driving issues do suck, but they still will suck with self-driving cars, as any computerized system has its own vulnerabilities. It's just they might be systemic and affect a lot more people in ways they can't compensate for.

kbenson 10y ago

> The correct answer to that is very simple, and very expensive: just make correct software. I used to think that was unlikely. Then I had a few discussions with people about the state of C, and things that could be done to make it default to a slightly more deterministic case by changing how undefined behavior is dealt with in regards to optimization. Now I think it's impossible, because nobody is willing to give up even theoretical unknown performance increases for more security. At least for C. You would have to use something that's much more strict about behavior, like Ada with SPARK, or possibly Rust. I'm willing to bet most the code already written for these projects is C or C++. Good luck getting that changed if so.

loup-vaillant 10y ago
Forcing a change away from C/C++ is easy: regulation. If a software error (or vulnerability) kills anyone, the vendor is to stop all sales and recall all cars immediately. Sales can resume pending an evaluation period, and the evaluation is to be made by independent experts, at the expense of the vendor. (We should heavily imply that such an evaluation can take months, killing the company in the process.) Correct software is expensive. But if we make sure incorrect software is even more expensive, we'll get correct software. Once the industry is forced to get serious about correctness, they will move away from the C/C++ minefield real quick —or at least come up with safe ways of using C and C++.

dorfsmay 10y ago

+ a mechanical override.

c22 10y ago

Yeah, let's just ask the engineers who are solving one of the toughest robotics problems ever to rewrite the entire house of cards that is their technology stack without any vulnerabilities. Of course even machine checked proofs will have trouble finding vulnerabilities that creep in during the specifications stage or exist in the realm of hardware. How do you isolate your sensors from external input and expect to do anything?

loup-vaillant 10y ago
> Yeah, let's just ask the engineers who are solving one of the toughest robotics problems ever to rewrite the entire house of cards that is their technology stack without any vulnerabilities. I did say "very expensive". But yes, if we're serious about correctness, safety, and security, I don't see any other choice. We should scrap the crap and spend the $billions necessary to rebuild it right. Ideally. We don't really need to go that far. Isolation is possible and not that hard, see Qmail: http://hillside.net/plop/2004/papers/mhafiz1/PLoP2004_mhafiz1_0.pdf http://hillside.net/plop/2004/papers/mhafiz1/PLoP2004_mhafiz... > Of course even machine checked proofs will have trouble finding vulnerabilities that creep in during the specifications stage or exist in the realm of hardware. I'm no hardware specialist, but you can still prove the correctness of the design of such and such hardware. The actual chips can still fail the specifications, but that's easier to test once you know the design itself is correct. Also, specifications themselves can be checked. Only the high-level properties must ultimately be decided and reviewed by hand. Not that they won't be complicated or numerous, but that's still much smaller and easier to deal with than the entire implementation. > How do you isolate your sensors from external input and expect to do anything? I wasn't talking about the normal sensor input, which of course can be tricked the same way human sensory input can (for instance with a big flash or something). I was thinking about sensor command, such as which way they should be oriented or something. Though I expect most sensors will have no such input, and will only output to the system.
ThrowawayR2 10y ago
It's not as if embedded systems work the same way as the jumbled mess of technology that web servers use. It's perfectly possible to firewall off core system functions into a separate network and separate individual processors for each function. Existing vehicles today have already a strict hardware firewall between the CAN bus[1] and the IVI[2] system to prevent bugs in the more vulnerable and less strictly designed IVI software from sending data that disrupts safety-critical systems. [1] https://en.wikipedia.org/wiki/CAN_bus https://en.wikipedia.org/wiki/CAN_bus [2] https://en.wikipedia.org/wiki/In-car_entertainment https://en.wikipedia.org/wiki/In-car_entertainment

awqrre 10y ago

There are no perfectly safe computers... even when air-gapped, they can get compromised. But one problem with modern cars is that they are connected whether you like it or not.

sangnoir 10y ago

> The correct answer to that is very simple, and very expensive: just make correct software Correct software can be compromised via the update process, or social engineering. It's necessary, but not sufficient to ensure integrity. A cheaper, correct option would be a mechanical lever/breaker labelled "manual override" for licensed drivers or "emergency stop" like they have on every industrial robot or heavy machinery since the 60's. Sometimes the answer to a software problem isn't more/better software.

Noos 10y ago

It's not ransomware i worry about. What about targeting the lidar/radar directly? Some people still have radar/laser jammers, and only ten states ban them. What on earth will your programming do to counter that? Or what if an actor uses something like this: https://techcrunch.com/2016/06/29/now-you-see-me-now-you-dont/ https://techcrunch.com/2016/06/29/now-you-see-me-now-you-don... that confounds the car's sensors. Self-driving cars have a whole spectrum of attack vectors that human beings don't have.

loup-vaillant 10y ago
It is likewise quite easy to throw a rock on the highway from a bridge. What you are really worrying about is new creative ways to kill and destroy. Yes, hostile input will have to be taken into account. Fortunately, this is easily detectable (just look at the logs). And if people die as the result, it will count as murder. Finally, one does not simply jam sensors from across the planet. You need a physical presence on site, and that's riskier than a remote hack.
- Noos 10y ago
  Self-driving cars can make it easier to do this systemically, and not even purposefully. The jamming sensors I meant that another car passes by with a radar jammer on, for its intended purpose (jamming police radar sensors for speeding) and it hits the self-driving car too. A lot of the current driving issues do suck, but they still will suck with self-driving cars, as any computerized system has its own vulnerabilities. It's just they might be systemic and affect a lot more people in ways they can't compensate for.