5 ms·

I worry less about software / hardware crashes leading to death and more about cyberattack. I've written about it before[0]: We need regulations before a mass b

I worry less about software / hardware crashes leading to death and more about cyberattack. I've written about it before[0]: We need regulations before a mass breach of autonomous systems happens. Not after. This whole bullshit where corporations claim all sorts of things about their security then get owned trivially has happened for every OS and device out there. The supposedly secure QNX was installed into countless tanks, nuclear power plants, and autonomous devices before getting owned in 2017[1]. I don't buy this "just trust us" attitude from automotive and other autonomous manufacturers. One of them is going to fuck up at some point. [0] https://www.zachaysan.com/cars https://www.zachaysan.com/cars and apologies for the length; it was a very stressful time in my life. [1] http://support.blackberry.com/kb/articleDetail?articleNumber=000046674 http://support.blackberry.com/kb/articleDetail?articleNumber...

mhneu 8y ago
YES. Security compromise is the huge achilles heel of self-driving cars - and anything that is algorithmically controlled. When you have 100,000 self-driving cars on the road, and someone figures out how to hack them, and drive them simultaneously into crowds, that will be a BIG BIG problem. And this problem isn't just about self-driving cars. It is a fundamental issue with all algorithms. It's a lot harder to hack humans to do bad things (although it can be done with sustained messaging and propaganda). But algorithms can be compromised and exploited.
- TeMPOraL 8y ago
  > It's a lot harder to hack humans to do bad things I'd say, it's probably easier to hack individual humans, but - like everything in computing - hacking algorithms scales well, while human factors generally don't.
  - mhneu 8y ago
    A thoughtful point.
- Shivetya 8y ago
  Let us not also forget, we also want to prevent governments from having the ability to shut down them down as well. While there may be a need to disable a car in a chase or such I am against the idea of wholesale shutting down a highway full of cars or similar scenarios.
- 27182818284 8y ago
  Has a flight ever been brought down through a cyberattack? I've heard a lot of concern over that over the years, but (thank goodness) I don't recall any news of an actual flight being brought down.
  - tjoff 8y ago
    It has already been demonstrated non-self-driving cars. https://www.bbc.com/news/technology-33650491 https://www.bbc.com/news/technology-33650491
  - gamblor956 8y ago
    Pilots have the ability to manually disengage airplane autopilots, and more importantly the time to do so (on the order of minutes). It's very different in a car, where the difference between life and death can be a split-second reaction.
  - mrguyorama 8y ago
    The difference is the massive amounts of regulations concerning flight systems.
  - 3pt14159 8y ago
    There are multiple safeguards for planes that don't work for self-driving cars. First, they're not physically inspectable by every Tom, Dick, and Harry. This means it's easier to obscure interfaces and other potential areas of attack. Not state-actor proof, but probably ISIS-proof. Second, they only get software updates while parked at an airport, so it makes MITM attacks harder (though not impossible). Third, they're in the air where death isn't a sudden movement away. Pilots can override the system and fly it manually. Fourth, they're heavily monitored with errant flightpaths reported to militaries around the world (to stop another 9/11). We have none of these safeguards for self-driving cars and there are going to be hundreds of millions of them.
  - goodmachine 8y ago
    Not in-flight as far as we know, but a Boeing 757 was hacked in a proof of concept attack on the tarmac quite recently https://www.computing.co.uk/ctg/news/3020901/dhs-team-manages-to-hack-boeing-757 https://www.computing.co.uk/ctg/news/3020901/dhs-team-manage...
nbeleski 8y ago
We already have a number on embedded processors and controllers in cars today, including systems responsible for critical operational parts of the car. How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? I don't intend to be critical at all, I believe this is a very interesting point too that should be discussed. (I hope to check the links on my interval).
- 394549 8y ago
  > We already have a number on embedded processors and controllers in cars today, including systems responsible for critical operational parts of the car. > How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? It might make always on data connections near-mandatory (to get maps data, etc). At least in my car, the insecure embedded computers are air-gapped from the internet.
- TeMPOraL 8y ago
  > How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? Part of the reason is that, in current ecosystem, companies will find ways in which self-driving requires being constantly on-line and connected to vendor's server. Off-line processing is not in fashion these days.
  - JimmyAustin 8y ago
    Off-line processing is absolutely in fashion with SDCs. Can't quickly react to a pedestrian jumping in front of your car when you have to bounce 12 1080p video streams to a AWS server.
  - Piskvorrr 8y ago
    Fashion or not, the speed of light is not going anywhere - latency can only be added, never removed. If the choice is "process locally or receive stale data", it's not actually a choice.

mhneu 8y ago

YES. Security compromise is the huge achilles heel of self-driving cars - and anything that is algorithmically controlled. When you have 100,000 self-driving cars on the road, and someone figures out how to hack them, and drive them simultaneously into crowds, that will be a BIG BIG problem. And this problem isn't just about self-driving cars. It is a fundamental issue with all algorithms. It's a lot harder to hack humans to do bad things (although it can be done with sustained messaging and propaganda). But algorithms can be compromised and exploited.

TeMPOraL 8y ago
> It's a lot harder to hack humans to do bad things I'd say, it's probably easier to hack individual humans, but - like everything in computing - hacking algorithms scales well, while human factors generally don't.
- mhneu 8y ago
  A thoughtful point.
Shivetya 8y ago
Let us not also forget, we also want to prevent governments from having the ability to shut down them down as well. While there may be a need to disable a car in a chase or such I am against the idea of wholesale shutting down a highway full of cars or similar scenarios.
27182818284 8y ago
Has a flight ever been brought down through a cyberattack? I've heard a lot of concern over that over the years, but (thank goodness) I don't recall any news of an actual flight being brought down.
- tjoff 8y ago
  It has already been demonstrated non-self-driving cars. https://www.bbc.com/news/technology-33650491 https://www.bbc.com/news/technology-33650491
- gamblor956 8y ago
  Pilots have the ability to manually disengage airplane autopilots, and more importantly the time to do so (on the order of minutes). It's very different in a car, where the difference between life and death can be a split-second reaction.
- mrguyorama 8y ago
  The difference is the massive amounts of regulations concerning flight systems.
- 3pt14159 8y ago
  There are multiple safeguards for planes that don't work for self-driving cars. First, they're not physically inspectable by every Tom, Dick, and Harry. This means it's easier to obscure interfaces and other potential areas of attack. Not state-actor proof, but probably ISIS-proof. Second, they only get software updates while parked at an airport, so it makes MITM attacks harder (though not impossible). Third, they're in the air where death isn't a sudden movement away. Pilots can override the system and fly it manually. Fourth, they're heavily monitored with errant flightpaths reported to militaries around the world (to stop another 9/11). We have none of these safeguards for self-driving cars and there are going to be hundreds of millions of them.
- goodmachine 8y ago
  Not in-flight as far as we know, but a Boeing 757 was hacked in a proof of concept attack on the tarmac quite recently https://www.computing.co.uk/ctg/news/3020901/dhs-team-manages-to-hack-boeing-757 https://www.computing.co.uk/ctg/news/3020901/dhs-team-manage...

nbeleski 8y ago

We already have a number on embedded processors and controllers in cars today, including systems responsible for critical operational parts of the car. How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? I don't intend to be critical at all, I believe this is a very interesting point too that should be discussed. (I hope to check the links on my interval).

394549 8y ago
> We already have a number on embedded processors and controllers in cars today, including systems responsible for critical operational parts of the car. > How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? It might make always on data connections near-mandatory (to get maps data, etc). At least in my car, the insecure embedded computers are air-gapped from the internet.
TeMPOraL 8y ago
> How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars? Part of the reason is that, in current ecosystem, companies will find ways in which self-driving requires being constantly on-line and connected to vendor's server. Off-line processing is not in fashion these days.
- JimmyAustin 8y ago
  Off-line processing is absolutely in fashion with SDCs. Can't quickly react to a pedestrian jumping in front of your car when you have to bounce 12 1080p video streams to a AWS server.
- Piskvorrr 8y ago
  Fashion or not, the speed of light is not going anywhere - latency can only be added, never removed. If the choice is "process locally or receive stale data", it's not actually a choice.