6 ms·

The future will be interesting. Imagine what havic you could wreak with a worm that is able to control a car (this worm was not). Let the worm spread for a wh

by sjaak 5y ago

The future will be interesting. Imagine what havic you could wreak with a worm that is able to control a car (this worm was not). Let the worm spread for a while through the fleet and activate a malicious piece of code at a set moment that accelerates and steers the cars into an object. There's not enough medical personnel to tend to all these accidents. Total chaos. Am I wrong in thinking that with the passing of time the probability of such an event tends to 1?

gorpomon 5y ago
You're not wrong, that scenario was in a Fast & Furious movie. I'm guessing it's something numerous parties are considering in some capacity.
anotha1 5y ago
You're wrong. Computer "worms" as people think of them are not the problem here. They're actually well understood. And while propagation seems like it could be so fast it's overwhelming, I think there are a lot of reasons to consider that exceedingly unlikely. These "worms" are the equivalent of catching a communicable illness that is *physically manipulating* the car (even if it's just software). What's the bigger issue? Adversarial manipulation of the sensor inputs. This can be equated to verbal or visual manipulation in humans, something that becomes much harder to detect. While most of these attacks would be against a local target, I could also see a widespread deployment that goes unnoticed and slowly degrades many neural networks, and those being erroneously propagated. The latter is a much bigger problem than "worms" because it's effectively invisible. We can audit and identify malicious code, there's an entire industry built around that. But, neural networks are for the most part still a black box solution. How does one detect and solve manipulation in a black box solution? Well, maybe my Tesla will meet your Tesla in therapy and they can talk about it.
- xorfish 5y ago
  > "I think there are a lot of reasons to consider that exceedingly unlikely" What are those reasons?
  - anotha1 5y ago
    The code-based manipulations are relatively well understood by cyber security professionals, which Tesla undoubtedly has on staff. There are existing solutions to at least slow the spread of these, even if it's as rudimentary as an emergency shutoff as soon as Tesla recognizes an anomaly. Edit: downvote for what?
    - buran77 5y ago
      Don't forget that the "Move fast and break things" motto was born out of the software world. So claiming something is a software company works both for and against them in this case. Think of Heartbleed to understand why having software engineers and literally millions of eyes on a problem might still mean nothing in the grand scheme of things. And understanding a problem only takes you one step closer to solving it. How you solve it, and how you keep doing this day after day going forward makes a lot of difference either way. The truth is the moment it's technically possible to hack a car remotely, cars will be hacked remotely. We've had computers of all kinds for decades and couldn't manage to make them "hack-proof". Consoles are as close as it gets and I'm sure if they were as critical as a car they would have been thoroughly hacked by now. Having any kind of "self driving" feature means safety critical systems (acceleration, braking, steering) can be controlled entirely by the car's computer. And having OTA updates means there is some link between that critical computer and the outside world. And in that outside world people managed to hack airgapped computers in a military nuclear facility. If only that facility was "a software company"... they could have CI/CDed the malware in their infrastructure.
    - ceejayoz 5y ago
      Iran has cyber security professionals - and physical controls over the hardware - but thus far that doesn't seem to be perfect protection against stuff like Stuxnet. The idea that cars can't be effectively and rapidly hacked seems overconfident.
      - anotha1 5y ago
        Iran has insufficient resources compared to the nations targeting it. Unless a nation state decides to target Tesla, we're probably alright.
- bsanr2 5y ago
  In short: Google still hasn't figured out how to stop black people from showing up in Photos searches for "gorillas" (hence, such searches return no results; go ahead, try it). The irrevocable "poisoning" of computer vision systems is a real threat.
  - anotha1 5y ago
    Yes, that's exactly my point. No code required.
    - bsanr2 5y ago
      The real fun will begin when some new technology comes along that appears similar but operates differently from what it's replacing, leading to complications with the vision systems. We already had a traffic-related version of this happen: when LEDs began replacing incandescent bulbs in streetlights, engineers had to add heaters to make up for the fact that LEDs didn't melt the snow that accumulated on the housing.
_Microft 5y ago
> The future will be interesting. Imagine what havoc you could wreak with a worm that is able to control your brain-computer interface. Fixed that for you.
aboringusername 5y ago
I certainly think we'll have a new cause of death "computer - AI", whether malicious or not it's something we need to start tracking and keeping tabs on. I wonder if, as we do at the moment with human piloted cars, we'll just shrug it off and offer a passing "poor human" (in the case of injury/death) and continue with our day. Death is around us 24/7 and I am not convinced even if cars were hacked and told to drive into objects we'd care very much, we'd probably fix the bug and move on.
- colechristensen 5y ago
  Yes but when you can sue for wrongful death... problems get taken care of. Insurance companies will learn that their bottom lines are deeply connected to car computer security and then either the network or the vulnerabilities will be gone. A $10 insurance surcharge can move mountains.
colechristensen 5y ago
This is why I intend to keep my 2009 car for decades. The only thing it does for me is a bit of traction control and the only radio it has in it plays music and jesus stations.
kvz 5y ago
Agreed, and there are other viable attack vectors too. Instead of a worm, an unfriendly state or other bad actor with deep enough pockets could hack the update servers that cars get their over-the-air updates from. Employees of the car manufacturer could be compromised to make that easier. Tesla is a software company and probably has a lower chance of getting hit due to expertise and funds being poured into security (even tho not infallible as this post shows), but there’s a race to the bottom and soon enough Car companies that couldn’t pull of decent navigation will have some form of computer-controlled-steering as stockholders are looking at Tesla stock price and breathing down their necks. I have been worrying about this for some time, even tho I am also a tech lover and Tesla driver. was thinking of writing a blog post about this, but seeing this comment, maybe it isn’t a new thought and everybody is already aware of this risk. I think you are right about medical personel. But also: if you make everyone crash around rush hour, you take out a significant share of the working population. And how do you clean up the infrastructure to let trucks and ambulances through again. Not to mention the catastrophe of cars crashing into stores and pedestrians in city centers.
- adamhp 5y ago
  Can we not introduce some sort of hardware short-circuit? Hit this button and it cuts all power and engages brakes at full force.
  - LinuxBender 5y ago
    On a general aviation aircraft this would be the power switch for the ECM. Super easy to add a kill switch, if the car manufacturer chose to do so. No idea what a self driving car would do if it lost its ECM and related controllers.
  - kvz 5y ago
    Terrorists would probably have an easier time exploiting this button, which isn’t exactly safe either, than update servers. Or not, but it does introduce yet another attack vector, or technology that could have a bug that makes it trigger by accident.
indigochill 5y ago
> Am I wrong in thinking that with the passing of time the probability of such an event tends to 1? You're forgetting to factor in the human element. Technology doesn't progress independently. There are dampening effects when the "real world" decides technological possibilities don't fit the world they want (for example, copyright applies artificial limits to the infinite copying potential of digital assets, or the recent EU politics around AI). So I suspect relatively isolated cases like this will eventually lead to a push for legislation on automotive digital security. Cynically, I suspect in a way that raises barriers to entry to the market after the incumbents have secured their market share, but that's still probably better than the alternative.
ifdefdebug 5y ago
I think it should be possible to have two mandatory requirements for all car designs: the steering wheel and the brake pedal override everything else, and that has to be hard-wired in a way impossible to bypass without modifying the hardware. No good for self-driving cars though.
EricE 5y ago
I dunno - you don’t need much more control than killing the engine, which can be done remotely on many vehicles today. Yeah, if you have visions of the Joker hacking control of the Batmobile turning into a large RC car then we are safe from that. But that’s far from saying that remote control isn’t an issue.

by sjaak 5y ago

gorpomon 5y ago

You're not wrong, that scenario was in a Fast & Furious movie. I'm guessing it's something numerous parties are considering in some capacity.

anotha1 5y ago

You're wrong. Computer "worms" as people think of them are not the problem here. They're actually well understood. And while propagation seems like it could be so fast it's overwhelming, I think there are a lot of reasons to consider that exceedingly unlikely. These "worms" are the equivalent of catching a communicable illness that is *physically manipulating* the car (even if it's just software). What's the bigger issue? Adversarial manipulation of the sensor inputs. This can be equated to verbal or visual manipulation in humans, something that becomes much harder to detect. While most of these attacks would be against a local target, I could also see a widespread deployment that goes unnoticed and slowly degrades many neural networks, and those being erroneously propagated. The latter is a much bigger problem than "worms" because it's effectively invisible. We can audit and identify malicious code, there's an entire industry built around that. But, neural networks are for the most part still a black box solution. How does one detect and solve manipulation in a black box solution? Well, maybe my Tesla will meet your Tesla in therapy and they can talk about it.

xorfish 5y ago
> "I think there are a lot of reasons to consider that exceedingly unlikely" What are those reasons?
- anotha1 5y ago
  The code-based manipulations are relatively well understood by cyber security professionals, which Tesla undoubtedly has on staff. There are existing solutions to at least slow the spread of these, even if it's as rudimentary as an emergency shutoff as soon as Tesla recognizes an anomaly. Edit: downvote for what?
  - buran77 5y ago
    Don't forget that the "Move fast and break things" motto was born out of the software world. So claiming something is a software company works both for and against them in this case. Think of Heartbleed to understand why having software engineers and literally millions of eyes on a problem might still mean nothing in the grand scheme of things. And understanding a problem only takes you one step closer to solving it. How you solve it, and how you keep doing this day after day going forward makes a lot of difference either way. The truth is the moment it's technically possible to hack a car remotely, cars will be hacked remotely. We've had computers of all kinds for decades and couldn't manage to make them "hack-proof". Consoles are as close as it gets and I'm sure if they were as critical as a car they would have been thoroughly hacked by now. Having any kind of "self driving" feature means safety critical systems (acceleration, braking, steering) can be controlled entirely by the car's computer. And having OTA updates means there is some link between that critical computer and the outside world. And in that outside world people managed to hack airgapped computers in a military nuclear facility. If only that facility was "a software company"... they could have CI/CDed the malware in their infrastructure.
  - ceejayoz 5y ago
    Iran has cyber security professionals - and physical controls over the hardware - but thus far that doesn't seem to be perfect protection against stuff like Stuxnet. The idea that cars can't be effectively and rapidly hacked seems overconfident.
    - anotha1 5y ago
      Iran has insufficient resources compared to the nations targeting it. Unless a nation state decides to target Tesla, we're probably alright.
bsanr2 5y ago
In short: Google still hasn't figured out how to stop black people from showing up in Photos searches for "gorillas" (hence, such searches return no results; go ahead, try it). The irrevocable "poisoning" of computer vision systems is a real threat.
- anotha1 5y ago
  Yes, that's exactly my point. No code required.
  - bsanr2 5y ago
    The real fun will begin when some new technology comes along that appears similar but operates differently from what it's replacing, leading to complications with the vision systems. We already had a traffic-related version of this happen: when LEDs began replacing incandescent bulbs in streetlights, engineers had to add heaters to make up for the fact that LEDs didn't melt the snow that accumulated on the housing.

_Microft 5y ago

> The future will be interesting. Imagine what havoc you could wreak with a worm that is able to control your brain-computer interface. Fixed that for you.

aboringusername 5y ago

I certainly think we'll have a new cause of death "computer - AI", whether malicious or not it's something we need to start tracking and keeping tabs on. I wonder if, as we do at the moment with human piloted cars, we'll just shrug it off and offer a passing "poor human" (in the case of injury/death) and continue with our day. Death is around us 24/7 and I am not convinced even if cars were hacked and told to drive into objects we'd care very much, we'd probably fix the bug and move on.

colechristensen 5y ago
Yes but when you can sue for wrongful death... problems get taken care of. Insurance companies will learn that their bottom lines are deeply connected to car computer security and then either the network or the vulnerabilities will be gone. A $10 insurance surcharge can move mountains.

colechristensen 5y ago

This is why I intend to keep my 2009 car for decades. The only thing it does for me is a bit of traction control and the only radio it has in it plays music and jesus stations.

kvz 5y ago

Agreed, and there are other viable attack vectors too. Instead of a worm, an unfriendly state or other bad actor with deep enough pockets could hack the update servers that cars get their over-the-air updates from. Employees of the car manufacturer could be compromised to make that easier. Tesla is a software company and probably has a lower chance of getting hit due to expertise and funds being poured into security (even tho not infallible as this post shows), but there’s a race to the bottom and soon enough Car companies that couldn’t pull of decent navigation will have some form of computer-controlled-steering as stockholders are looking at Tesla stock price and breathing down their necks. I have been worrying about this for some time, even tho I am also a tech lover and Tesla driver. was thinking of writing a blog post about this, but seeing this comment, maybe it isn’t a new thought and everybody is already aware of this risk. I think you are right about medical personel. But also: if you make everyone crash around rush hour, you take out a significant share of the working population. And how do you clean up the infrastructure to let trucks and ambulances through again. Not to mention the catastrophe of cars crashing into stores and pedestrians in city centers.

adamhp 5y ago
Can we not introduce some sort of hardware short-circuit? Hit this button and it cuts all power and engages brakes at full force.
- LinuxBender 5y ago
  On a general aviation aircraft this would be the power switch for the ECM. Super easy to add a kill switch, if the car manufacturer chose to do so. No idea what a self driving car would do if it lost its ECM and related controllers.
- kvz 5y ago
  Terrorists would probably have an easier time exploiting this button, which isn’t exactly safe either, than update servers. Or not, but it does introduce yet another attack vector, or technology that could have a bug that makes it trigger by accident.

indigochill 5y ago

> Am I wrong in thinking that with the passing of time the probability of such an event tends to 1? You're forgetting to factor in the human element. Technology doesn't progress independently. There are dampening effects when the "real world" decides technological possibilities don't fit the world they want (for example, copyright applies artificial limits to the infinite copying potential of digital assets, or the recent EU politics around AI). So I suspect relatively isolated cases like this will eventually lead to a push for legislation on automotive digital security. Cynically, I suspect in a way that raises barriers to entry to the market after the incumbents have secured their market share, but that's still probably better than the alternative.

ifdefdebug 5y ago

I think it should be possible to have two mandatory requirements for all car designs: the steering wheel and the brake pedal override everything else, and that has to be hard-wired in a way impossible to bypass without modifying the hardware. No good for self-driving cars though.

EricE 5y ago

I dunno - you don’t need much more control than killing the engine, which can be done remotely on many vehicles today. Yeah, if you have visions of the Joker hacking control of the Batmobile turning into a large RC car then we are safe from that. But that’s far from saying that remote control isn’t an issue.