10 ms·

Man hacks Tesla firmware, finds new model, has car remotely downgraded

239 points by antman 11y ago

SocksCanClose 11y ago
Great article, and the best is Musk's response. Just one more example of how, ala Allison and Zeilkow's "Essence of Decision," so much of what happens in industrial organizations ("industrial" in the philosophical sense -- meaning organizations with hierarchies, divisions, etc.) is motivated not by the overall interest of the organization (cf. Musk's response), but rather of the more parochial needs of the individual managers. Which is to say someone within the Tesla organization, fearful of an error they made, sought to retaliate against the guy -- even though the retaliation was unauthorized, and even counterproductive. An amazing use case for how Twitter, when optimally utilized as a total free-speech zone, can really help move the world forward, as things like this enable information to percolate directly to the top without winding its way through the "mittelebureaucracy."
- Theodores 11y ago
  What would you do if your latest commits went live and they indeed did have new things in there, e.g. attribute values? You might just roll the whole thing back a release, in the first instance to 'update' the hacker to a 'safe' version. If there were a need to roll everyone back then some type of patched new version would need to be released, or, if speed really mattered, just remote downgrade everyone to something safe. The major version numbers might not be the safe releases, the last release from a previous major version might be safer. Hence back to v.12.x.y for him. No malice be involved, just prudent reaction.
  - maratd 11y ago
    > No malice be involved, just prudent reaction. It's not malice, it's panic and it's profoundly stupid. Only those with root access to the car can access the secret info. There are a handful of people out there who rooted their cars. They are individuals of extraordinary technical ability. You think they don't know how to make a backup? Or how to stop remote access? That's exactly what happened. He blocked them and then deleted the pending downgrade.
    - 21 11y ago
      Let's assume there is a problem caused be the rooting and as a result the Tesla spins out of control and kills a bystander. Would you agree that this extraordinary technical ability individual should be criminally liable in this case?
      - maratd 11y ago
        > Would you agree that this extraordinary technical ability individual should be criminally liable in this case? No. Cars have had software for a while. Tesla is the FIRST and ONLY company to do over-the-air updates. I don't think you should be criminally liable for turning a feature off. Sorry, but once you push to production, the cat is out of the bag. I absolutely HATE this new attitude that it's OK to not thoroughly test code before pushing it to production, because hey, you can always fix it later or roll back, right?
      - maratd 11y ago
        > Would you agree that this extraordinary technical ability individual should be criminally liable in this case? No. There's no mens rea. He should be liable for damages in civil court.
        marme 11y ago
        there is if he is told the downgrade is for a critical safety bug that needs to be rolled back. Then he has shown criminal negligence for not allowing his car to be fixed
        maratd 11y ago
        That's not what the OP said. He didn't say anything about downgrades. All he said was rooting and presumably, messing with the system. Neither qualifies as criminal negligence in my mind. I'm not sure I want to live in a society where gaining access to hardware I own gets me thrown in jail? I would also like to emphasize that this has never happened and is totally hypothetical.
  - gutnor 11y ago
    Or you know, you do it like Apple. You make no comment whatsoever, keep the software and let the rumor machine keep you on top of tech news sites.
fosco 11y ago
I do not understand "Good Hacking is a gift" remark. I would love if Tesla's were open for hacking but a previous article [0] pointed out that there appears to be a strong stance [1] against this. Disclosure:I am a huge fan of the Tesla brand, just cautiously optimistic. [0] https://news.ycombinator.com/item?id=11233898 https://news.ycombinator.com/item?id=11233898 [1] https://news.ycombinator.com/item?id=11234465 https://news.ycombinator.com/item?id=11234465
- condescendence 11y ago
  Elon Musk is someone who takes personal offense to criticism of his company/products. If a hack is to showoff an awesome new car, which this is the case, I doubt he would have any issues with it. On the other hand, if some guy found a major security flaw and posted it all over the internet I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. http://www.huffingtonpost.com/2013/06/24/michael-hastings-car-hacked_n_3492339.html http://www.huffingtonpost.com/2013/06/24/michael-hastings-ca...
  - gherkin0 11y ago
    > On the other hand, if some guy found a major security flaw and posted it all over the internet I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. That's a bit much. They'd probably just revoke his license to use their software, leaving him with an expensive brick.
    - adam12 11y ago
      > I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. That sounds like something the government would do to your car if they suspected you were a terrorist.
      - ChristianBundy 11y ago
        Or a journalist: https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#Death https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#...
grav 11y ago
How did someone manage to find the image from the SHA hash?
- evmar 11y ago
  I just made a lucky guess from the keywords in the article: $ echo -n 'P100D' | sha256sum 5fc38436ec295b0049f186651ebba5fd55e8d7b81eb61cbd00d3f1bf18dd9c81
  - mikeash 11y ago
    It's so short that you can also just brute force it. Using hashcat in "try all short alphanumeric+symbols strings in increasing length" mode on this hash produces the resulting string in about ten seconds on my computer. I imagine it was just guessed, though. Any Tesla fan thinking of "cool new upgrade to the car, currently secret" would put a P100D model near the top of the list of things to try.
    - ChristianBundy 11y ago
      Yep. There are plenty of sites that have already brute-forced hashes for arbitrary n-length alphanumeric strings. https://md5hashing.net/hash/sha256/5fc38436ec295b0049f186651ebba5fd55e8d7b81eb61cbd00d3f1bf18dd9c81 https://md5hashing.net/hash/sha256/5fc38436ec295b0049f186651...
    - nacs 11y ago
      He clearly should have added a salt to that hash..
- pizza 11y ago
  > P100D > (a reverse lookup of the hash posted above yields this) 5 days ago, http://www.teslamotorsclub.com/showthread.php/58951-Let-the-hacking-begin-(Model-S-parts-on-the-bench)/page104?p=1398937&viewfull=1#post1398937 http://www.teslamotorsclub.com/showthread.php/58951-Let-the-...
- takeda 11y ago
Apocryphon 11y ago
So a rogue corporate exec ordered an unauthorized op against a ronin hacker who Robin Hood'ed the underhood of his electric car. We are living in cyberpunk times.
- seiji 11y ago
  If Tesla employees can address individual cars with no oversight or audit trail or "2+ man rule" in place, that's a huge lapse of professional responsibility. We need something like Certificate Transparency to log ACL hits for all these "connected, but proprietary" things in our lives. Do we have any guarantees a rogue tesla employee can't just tell your car to drive off a bridge while you're inside it?
  - benologist 11y ago
    Uber got busted for allowing general access to a "god mode" that let employees tail passengers. Musk may have brought more appropriate practices from PayPal though. http://nypost.com/2016/01/06/uber-will-no-longer-allow-creepy-drivers-to-spy-on-riders-travels/ http://nypost.com/2016/01/06/uber-will-no-longer-allow-creep...
- tjohns 11y ago
  Or perhaps the car detected its firmware had been tampered with, and auto-reverted to a backup image. Not enough evidence to say for sure, so Occam's razor applies.
  - nivla 11y ago
    Wouldn't that be worse security wise? Say if there is an exploit in the wild. The customer upgrades it to the latest version. Now all the bad guy has to do is to mess around with the firmware enough to trick the system into downgrading to the exploitable version.
    - michaelchisari 11y ago
      If you have enough access to tamper with the firmware, you don't need a more exploitable version.
johansch 11y ago
http://mashable.com/2016/02/03/tesla-refuses-rude-customer/ http://mashable.com/2016/02/03/tesla-refuses-rude-customer/
zyxley 11y ago
"They punished him by remotely downgrading his firmware" is a frankly absurd claim... and incorrect, too. The guy's own post [1] said it was "pending" (e.g. downloaded and ready for an automatic update), not installed. Given that someone else in the same thread notes that the update when installed on their car didn't actually fix the important charging bug it was supposed to, it's much more likely that it was "somebody in engineering decided to cancel a bad update" than his paranoid claims. [1]: http://www.teslamotorsclub.com/showthread.php/63905-Tesla-s-response-to-me-leaking-info-about-the-P100D http://www.teslamotorsclub.com/showthread.php/63905-Tesla-s-...
- maratd 11y ago
  > The guy's own post [1] said it was "pending" (e.g. downloaded and ready for an automatic update), not installed. This is correct. It was an attempted downgrade. He stopped it. > it's much more likely that it was "somebody in engineering decided to cancel a bad update" than his paranoid claims. This is unlikely. He wasn't being downgraded to the version prior to the current one. He was being downgraded to a much older version that did not contain any of the secret info. And nobody else was experiencing the downgrade. If there is a critical flaw that requires a rollback, you do it for everyone and it's big news. He also made it clear that he found a ton of other stuff and was not disclosing that (yet). It looks like somebody at Tesla pushed code to production that shouldn't have been pushed since it isn't relevant to any production vehicles. Once the leak happened, there was a bit of a panic and to stop anymore leaks, they tried to downgrade ... which was silly, because if you know how to root the car, you probably know how to make a backup and stop remote access.
sabujp 11y ago
Do you own your car? Just the metal, but not the software that's required to actually make it do anything useful.
PhasmaFelis 11y ago
I don't see any evidence that this guy's car was targeted specifically. From the info in the article, a much more likely scenario is that they accidentally pushed an update with private data to all cars, realized their mistake after the tweet was posted, then rolled back the update on all cars, so they could fix it before rolling it out again. That's exactly what you'd expect them to do and not sinister in the least. I haven't read the 48-page forum discussion linked in the article, so there may be more info there, but at the very least the article writer hasn't adequately backed up his claims.
- PhasmaFelis 11y ago
  I seriously have no idea why this is getting downvoted. Anyone?
  - jessaustin 11y ago
    In TFA it quotes Musk himself confirming OP's claims. Why would Musk do that if they weren't true?
    - betenoire 11y ago
      where does it say that? Are you inferring that from his tweet?
    - PhasmaFelis 11y ago
      He doesn't say that, though. "Wasn't done at my request" means that if it was done, he didn't order it; it neither confirms nor denies that it actually happened. Musk is being cagey; it's possible that he didn't yet know exactly what had occurred or who had ordered it. Again, maybe there's something in that 48-page thread that proves me wrong. I'm fine with that. But it makes no sense to swallow the article's simplified interpretation without first doing the homework.
- betenoire 11y ago
  Makes sense to me. Guy messes around with highly proprietary computer system, computer system does something weird. I also didn't read the forum, but nothing in the article suggests anything retaliatory. Or maybe the computer noticed something weird, and reverted to a known stable configuration.
marincounty 11y ago
$100 a day to view basic service manuals? I will Never buy a Tesla; even if one day I can afford one. I can't believe you wealthy boys are putting up with this. Do you guys really want to be sitting on the side of a road clueless over your toy? I don't expect you boys to pull out the DVOM, and Snap-on tools, but a little knowledge of why it broke down? Isn't it kinda the American way to at least know what the underlying problem is? Or, have we been conditioned into being good obedient victims? Personally, I feel emasculated if I need to ring a ding ding AAA? Especially, if the problem is minor. Will never know if we aren't able to read up on the toy?
- ocdtrekkie 11y ago
  If you can afford a Tesla, you can probably buy a new one before the warranty wears out. And you probably/definitely have roadside service.
  - Vraxx 11y ago
    Not a very good argument considering they're coming out with cheaper models. Not to mention, just because someone bought something doesn't mean they can afford it by a wide margin, it's sad but true. I think the poster you respond to has some points on why it isn't optimal for the end user, but they certainly could have phrased it a bit better.
    - ocdtrekkie 11y ago
      I'm not saying I endorse this position. (I'm amazed I got downvoted as if I did.) I'm just saying this is why people have put up with it: Almost nobody realizes it, because people who buy Teslas generally aren't servicing their own cars, and most of them are still under warranty and they'll probably upgrade before that date arrives.
  - seanp2k2 11y ago
    Same thing with Apple devices; locked down, no user-serviceable parts, replacement parts not sold to consumers...worked out pretty well for them. Most people just don't care and want the New Shiny more, and that's enough to persuade them.
Overtonwindow 11y ago
"Root the car" man what an amazing time we live in.

239 points by antman 11y ago

Great article, and the best is Musk's response. Just one more example of how, ala Allison and Zeilkow's "Essence of Decision," so much of what happens in industrial organizations ("industrial" in the philosophical sense -- meaning organizations with hierarchies, divisions, etc.) is motivated not by the overall interest of the organization (cf. Musk's response), but rather of the more parochial needs of the individual managers. Which is to say someone within the Tesla organization, fearful of an error they made, sought to retaliate against the guy -- even though the retaliation was unauthorized, and even counterproductive. An amazing use case for how Twitter, when optimally utilized as a total free-speech zone, can really help move the world forward, as things like this enable information to percolate directly to the top without winding its way through the "mittelebureaucracy."

Theodores 11y ago
What would you do if your latest commits went live and they indeed did have new things in there, e.g. attribute values? You might just roll the whole thing back a release, in the first instance to 'update' the hacker to a 'safe' version. If there were a need to roll everyone back then some type of patched new version would need to be released, or, if speed really mattered, just remote downgrade everyone to something safe. The major version numbers might not be the safe releases, the last release from a previous major version might be safer. Hence back to v.12.x.y for him. No malice be involved, just prudent reaction.
- maratd 11y ago
  > No malice be involved, just prudent reaction. It's not malice, it's panic and it's profoundly stupid. Only those with root access to the car can access the secret info. There are a handful of people out there who rooted their cars. They are individuals of extraordinary technical ability. You think they don't know how to make a backup? Or how to stop remote access? That's exactly what happened. He blocked them and then deleted the pending downgrade.
  - 21 11y ago
    Let's assume there is a problem caused be the rooting and as a result the Tesla spins out of control and kills a bystander. Would you agree that this extraordinary technical ability individual should be criminally liable in this case?
    - maratd 11y ago
      > Would you agree that this extraordinary technical ability individual should be criminally liable in this case? No. Cars have had software for a while. Tesla is the FIRST and ONLY company to do over-the-air updates. I don't think you should be criminally liable for turning a feature off. Sorry, but once you push to production, the cat is out of the bag. I absolutely HATE this new attitude that it's OK to not thoroughly test code before pushing it to production, because hey, you can always fix it later or roll back, right?
    - maratd 11y ago
      > Would you agree that this extraordinary technical ability individual should be criminally liable in this case? No. There's no mens rea. He should be liable for damages in civil court.
      - marme 11y ago
        there is if he is told the downgrade is for a critical safety bug that needs to be rolled back. Then he has shown criminal negligence for not allowing his car to be fixed
        maratd 11y ago
        That's not what the OP said. He didn't say anything about downgrades. All he said was rooting and presumably, messing with the system. Neither qualifies as criminal negligence in my mind. I'm not sure I want to live in a society where gaining access to hardware I own gets me thrown in jail? I would also like to emphasize that this has never happened and is totally hypothetical.
- gutnor 11y ago
  Or you know, you do it like Apple. You make no comment whatsoever, keep the software and let the rumor machine keep you on top of tech news sites.

fosco 11y ago

I do not understand "Good Hacking is a gift" remark. I would love if Tesla's were open for hacking but a previous article [0] pointed out that there appears to be a strong stance [1] against this. Disclosure:I am a huge fan of the Tesla brand, just cautiously optimistic. [0] https://news.ycombinator.com/item?id=11233898 https://news.ycombinator.com/item?id=11233898 [1] https://news.ycombinator.com/item?id=11234465 https://news.ycombinator.com/item?id=11234465

condescendence 11y ago
Elon Musk is someone who takes personal offense to criticism of his company/products. If a hack is to showoff an awesome new car, which this is the case, I doubt he would have any issues with it. On the other hand, if some guy found a major security flaw and posted it all over the internet I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. http://www.huffingtonpost.com/2013/06/24/michael-hastings-car-hacked_n_3492339.html http://www.huffingtonpost.com/2013/06/24/michael-hastings-ca...
- gherkin0 11y ago
  > On the other hand, if some guy found a major security flaw and posted it all over the internet I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. That's a bit much. They'd probably just revoke his license to use their software, leaving him with an expensive brick.
  - adam12 11y ago
    > I bet the car would be remotely driven into a brick wall at 100mph+ with the driver inside. That sounds like something the government would do to your car if they suspected you were a terrorist.
    - ChristianBundy 11y ago
      Or a journalist: https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#Death https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#...

grav 11y ago

How did someone manage to find the image from the SHA hash?

evmar 11y ago
I just made a lucky guess from the keywords in the article: $ echo -n 'P100D' | sha256sum 5fc38436ec295b0049f186651ebba5fd55e8d7b81eb61cbd00d3f1bf18dd9c81
- mikeash 11y ago
  It's so short that you can also just brute force it. Using hashcat in "try all short alphanumeric+symbols strings in increasing length" mode on this hash produces the resulting string in about ten seconds on my computer. I imagine it was just guessed, though. Any Tesla fan thinking of "cool new upgrade to the car, currently secret" would put a P100D model near the top of the list of things to try.
  - ChristianBundy 11y ago
    Yep. There are plenty of sites that have already brute-forced hashes for arbitrary n-length alphanumeric strings. https://md5hashing.net/hash/sha256/5fc38436ec295b0049f186651ebba5fd55e8d7b81eb61cbd00d3f1bf18dd9c81 https://md5hashing.net/hash/sha256/5fc38436ec295b0049f186651...
  - nacs 11y ago
    He clearly should have added a salt to that hash..
pizza 11y ago
> P100D > (a reverse lookup of the hash posted above yields this) 5 days ago, http://www.teslamotorsclub.com/showthread.php/58951-Let-the-hacking-begin-(Model-S-parts-on-the-bench)/page104?p=1398937&viewfull=1#post1398937 http://www.teslamotorsclub.com/showthread.php/58951-Let-the-...
takeda 11y ago

Apocryphon 11y ago

So a rogue corporate exec ordered an unauthorized op against a ronin hacker who Robin Hood'ed the underhood of his electric car. We are living in cyberpunk times.

seiji 11y ago
If Tesla employees can address individual cars with no oversight or audit trail or "2+ man rule" in place, that's a huge lapse of professional responsibility. We need something like Certificate Transparency to log ACL hits for all these "connected, but proprietary" things in our lives. Do we have any guarantees a rogue tesla employee can't just tell your car to drive off a bridge while you're inside it?
- benologist 11y ago
  Uber got busted for allowing general access to a "god mode" that let employees tail passengers. Musk may have brought more appropriate practices from PayPal though. http://nypost.com/2016/01/06/uber-will-no-longer-allow-creepy-drivers-to-spy-on-riders-travels/ http://nypost.com/2016/01/06/uber-will-no-longer-allow-creep...
tjohns 11y ago
Or perhaps the car detected its firmware had been tampered with, and auto-reverted to a backup image. Not enough evidence to say for sure, so Occam's razor applies.
- nivla 11y ago
  Wouldn't that be worse security wise? Say if there is an exploit in the wild. The customer upgrades it to the latest version. Now all the bad guy has to do is to mess around with the firmware enough to trick the system into downgrading to the exploitable version.
  - michaelchisari 11y ago
    If you have enough access to tamper with the firmware, you don't need a more exploitable version.

johansch 11y ago

http://mashable.com/2016/02/03/tesla-refuses-rude-customer/ http://mashable.com/2016/02/03/tesla-refuses-rude-customer/

zyxley 11y ago

"They punished him by remotely downgrading his firmware" is a frankly absurd claim... and incorrect, too. The guy's own post [1] said it was "pending" (e.g. downloaded and ready for an automatic update), not installed. Given that someone else in the same thread notes that the update when installed on their car didn't actually fix the important charging bug it was supposed to, it's much more likely that it was "somebody in engineering decided to cancel a bad update" than his paranoid claims. [1]: http://www.teslamotorsclub.com/showthread.php/63905-Tesla-s-response-to-me-leaking-info-about-the-P100D http://www.teslamotorsclub.com/showthread.php/63905-Tesla-s-...

maratd 11y ago
> The guy's own post [1] said it was "pending" (e.g. downloaded and ready for an automatic update), not installed. This is correct. It was an attempted downgrade. He stopped it. > it's much more likely that it was "somebody in engineering decided to cancel a bad update" than his paranoid claims. This is unlikely. He wasn't being downgraded to the version prior to the current one. He was being downgraded to a much older version that did not contain any of the secret info. And nobody else was experiencing the downgrade. If there is a critical flaw that requires a rollback, you do it for everyone and it's big news. He also made it clear that he found a ton of other stuff and was not disclosing that (yet). It looks like somebody at Tesla pushed code to production that shouldn't have been pushed since it isn't relevant to any production vehicles. Once the leak happened, there was a bit of a panic and to stop anymore leaks, they tried to downgrade ... which was silly, because if you know how to root the car, you probably know how to make a backup and stop remote access.

sabujp 11y ago

Do you own your car? Just the metal, but not the software that's required to actually make it do anything useful.

PhasmaFelis 11y ago

I don't see any evidence that this guy's car was targeted specifically. From the info in the article, a much more likely scenario is that they accidentally pushed an update with private data to all cars, realized their mistake after the tweet was posted, then rolled back the update on all cars, so they could fix it before rolling it out again. That's exactly what you'd expect them to do and not sinister in the least. I haven't read the 48-page forum discussion linked in the article, so there may be more info there, but at the very least the article writer hasn't adequately backed up his claims.

PhasmaFelis 11y ago
I seriously have no idea why this is getting downvoted. Anyone?
- jessaustin 11y ago
  In TFA it quotes Musk himself confirming OP's claims. Why would Musk do that if they weren't true?
  - betenoire 11y ago
    where does it say that? Are you inferring that from his tweet?
  - PhasmaFelis 11y ago
    He doesn't say that, though. "Wasn't done at my request" means that if it was done, he didn't order it; it neither confirms nor denies that it actually happened. Musk is being cagey; it's possible that he didn't yet know exactly what had occurred or who had ordered it. Again, maybe there's something in that 48-page thread that proves me wrong. I'm fine with that. But it makes no sense to swallow the article's simplified interpretation without first doing the homework.
betenoire 11y ago
Makes sense to me. Guy messes around with highly proprietary computer system, computer system does something weird. I also didn't read the forum, but nothing in the article suggests anything retaliatory. Or maybe the computer noticed something weird, and reverted to a known stable configuration.

marincounty 11y ago

$100 a day to view basic service manuals? I will Never buy a Tesla; even if one day I can afford one. I can't believe you wealthy boys are putting up with this. Do you guys really want to be sitting on the side of a road clueless over your toy? I don't expect you boys to pull out the DVOM, and Snap-on tools, but a little knowledge of why it broke down? Isn't it kinda the American way to at least know what the underlying problem is? Or, have we been conditioned into being good obedient victims? Personally, I feel emasculated if I need to ring a ding ding AAA? Especially, if the problem is minor. Will never know if we aren't able to read up on the toy?

ocdtrekkie 11y ago
If you can afford a Tesla, you can probably buy a new one before the warranty wears out. And you probably/definitely have roadside service.
- Vraxx 11y ago
  Not a very good argument considering they're coming out with cheaper models. Not to mention, just because someone bought something doesn't mean they can afford it by a wide margin, it's sad but true. I think the poster you respond to has some points on why it isn't optimal for the end user, but they certainly could have phrased it a bit better.
  - ocdtrekkie 11y ago
    I'm not saying I endorse this position. (I'm amazed I got downvoted as if I did.) I'm just saying this is why people have put up with it: Almost nobody realizes it, because people who buy Teslas generally aren't servicing their own cars, and most of them are still under warranty and they'll probably upgrade before that date arrives.
- seanp2k2 11y ago
  Same thing with Apple devices; locked down, no user-serviceable parts, replacement parts not sold to consumers...worked out pretty well for them. Most people just don't care and want the New Shiny more, and that's enough to persuade them.

Overtonwindow 11y ago

"Root the car" man what an amazing time we live in.