4 ms·

Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed wit

Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed with the wrong cert. Not a bug in the software itself. That is independent of testing the software, but still a distribution issue.

jandrese 3y ago
Yeah, but how did the vehicle not just reject the wrong cert and refuse to flash the update?
- DannyBee 3y ago
  The firmware is probably not just a signed package but signed binaries in the package as well. One is probably signed with the wrong cert. This would not cause the updater to fail unless it verified the certs of all the binaries in the package, which most don't
- mkipper 3y ago
  I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices. It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time. Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.
mytailorisrich 3y ago
My 2c based on your comment: * "signed with the wrong cert" should mean the software package is rejected before it it is installed. * software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.
- DannyBee 3y ago
  The software package probably is signed right but contains multiple signed binaries of which one is signed wrong. Or is multiple signed packages and one is wrong. Or the test cars accept prod and test certs. Or some combo of the above. There are lots of ways this could have broken that doesn't amount to rivian not being able to write software
  - mytailorisrich 3y ago
    I am not criticising Rivian, not least because I don't know the details. That being said, "signed wrong", including all your hypotheses, that results in a bricked unit is definitely a serious oversight in general. This also might highlight why production tests should be run on exactly production units. No tweaks allowed.
    - DannyBee 3y ago
      It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple. Talking about those is much more interesting than just asserting that everything should be a certain way, without any consideration for real world constraints, like cost of units, etc.
      - mytailorisrich 3y ago
        I am not sure why you are being so defensive. It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.
        DannyBee 3y ago
        I'm not defensive, I just find your extreme position remarkably silly. You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units. There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eating your lunch. Which is precisely why its not done, and in the real world tradeoffs are made between what really needs 100% assurance and not. Spending money or losing customers for 5 9s of reliability through testing when two are needed is not a best practice, and is often explicitly called out as such. In the case of rivian, maintaining a significant fleet of expensive, pristine, exact customer spec (ie not debuggable) cars just to try to get 100% prod ota success assurance is unlikely to provide value vs getting 98% assurance and not doing that (by rough calculation, it stands at 98% after this incident).

jandrese 3y ago

Yeah, but how did the vehicle not just reject the wrong cert and refuse to flash the update?

DannyBee 3y ago
The firmware is probably not just a signed package but signed binaries in the package as well. One is probably signed with the wrong cert. This would not cause the updater to fail unless it verified the certs of all the binaries in the package, which most don't
mkipper 3y ago
I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices. It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time. Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.

mytailorisrich 3y ago

My 2c based on your comment: * "signed with the wrong cert" should mean the software package is rejected before it it is installed. * software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.

DannyBee 3y ago
The software package probably is signed right but contains multiple signed binaries of which one is signed wrong. Or is multiple signed packages and one is wrong. Or the test cars accept prod and test certs. Or some combo of the above. There are lots of ways this could have broken that doesn't amount to rivian not being able to write software
- mytailorisrich 3y ago
  I am not criticising Rivian, not least because I don't know the details. That being said, "signed wrong", including all your hypotheses, that results in a bricked unit is definitely a serious oversight in general. This also might highlight why production tests should be run on exactly production units. No tweaks allowed.
  - DannyBee 3y ago
    It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple. Talking about those is much more interesting than just asserting that everything should be a certain way, without any consideration for real world constraints, like cost of units, etc.
    - mytailorisrich 3y ago
      I am not sure why you are being so defensive. It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.
      - DannyBee 3y ago
        I'm not defensive, I just find your extreme position remarkably silly. You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units. There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eating your lunch. Which is precisely why its not done, and in the real world tradeoffs are made between what really needs 100% assurance and not. Spending money or losing customers for 5 9s of reliability through testing when two are needed is not a best practice, and is often explicitly called out as such. In the case of rivian, maintaining a significant fleet of expensive, pristine, exact customer spec (ie not debuggable) cars just to try to get 100% prod ota success assurance is unlikely to provide value vs getting 98% assurance and not doing that (by rough calculation, it stands at 98% after this incident).