9 ms·

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an ea

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

mlyle 3y ago
The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).
- MichaelZuo 3y ago
  The 'early release tests' weren't testing an identical copy of the actual update?
  - DannyBee 3y ago
    It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software. Or something like this. It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update. In other words, i would not assume they are idiots.
    - AlotOfReading 3y ago
      It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
      - intern4tional 3y ago
        Likely not their code. OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)
      - DannyBee 3y ago
        Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about. IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key. While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
        AlotOfReading 3y ago
        That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.
        mlyle 3y ago
        It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious. So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid. You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).
        DannyBee 3y ago
        Uh, this is a perfectly normal ota process when dealing with multiple embedded systems. It seems silly to off the cuff assert that it is a bad process or only done by those who have no idea what they are doing.
- whalesalad 3y ago
  A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
  - AlotOfReading 3y ago
    A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that. Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot. I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.
    - Dalewyn 3y ago
      With all due respect, that all sounds like Programmer Induced Problems(tm). Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.
      - whalesalad 3y ago
        I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard. The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis. You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.
- hef19898 3y ago
  I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless... Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
  - enragedcacti 3y ago
    Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle. > Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car? As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle. https://rivian.software/category/public/ https://rivian.software/category/public/
    - hef19898 3y ago
      Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to brakes. And please, please, finish developing the embedded software running on car before shipping said car. Then it can be air gapped, if not it requires OTA and internet access, raising all kinds of security issues...
- refulgentis 3y ago
  Rollouts don't solve problems, they limit who they effect.
  - mlyle 3y ago
    Is not reducing the effective cost of a bad update by 10x or more worthwhile? Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem. While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.
michaelt 3y ago
To me it's all-too-understandable how this is possible. Maybe they've got a test fleet, but it accepts code signed with the test build key. Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. I wouldn't be surprised if, as a newer company, they'd made a 'Minimum Viable Product' secure boot setup & release procedure, and the auto-fallback and fat-finger-protection were waiting to get to the top of the backlog.
- psunavy03 3y ago
  Exhibit A of why a Minimum Viable Product still needs a proper Definition of Done which includes quality standards.
- qmarchi 3y ago
  So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive. > Maybe they've got a test fleet, but it accepts code signed with the test build key. Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection. > Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Based on what the Rivian reports are showing (Speedometer, cameras, safety systems are working), they likely are running their infotainment as a "virtual machine" within their systems. Again, something that Polestar does. Implementation of a watchdog with a "sub-system" like this is relatively braindead simple. > Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. Generally, A/B partitioning is part of the bootloader, the first program that executes after the reset (on many modern processors) pin is released. This also leads to reboot counters and such being stored as part of the NVRAM that is available at boot. Opinion: Maybe I'm biased, but maybe if you can't develop something yourself, there's reason for you to get an off the shelf option that handles a lot of these things. Disclaimer: Former Googler, Worked closely with Automotive.
xyst 3y ago
When a car company is losing money on every car sale. C level execs going to cut corners
- dewski 3y ago
  This is a bad take.
  - worik 3y ago
    Why?
  - xyst 3y ago
    Rivian layoffs earlier this year [2] combined with reports of $33K loss per sale [1]. Rivian is hemorrhaging money right now. RVN IPO’d at $150/share. Now it’s trading at $16/share. All of these indicators of poor leadership to me. No sustainability. Burning cash. Poor company outlook. Poor products. [1] https://tfltruck.com/2023/10/rivian-financial-results-losses-per-truck-news/ https://tfltruck.com/2023/10/rivian-financial-results-losses... [2] https://www.theverge.com/2023/2/1/23581642/rivian-layoff-ev-workers-employees-price-cut https://www.theverge.com/2023/2/1/23581642/rivian-layoff-ev-...
  - OneLeggedCat 3y ago
    This is an inadequate comment.
worik 3y ago
What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones Owners should have to bring the vehicle into a shop to have changes made, and they should be very rare. This lazy, control freakery of the worst kind Something very bad is going on happen and people will die before we realize that it is a stupid dangerous practice
- qmarchi 3y ago
  I understand the sentiment, but think about the alternatives. There are a few different kinds of updates that can be applied, each with their own protective layers. Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Then there's system updates, which is where danger noodle things happen. Automotive manufacturers are significantly more risk averse to updating these components, and generally, if _anything_ within the system looks wonky, it's an immediate revert. If I, as a Polestar owner, wanted to get an update for my vehicle, the nearest service center is 1.5h away. If I lived in Montana (United States), it would be realistically impossible for me to update my car. Thus, if we want to enable competition within the markets, we shouldn't have regulations that force a new manufacturer to have a global network just to add CarPlay to a screen.
  - neoromantique 3y ago
    >Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Also speedometer, which is hardly a convenience feature.
    - fyrn_ 3y ago
      The dashboard panel is working, at least according to Twitter updates. "Only" affects the infotainment console.
      - 3y ago
DannyBee 3y ago
Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed with the wrong cert. Not a bug in the software itself. That is independent of testing the software, but still a distribution issue.
- jandrese 3y ago
  Yeah, but how did the vehicle not just reject the wrong cert and refuse to flash the update?
  - DannyBee 3y ago
    The firmware is probably not just a signed package but signed binaries in the package as well. One is probably signed with the wrong cert. This would not cause the updater to fail unless it verified the certs of all the binaries in the package, which most don't
  - mkipper 3y ago
    I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices. It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time. Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.
- mytailorisrich 3y ago
  My 2c based on your comment: * "signed with the wrong cert" should mean the software package is rejected before it it is installed. * software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.
spaceywilly 3y ago
Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check the number of failed boot attempts. If it was >= 5 reboot attempts without booting successfully, it would go into the recovery partition. There's really no excuse from Rivian on this, this is shoddy
- LargeTomato 3y ago
  I interviewed at Rivian. They told me about how they needed to grant users access to things like keys, AC, ignition, etc. So they built a hierarchical, recursive group checking IAM system. That just felt like a massive product to build and maintain for what really could have been backed by AWS iam. GCP IAM if they really really needed hierarchy. I guess I'm not surprised at this outage.

mlyle 3y ago

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

MichaelZuo 3y ago
The 'early release tests' weren't testing an identical copy of the actual update?
- DannyBee 3y ago
  It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software. Or something like this. It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update. In other words, i would not assume they are idiots.
  - AlotOfReading 3y ago
    It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
    - intern4tional 3y ago
      Likely not their code. OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)
    - DannyBee 3y ago
      Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about. IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key. While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
      - AlotOfReading 3y ago
        That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.
        mlyle 3y ago
        It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious. So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid. You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).
        DannyBee 3y ago
        Uh, this is a perfectly normal ota process when dealing with multiple embedded systems. It seems silly to off the cuff assert that it is a bad process or only done by those who have no idea what they are doing.
whalesalad 3y ago
A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
- AlotOfReading 3y ago
  A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that. Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot. I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.
  - Dalewyn 3y ago
    With all due respect, that all sounds like Programmer Induced Problems(tm). Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.
    - whalesalad 3y ago
      I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard. The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis. You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.
hef19898 3y ago
I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless... Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
- enragedcacti 3y ago
  Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle. > Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car? As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle. https://rivian.software/category/public/ https://rivian.software/category/public/
  - hef19898 3y ago
    Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to brakes. And please, please, finish developing the embedded software running on car before shipping said car. Then it can be air gapped, if not it requires OTA and internet access, raising all kinds of security issues...
refulgentis 3y ago
Rollouts don't solve problems, they limit who they effect.
- mlyle 3y ago
  Is not reducing the effective cost of a bad update by 10x or more worthwhile? Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem. While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.

michaelt 3y ago

To me it's all-too-understandable how this is possible. Maybe they've got a test fleet, but it accepts code signed with the test build key. Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. I wouldn't be surprised if, as a newer company, they'd made a 'Minimum Viable Product' secure boot setup & release procedure, and the auto-fallback and fat-finger-protection were waiting to get to the top of the backlog.

psunavy03 3y ago
Exhibit A of why a Minimum Viable Product still needs a proper Definition of Done which includes quality standards.
qmarchi 3y ago
So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive. > Maybe they've got a test fleet, but it accepts code signed with the test build key. Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection. > Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Based on what the Rivian reports are showing (Speedometer, cameras, safety systems are working), they likely are running their infotainment as a "virtual machine" within their systems. Again, something that Polestar does. Implementation of a watchdog with a "sub-system" like this is relatively braindead simple. > Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. Generally, A/B partitioning is part of the bootloader, the first program that executes after the reset (on many modern processors) pin is released. This also leads to reboot counters and such being stored as part of the NVRAM that is available at boot. Opinion: Maybe I'm biased, but maybe if you can't develop something yourself, there's reason for you to get an off the shelf option that handles a lot of these things. Disclaimer: Former Googler, Worked closely with Automotive.

xyst 3y ago

When a car company is losing money on every car sale. C level execs going to cut corners

dewski 3y ago
This is a bad take.
- worik 3y ago
  Why?
- xyst 3y ago
  Rivian layoffs earlier this year [2] combined with reports of $33K loss per sale [1]. Rivian is hemorrhaging money right now. RVN IPO’d at $150/share. Now it’s trading at $16/share. All of these indicators of poor leadership to me. No sustainability. Burning cash. Poor company outlook. Poor products. [1] https://tfltruck.com/2023/10/rivian-financial-results-losses-per-truck-news/ https://tfltruck.com/2023/10/rivian-financial-results-losses... [2] https://www.theverge.com/2023/2/1/23581642/rivian-layoff-ev-workers-employees-price-cut https://www.theverge.com/2023/2/1/23581642/rivian-layoff-ev-...
- OneLeggedCat 3y ago
  This is an inadequate comment.

worik 3y ago

What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones Owners should have to bring the vehicle into a shop to have changes made, and they should be very rare. This lazy, control freakery of the worst kind Something very bad is going on happen and people will die before we realize that it is a stupid dangerous practice

qmarchi 3y ago
I understand the sentiment, but think about the alternatives. There are a few different kinds of updates that can be applied, each with their own protective layers. Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Then there's system updates, which is where danger noodle things happen. Automotive manufacturers are significantly more risk averse to updating these components, and generally, if _anything_ within the system looks wonky, it's an immediate revert. If I, as a Polestar owner, wanted to get an update for my vehicle, the nearest service center is 1.5h away. If I lived in Montana (United States), it would be realistically impossible for me to update my car. Thus, if we want to enable competition within the markets, we shouldn't have regulations that force a new manufacturer to have a global network just to add CarPlay to a screen.
- neoromantique 3y ago
  >Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Also speedometer, which is hardly a convenience feature.
  - fyrn_ 3y ago
    The dashboard panel is working, at least according to Twitter updates. "Only" affects the infotainment console.
    - 3y ago

DannyBee 3y ago

Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed with the wrong cert. Not a bug in the software itself. That is independent of testing the software, but still a distribution issue.

jandrese 3y ago
Yeah, but how did the vehicle not just reject the wrong cert and refuse to flash the update?
- DannyBee 3y ago
  The firmware is probably not just a signed package but signed binaries in the package as well. One is probably signed with the wrong cert. This would not cause the updater to fail unless it verified the certs of all the binaries in the package, which most don't
- mkipper 3y ago
  I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices. It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time. Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.
mytailorisrich 3y ago
My 2c based on your comment: * "signed with the wrong cert" should mean the software package is rejected before it it is installed. * software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.

spaceywilly 3y ago

Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check the number of failed boot attempts. If it was >= 5 reboot attempts without booting successfully, it would go into the recovery partition. There's really no excuse from Rivian on this, this is shoddy

LargeTomato 3y ago
I interviewed at Rivian. They told me about how they needed to grant users access to things like keys, AC, ignition, etc. So they built a hierarchical, recursive group checking IAM system. That just felt like a massive product to build and maintain for what really could have been backed by AWS iam. GCP IAM if they really really needed hierarchy. I guess I'm not surprised at this outage.