3 ms·

Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devic

Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an ephemeral boot value that causes device to boot the alternate image, and reboot - only when the image is declared "up" does that get persisted (and then the alternate is upgraded, in order to prevent rollback in the event of a media error). You use watchdogs that are tied to actual forward progress (and not just some demon that the kernel schedules and bangs on the watchdog even if the rest of the system is hung) and if they fail, the WD reboots you. (This is one of the reasons that event driven programming is somewhat preferred - actually processing events from a single dispatch thread makes it easier to reason about the system.) On top of that, you make sure that the core system is an immutable filesystem so that you can validate the _offline_ alternate image before rebooting (write-and-read-back-uncached) and periodically scrub the alternate image (same). Like.. this is all embedded 101, stuff people have been widely doing since the mid 1990s and I think I can find examples going back to the 70s. Sometimes you get a little more sophisticated (allow sub-packages or overlays and use a manifest to check the ensemble instead of just a single image), but it's very standard.

ikiris 3y ago
That sounds out of scope for the MVP. We can worry about redundancies later after we ship.
- roland35 3y ago
  Hey now, preventing SEVs doesn't lead to impact. If we all collectively let this become a raging dumpster fire we can all heroically fix it and greatly exceed expectations for the half.
  - bluGill 3y ago
    You too have noticed most great employee rewards go to someone who if they had done their job well wouldn't have been notived
  - ikiris 3y ago
    This guy FANGs
dcow 3y ago
Assuming Rivian does know embedded 101, my guess is that the infotainment system is running Android and the watchdog reported all green once the system services all came online and that it doesn't actually check whether the application layer is really working because, as you know, that would require the watchdog to run a full regression suite before giving the okay, which isn’t practical. Since the update swapped the system to an internal dev cert, they cant push an immediate update to change the boot args because the management plane daemon won’t connect to the C&C server, or it can but the blob they push wouldn’t pass signature validation, or the TEE won’t unlock the device keys because the roots changed. Whatever the case, someone has to go blow a fuse and re-flash the thing, or at least rewrite the boot args via serial. Just a guess. If it is the most likely “management plane TLS certs” issue, I bet the watchdog won’t confirm the new boot args until the command dispatch daemon gets a pong from the C&C server moving forward (:
KingMachiavelli 3y ago
Did you just use standard Yocto or similar tools to build such images? Are there standard daemons for managing hardware watchdogs (besides systemd since that's too simple as you say)? I think there's a lot of niche knowledge in the embedded space and many programmers are used to cloud systems and at most target. The most embedded experience most programmers have is likely iOS/Android development where all of the actual embedded concerns are handled for you. Even Google (soft)bricked a bunch of phones with the latest Android 14 update [1]. IMO there's not a lot of regular OSS for building embedded systems that comes with A/B partitioning, watchdogs, secure and verified boot - it's all custom at every org and tailored for individual products. [1] https://arstechnica.com/gadgets/2023/11/android-14-patches-ransomware-storage-bug-but-some-users-will-lose-data/ https://arstechnica.com/gadgets/2023/11/android-14-patches-r...
- MarkSweep 3y ago
  I quit my job before I got to deploy this, but RAUC looked like it would handle this for Yocto: https://github.com/rauc/rauc https://github.com/rauc/rauc https://github.com/rauc/meta-rauc https://github.com/rauc/meta-rauc For microcontrollers, Memfault had a good article: https://interrupt.memfault.com/blog/device-firmware-update-cookbook https://interrupt.memfault.com/blog/device-firmware-update-c...
neuralRiot 3y ago
> including devices that are unreachable for example for 6 months of the year That made me think, imagine NASA bricking up the voyager with a SW update.

ikiris 3y ago

That sounds out of scope for the MVP. We can worry about redundancies later after we ship.

roland35 3y ago
Hey now, preventing SEVs doesn't lead to impact. If we all collectively let this become a raging dumpster fire we can all heroically fix it and greatly exceed expectations for the half.
- bluGill 3y ago
  You too have noticed most great employee rewards go to someone who if they had done their job well wouldn't have been notived
- ikiris 3y ago
  This guy FANGs

dcow 3y ago

Assuming Rivian does know embedded 101, my guess is that the infotainment system is running Android and the watchdog reported all green once the system services all came online and that it doesn't actually check whether the application layer is really working because, as you know, that would require the watchdog to run a full regression suite before giving the okay, which isn’t practical. Since the update swapped the system to an internal dev cert, they cant push an immediate update to change the boot args because the management plane daemon won’t connect to the C&C server, or it can but the blob they push wouldn’t pass signature validation, or the TEE won’t unlock the device keys because the roots changed. Whatever the case, someone has to go blow a fuse and re-flash the thing, or at least rewrite the boot args via serial. Just a guess. If it is the most likely “management plane TLS certs” issue, I bet the watchdog won’t confirm the new boot args until the command dispatch daemon gets a pong from the C&C server moving forward (:

KingMachiavelli 3y ago

Did you just use standard Yocto or similar tools to build such images? Are there standard daemons for managing hardware watchdogs (besides systemd since that's too simple as you say)? I think there's a lot of niche knowledge in the embedded space and many programmers are used to cloud systems and at most target. The most embedded experience most programmers have is likely iOS/Android development where all of the actual embedded concerns are handled for you. Even Google (soft)bricked a bunch of phones with the latest Android 14 update [1]. IMO there's not a lot of regular OSS for building embedded systems that comes with A/B partitioning, watchdogs, secure and verified boot - it's all custom at every org and tailored for individual products. [1] https://arstechnica.com/gadgets/2023/11/android-14-patches-ransomware-storage-bug-but-some-users-will-lose-data/ https://arstechnica.com/gadgets/2023/11/android-14-patches-r...

MarkSweep 3y ago
I quit my job before I got to deploy this, but RAUC looked like it would handle this for Yocto: https://github.com/rauc/rauc https://github.com/rauc/rauc https://github.com/rauc/meta-rauc https://github.com/rauc/meta-rauc For microcontrollers, Memfault had a good article: https://interrupt.memfault.com/blog/device-firmware-update-cookbook https://interrupt.memfault.com/blog/device-firmware-update-c...

neuralRiot 3y ago

> including devices that are unreachable for example for 6 months of the year That made me think, imagine NASA bricking up the voyager with a SW update.