7 ms·

Safe and reliable production changes, and how Rivian recently got this wrong

60 points by kelp 3y ago

giantg2 3y ago
[flagged]
froh 3y ago
> In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices. and then it goes into canary testing, "pre flight tests" and rollback
- MattGrommes 3y ago
  Those are very common practices and he's clear about saying "that probably means..." and "This seems like...". Seems like fine writing to me.
  - froh 3y ago
    sure. my point is that the title doesn't say a thing about what it actually is about, and what the article is about indeed.
yazzku 3y ago
Software in cars and OTA is the stupidest thing in recent years. Like the damn laptop riddled with mediocre software wasn't already frustrating enough, let's fuck up cars too.
- csours 3y ago
  Same things with phones. A phone should just make calls, it doesn't need software.
  - esalman 3y ago
    Phone has become much more than a caller. It's become a cybernetic extension of our self, and are even used to validate our identity.
  - TheBlight 3y ago
    After the deluge of spam calls I receive daily I'm somewhat inclined to opt for the opposite direction. No more phonecalls and just using apps for communication.
    - adra 3y ago
      That's so much better. I only communicate with people who have the rich man's apps. No unintended negative side-effects can possibly be found there!
    - Spinfusor 3y ago
      At this point I would like to be there. Telecom companies don't seem to care about spam calls much.
- dopylitty 3y ago
  To be fair cars as currently designed are a pretty stupid idea to begin with. Let’s just waste energy carrying around 5000lbs of car at ridiculous speeds to move around a 200lb person.
  - dmoy 3y ago
    In this case it's even what, 7000 or 8500lbs car? Which is even crazier when you consider road wear scales on the 3rd or 4th power of weight. (Edit: 4th power, damn) https://en.m.wikipedia.org/wiki/Fourth_power_law https://en.m.wikipedia.org/wiki/Fourth_power_law So a 8500 car does 64x as much road wear as my small sedan.
    - infecto 3y ago
      Wrong https://twitter.com/ajisuzu1/status/1681123111364620294?s=46 https://twitter.com/ajisuzu1/status/1681123111364620294?s=46 TLDR; From a road wear perspective there is no real difference between a heavy EV and a lightweight smaller ICE. Edit: Not sure why I get downvoted so heavily. It is just a fact that the weight difference between an EV and comparable ICE has no measurable difference to road wear. People like the above poster just like to touch on the fourth power law but not how the calculations actually work. ESAL is part of that calculation. A 5 axel semi has a ESAL of 2.35, a dumptruck ~4, a 3.5ton vehicle .004, a 3ton vehicle .002. When we are talking about the difference in hundreds of pounds between EV and ICE, there is no wear difference.
      - huytersd 3y ago
        No one can read that thread.
        sarchertech 3y ago
        TLDR paved roads are generally designed to handle large trucks and construction equipment. On such roads passenger vehicles (even heavy electric vehicles) have a negligible impact on pavement life. The difference in road wear between a 2k lbs. vehicle and an 8k lbs vehicle is too small to matter.
steelframe 3y ago
Software is such a powerful tool that I understand motor vehicles having as much code in them as they do. What I don't want is for that software to be shoddy or for it to spy on me. I also want complete control over whether or when it changes, and I want to understand the nature of and reason for the updates, just as I do for my Linux laptop on which I use apt-listchanges before accepting upgrades. For example: apt-listchanges: Changelogs --------------------------- bind9 (1:9.16.48-1) bullseye-security; urgency=high * New upstream version 9.16.48 - CVE-2023-4408: Parsing large DNS messages may cause excessive CPU load - CVE-2023-5517: Querying RFC 1918 reverse zones may cause an assertion failure when "nxdomain-redirect" is enabled ... glibc (2.31-13+deb11u8) bullseye; urgency=medium * debian/patches/any/local-qsort-memory-corruption.patch: Fix a memory corruption in qsort() when using nontransitive comparison functions. ... imagemagick (8:6.9.11.60+dfsg-1.3+deb11u2) bullseye; urgency=medium * Fix CVE-2021-3574: memory leak was found in TIFF coder * Fix CVE-2021-4219: a special crafted file could lead to a DOS. * Fix CVE-2021-20241 / CVE-2021-20243: divide by zero in some coders (Closes: #1013282) And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too? For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.
- kelp 3y ago
  I've also often thought about what an open source car software stack might look like, but with different motivations. I'd love to be able to see more diagnostics about what the car is actually doing and to add 3rd party extensions. For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you. That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.
gambiting 3y ago
To be fair, Volvo did the same thing recently, so it's not just weird American startups that do this - Volvo never released the numbers on how many cars were affected, but I'm a member of several facebook groups for Volvo owners and it was just like an onslaught, people were posting daily warning others about not applying the latest software patch or it had a good chance of bricking your car. Absolutely no idea how that got released into the wild.
- kelp 3y ago
  I hadn't heard about the Volvo one! I had a 2022 Volvo C40 before I got my Rivian R1T. When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian. Was Volvo able to fix it with another OTA or did people have to go in for service?
  - earthscienceman 3y ago
    Imagine being rich enough to buy high end cars every year and worry about OTA updates. What a world.
    - ohthatsnotright 3y ago
      Beats being so poor you can't afford a car, or the only car you can afford is slowly rusting itself away or one no-longer-mass-produced part away from making a repair out of your financial reach. I'd take bad OTA updates anyday.
      - rokkitmensch 3y ago
        GP is referencing the "having the being stolen from stolen". Nevermind that the wealthy ain't fretting for one second over OTA updates. Imagine a personal vehicle budget where the Porsche is technically your daily driver, but you never drive it because you're driven everywhere. A /personal IT team/. A boy can still dream!
aranchelk 3y ago
> With the botched 2023.42 update Rivian explained that they pushed the wrong build with the wrong certificates to customer vehicles. This made me immediately think they probably don’t have a canary fleet of vehicles that they roll out to first. Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.
- kelp 3y ago
  Repeating what you said in my own words to ensure I understand it. I think you're suggesting they might have a canary fleet, but there wasn't anything/enough that preventing a mistake from bypassing the carry fleet before going to production? Could be!
  - aranchelk 3y ago
    Exactly.

60 points by kelp 3y ago

[flagged]

froh 3y ago

> In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices. and then it goes into canary testing, "pre flight tests" and rollback

MattGrommes 3y ago
Those are very common practices and he's clear about saying "that probably means..." and "This seems like...". Seems like fine writing to me.
- froh 3y ago
  sure. my point is that the title doesn't say a thing about what it actually is about, and what the article is about indeed.

yazzku 3y ago

Software in cars and OTA is the stupidest thing in recent years. Like the damn laptop riddled with mediocre software wasn't already frustrating enough, let's fuck up cars too.

csours 3y ago
Same things with phones. A phone should just make calls, it doesn't need software.
- esalman 3y ago
  Phone has become much more than a caller. It's become a cybernetic extension of our self, and are even used to validate our identity.
- TheBlight 3y ago
  After the deluge of spam calls I receive daily I'm somewhat inclined to opt for the opposite direction. No more phonecalls and just using apps for communication.
  - adra 3y ago
    That's so much better. I only communicate with people who have the rich man's apps. No unintended negative side-effects can possibly be found there!
  - Spinfusor 3y ago
    At this point I would like to be there. Telecom companies don't seem to care about spam calls much.
dopylitty 3y ago
To be fair cars as currently designed are a pretty stupid idea to begin with. Let’s just waste energy carrying around 5000lbs of car at ridiculous speeds to move around a 200lb person.
- dmoy 3y ago
  In this case it's even what, 7000 or 8500lbs car? Which is even crazier when you consider road wear scales on the 3rd or 4th power of weight. (Edit: 4th power, damn) https://en.m.wikipedia.org/wiki/Fourth_power_law https://en.m.wikipedia.org/wiki/Fourth_power_law So a 8500 car does 64x as much road wear as my small sedan.
  - infecto 3y ago
    Wrong https://twitter.com/ajisuzu1/status/1681123111364620294?s=46 https://twitter.com/ajisuzu1/status/1681123111364620294?s=46 TLDR; From a road wear perspective there is no real difference between a heavy EV and a lightweight smaller ICE. Edit: Not sure why I get downvoted so heavily. It is just a fact that the weight difference between an EV and comparable ICE has no measurable difference to road wear. People like the above poster just like to touch on the fourth power law but not how the calculations actually work. ESAL is part of that calculation. A 5 axel semi has a ESAL of 2.35, a dumptruck ~4, a 3.5ton vehicle .004, a 3ton vehicle .002. When we are talking about the difference in hundreds of pounds between EV and ICE, there is no wear difference.
    - huytersd 3y ago
      No one can read that thread.
      - sarchertech 3y ago
        TLDR paved roads are generally designed to handle large trucks and construction equipment. On such roads passenger vehicles (even heavy electric vehicles) have a negligible impact on pavement life. The difference in road wear between a 2k lbs. vehicle and an 8k lbs vehicle is too small to matter.

steelframe 3y ago

Software is such a powerful tool that I understand motor vehicles having as much code in them as they do. What I don't want is for that software to be shoddy or for it to spy on me. I also want complete control over whether or when it changes, and I want to understand the nature of and reason for the updates, just as I do for my Linux laptop on which I use apt-listchanges before accepting upgrades. For example: apt-listchanges: Changelogs --------------------------- bind9 (1:9.16.48-1) bullseye-security; urgency=high * New upstream version 9.16.48 - CVE-2023-4408: Parsing large DNS messages may cause excessive CPU load - CVE-2023-5517: Querying RFC 1918 reverse zones may cause an assertion failure when "nxdomain-redirect" is enabled ... glibc (2.31-13+deb11u8) bullseye; urgency=medium * debian/patches/any/local-qsort-memory-corruption.patch: Fix a memory corruption in qsort() when using nontransitive comparison functions. ... imagemagick (8:6.9.11.60+dfsg-1.3+deb11u2) bullseye; urgency=medium * Fix CVE-2021-3574: memory leak was found in TIFF coder * Fix CVE-2021-4219: a special crafted file could lead to a DOS. * Fix CVE-2021-20241 / CVE-2021-20243: divide by zero in some coders (Closes: #1013282) And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too? For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.

kelp 3y ago
I've also often thought about what an open source car software stack might look like, but with different motivations. I'd love to be able to see more diagnostics about what the car is actually doing and to add 3rd party extensions. For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you. That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.

gambiting 3y ago

To be fair, Volvo did the same thing recently, so it's not just weird American startups that do this - Volvo never released the numbers on how many cars were affected, but I'm a member of several facebook groups for Volvo owners and it was just like an onslaught, people were posting daily warning others about not applying the latest software patch or it had a good chance of bricking your car. Absolutely no idea how that got released into the wild.

kelp 3y ago
I hadn't heard about the Volvo one! I had a 2022 Volvo C40 before I got my Rivian R1T. When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian. Was Volvo able to fix it with another OTA or did people have to go in for service?
- earthscienceman 3y ago
  Imagine being rich enough to buy high end cars every year and worry about OTA updates. What a world.
  - ohthatsnotright 3y ago
    Beats being so poor you can't afford a car, or the only car you can afford is slowly rusting itself away or one no-longer-mass-produced part away from making a repair out of your financial reach. I'd take bad OTA updates anyday.
    - rokkitmensch 3y ago
      GP is referencing the "having the being stolen from stolen". Nevermind that the wealthy ain't fretting for one second over OTA updates. Imagine a personal vehicle budget where the Porsche is technically your daily driver, but you never drive it because you're driven everywhere. A /personal IT team/. A boy can still dream!

aranchelk 3y ago

> With the botched 2023.42 update Rivian explained that they pushed the wrong build with the wrong certificates to customer vehicles. This made me immediately think they probably don’t have a canary fleet of vehicles that they roll out to first. Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.

kelp 3y ago
Repeating what you said in my own words to ensure I understand it. I think you're suggesting they might have a canary fleet, but there wasn't anything/enough that preventing a mistake from bypassing the carry fleet before going to production? Could be!
- aranchelk 3y ago
  Exactly.