22 ms·

Rivian software update bricks infotainment system, fix not obvious

277 points by carlivar 3y ago

eschneider 3y ago
This is a bit of a nightmare scenario and why when remote updating, you always test update to your own fleet first. Always.
- kevin_nisbet 3y ago
  Yes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.
- toddmorey 3y ago
  It sounds like it was tested on their own fleet but they accidentally pushed the wrong bits when deploying the update more widely out to customers.
  - carlivar 3y ago
    Maybe they should have an additional phase between test deploy and customers such as "employee personal vehicles".
  - eschneider 3y ago
    The usual "best practice" thing for IoT deploys, is to deploy to "your" devices, what for everyone to go green, then allow that build to deploy more widely. In a well-functioning system, it shouldn't be possible to swap bits between those stages. But who knows what these guys were doing. :/
    - bink 3y ago
      Reading between the lines of their public comments it sounds like they did run a full test through their test fleet, their employees, and then were rolling out to customers when the promotion process was "fat fingered". Maybe someone accidentally promoted the wrong release version.
baz00 3y ago
This is why I have a Dumbcar connected to a Smartphone via bluetooth.
- barbazoo 3y ago
  That's why _I_ have a Dumbcar connected to a smartphone via FM for audio only :)
  - thatguy0900 3y ago
    I've tried that before, but it sounded terrible. What dongle do you use?
    - barbazoo 3y ago
      I mean, totally, it doesn't sound great but that's all I can do, my car doesn't have bluetooth :)
      - BobaFloutist 3y ago
        Does it not have an aux port? USB C to Aux cables are pretty cheap and simple.
        barbazoo 3y ago
        I know I know but then there's a cable from where the phone is to the middle console where the aux port is. But true, it'd for sure be better sound quality.
        ska 3y ago
        > I know I know but then there's a cable from where the phone is to the middle console where the aux port is. This doesn't seem unmanageable, no?
        3y ago
qmarchi 3y ago
It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.
- mlyle 3y ago
  The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).
  - MichaelZuo 3y ago
    The 'early release tests' weren't testing an identical copy of the actual update?
    - DannyBee 3y ago
      It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software. Or something like this. It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update. In other words, i would not assume they are idiots.
      - AlotOfReading 3y ago
        It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
adolph 3y ago
The vehicles are drivable but software and displays go black. It appears that the 2023.42 software update hangs at 90% on the vehicle screen or 50% on the app screen and then the vehicle screens black out. All systems appear to still work except for the displays. This is what I do with my Prius to get a comfortably distraction-free driving environment. Sounds like a feature not a bug.
- altairprime 3y ago
  Technically, the NTSB could order an immediate recall for all Rivian vehicles due to this issue, as the disabled defroster controls are a critical safety issue in cold and/or humid environments. Tesla was forced to issue a recall notice over the controls being buried in a menu; Rivian’s ”defroster unavailable during driving due to manufacturer error” is far worse — especially given the mass and torque of their vehicles, relative to unarmored road users.
- sturza 3y ago
  Instrument cluster display going black is a functional safety/QM issue. No blinker, transmission direction, speed etc confirmations.
  - bri3d 3y ago
    It looks like they correctly isolated the safety critical components on the instrument cluster and they are still functional without infotainment: https://twitter.com/RivianSoftware/status/1724438049675739626 https://twitter.com/RivianSoftware/status/172443804967573962...
wannacboatmovie 3y ago
Interesting to note that Ford's approach of updating software is far more conservative and car-like. It can be done fully offline via USB, but requests that you kindly upload the log files written to the memory stick back to them when complete, in the instructions as a necessary step. Presumably so they can track and stop incidents like this before they happen fleet-wide. Rivian seems more like a "ship it and we'll fix it in the next sprint!" company. How do other manufacturers handle updates?
- sturza 3y ago
  A/B partitions
  - barryrandall 3y ago
    The last time I built something like that, it used partition 1 for the current version, 1 for the last version, 1 with the as-shipped version, and 1 that could restore A or B from the internet or USB.
- post_break 3y ago
  Fords approach is flawed however. You can still update sync with a bad update and bork it over usb. Ask me how I know.
  - r00fus 3y ago
    Pray tell, how painful was your discovery?
    - post_break 3y ago
      Thankfully I was able to repackage it again and flash an upgrade and it unborked itself.
thrill 3y ago
Inexcusable, really.
cs702 3y ago
It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's neither easy nor cheap. All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates!
- Aurornis 3y ago
  > (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. Not really. Vehicle computers aren’t vastly different on every model year and every trim level or option package. These parts are standardized, tested, and carried across model years. Even with changes, the teams would be expected to have the different variants in their development and test cycles. The 2020, 2021, and 2022 model infotainment systems likely share a lot more in common than an iPhone 13, iPhone 14, and iPhone 15 with all of the non-Pro, Pro, and Max variants.
- wannacboatmovie 3y ago
  This isn't a bunch of Windows PCs home-built from a hodgepodge of components. They designed, built, and shipped all the hardware. There is ABSOLUTELY NO excuse for not having a database of the exact hardware configs by serial number. They have the ability to test every single shipped configuration. If they don't, they have already failed as a car company.
  - AlotOfReading 3y ago
    I guarantee they have a database with the hardware configs. It's required by NHTSA to do recalls and notices. They'll undoubtedly be using that to inform the right people to come in. The update servers almost certainly don't talk to that system though.
- xgbi 3y ago
samsquire 3y ago
This is actually a topic that I think about from time to time: how to do aggressive changes to software while they are running. In Ruby world you have monkeypatching. And Linux kernel has livepatching. For example, if you have a distributed system and you want to upgrade a component that every caller uses: you have a large exercise on your hands where you might have to roll out a change over time and then clean up your incremental branches where you have to handle two control flow paths through the code. It reminds me of Google's protobuf required field discussions. It reminds me of repository-per-microservice and a Java library that other microservices use and updating a dependency and having to deploy the change to every service. It's like trying to change wheels on a car while the car is moving or refueling a jet in flight. Unison lang is trying to solve this problem I think, by allowing multiple versions of a function to be available. https://www.unison-lang.org/ https://www.unison-lang.org/ Migrations in databases are painful too. One solution I've thought of which is probably overengineered is that API call sites are an abstract object and their schema and arguments is centrally deployed, I called this "protocol manager". The idea is you write all your code to use a "span" and have contextual data in a span, and you can include or exclude data in a span with a non-software rollout. Your communication schema of RPC and API calls is a runtime decided thing, not hardcoded. If you have N deployed versions of code and you want to upgrade to X, you have to test 1..N to X versions. So nobody does that.
- jbott 3y ago
  You might be interested in learning about Erlang – it supports hot code reloads natively: https://oozou.com/blog/understanding-elixir-otp-applications-part-4-hot-swapping-modules-151 https://oozou.com/blog/understanding-elixir-otp-applications...
- fabianlindfors 3y ago
  The database aspect of this problem is particularly interesting to me. I’ve previously built Reshape [0], a zero-downtime migration tool for Postgres, and am now working on ReshapeDB [1], a full database designed from the ground up to tackle this problem. [0] https://github.com/fabianlindfors/reshape https://github.com/fabianlindfors/reshape [1] https://reshapedb.com https://reshapedb.com
1970-01-01 3y ago
Lexus did the very same thing about 8 years ago: https://www.consumerreports.org/lexus/what-to-do-if-your-lexus-infotainment-system-crashed/ https://www.consumerreports.org/lexus/what-to-do-if-your-lex...
scardycat 3y ago
Bringing CI/CD mindset to cars is probably not a great idea. Software updates to commuter vehicles should have a high bar for operational standards, and a simple thing such as an expired certificate should have never been deployed. Having isolated networks in vehicles helps but doesn't prevent broken updates from, eventually, bricking the cars.
- nomel 3y ago
  I think this shows more of a fundamental flaw in their update mechanism, than anything. I don't think a botched update is a big deal. It happens, and should be expected, in a sane design. The fact that the customer noticed is a big deal. There are many implementations that could be used for an "auto rollback" feature. They either failed to implement that in a sane way, or they were goobers, and assumed things would always be rosy.
  - babypuncher 3y ago
    I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?
    - BuckRogers 3y ago
      It doesn’t. People and these tech companies are tools. And do it largely in search of ways to take more of your money. It’s not a favor.
      - theandrewbailey 3y ago
        If ICE tech was the hot new thing in cars, things like spark plugs would have a chip so that it would fire n times then break, but don't worry, there's a subscription for new ones and they will be automatically ordered when the car says so. If the credit card on file expires, your spark plugs won't work anymore, even if you just replaced them.
qudat 3y ago
What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.
- nomel 3y ago
  > where a "bug" has potentially life threatening consequences. What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence. “The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...” Also, you can't do an update while driving.
  - jawns 3y ago
    Based on the photo included in the article, what they're calling an infotainment system is actually two separate components, one of which appears to be taking the place of a traditional dashboard. If that's the case and there's no other way to monitor speed, fuel levels, engine temperature, warning lights, etc., I'd say that's quite a bit more worrisome than just not being able to play your favorite music while driving.
    - nomel 3y ago
      I understand, but the risk of life wouldn't be from the bug, it would be from conscious choice of driving without a speedo. There's a critical distinction there. In this case, mileage/battery are still present, and I would assume safety critical warnings would still be displayed.
      - jonplackett 3y ago
        If you need thr car in an emergency and it's unusable, that almost counts.
        thrill 3y ago
        "almost" only counts in horseshoes and nukes.
gravitronic 3y ago
I used to work for a company that built satellite receivers that would be installed in all sorts of weird remote environments in order to pull radio or tv from satellite and rebroadcast locally. If we pushed a broken update it might mean someone from the radio company would have to make a trip to go pull the device and send it to us physically. Our upgrader did not run as root, but one time we had to move a file as root.. so I had to figure out a way to exploit our machine reliably from a local user, gain root, and move the file out of the way. We'd then deploy this over the satellite head end and N remote units would receive and run the upgrade autonomously. Fun stuff. Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick.
- nomel 3y ago
  > Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick. No offense, but what a shit show. It makes me assume no source control, and a really good chance that state actors made their way into your network/product. This almost happened at a communication startup I know, with three letter agencies helping resolve it. State actors really like infiltrating communication stuffs.
  - gravitronic 3y ago
    oh, yeah, this place was a total shit show. BUT we were ISO9001 certified!! So we had source control (CVS) and a Process (with a capital P) to follow. In this case that code was added in a previous development iteration because someone needed to run something as root when a user pressed a certain button on the LCD panel in front and this was the decoupled solution they wrote intentionally. Somehow I feel like that makes it worse than if it was a malicious three letter agency lol.
    - nomel 3y ago
      Package this up and send it to https://thedailywtf.com https://thedailywtf.com It's beautiful.
martin8412 3y ago
Stuff like this is why I don't want OTA updates in my cars. Let the car dealership deal with it during regular maintenance. They'll be on the hook for fixing it before handing the car back to me.
- ben_jones 3y ago
  But what if your car doesn’t have the latest emojiset or social sharing functionality within the notes app?
- galangalalgol 3y ago
  Don't even need OTA. A seattle radio station bricked a bunch of mazdas. https://www.autoblog.com/2022/02/09/seattle-radio-station-breaks-mazda-infotainment-systems/ https://www.autoblog.com/2022/02/09/seattle-radio-station-br...
  - cozzyd 3y ago
    Amazing. Can't wait for some car software stack to be so poorly designed that an FM transmitter can remote takeover.
    - galangalalgol 3y ago
      It quite possibly could have with a well formed digital fm payload.
      - cozzyd 3y ago
        We'll hopefully no car manufacturer is dumb enough that the FM radio software can affect vehicle control functions but I wouldn't be surprised ..
        aftbit 3y ago
        I'm pretty sure they are all dumb enough for this. This is why people can steal (modern) cars by ripping a headlight out and accessing the CAN bus through the gap. https://kentindell.github.io/2023/04/03/can-injection/ https://kentindell.github.io/2023/04/03/can-injection/ AIUI the military uses "CAN bus firewalls" to prevent this. There is also some sort of encrypted/authenticated CAN bus protocol in the works. Neither are common in production cars as of 2022 or so, though they may be soon enough.
nicholasjarnold 3y ago
Is it possible, as a licensee of the Rivian vehicle system, to disable the automatic OTA updates without having expert-level knowledge or tooling? Also, yes, I'm specifically avoiding using the word "owner" above for obvious reasons.
- 55873445216111 3y ago
  Rivian "licensee" here. So far all updates have required you to press a button (in the car or on the app) to launch the update installer. Not sure how many weeks you can ignore it for as I never tried.
- bo1024 3y ago
  Confirming that updates are not automatic, and can be ignored indefinitely. For now.
deleted 3y ago
[deleted]
kevinventullo 3y ago
My 2019 car is not connected to the internet. Instead, I use Apple CarPlay for everything. Is there any reason not to do it this way?
- gmane 3y ago
  Then automakers can’t sell your information for fat stacks of cash.
  - lotsofpulp 3y ago
    This, and automakers do not want to become dumb appliances (even though they are for 90% of people).
  - pookha 3y ago
    ..And authoritarian governments can't take complete custody of your movements. Can't shutdown your car either. Seems to be a win-win for both sides (private and public industry). https://qz.com/1522309/how-chinas-electric-car-surveillance-system-works https://qz.com/1522309/how-chinas-electric-car-surveillance-... https://apnews.com/article/4a749a4211904784826b45e812cff4ca https://apnews.com/article/4a749a4211904784826b45e812cff4ca
- ChadMoran 3y ago
  Quite a few reasons but primarily CarPlay doesn't know about your vehicle's charge and road trip navigation requires external inputs to determine your charge route. Apple could potentially offer an API to have "reverse" CarPlay where the car's app can feed information into iOS. I recently rented an Mercedes EV which had Apple CarPlay and it was a weird experience having to manage two sets of experiences.
  - callalex 3y ago
    The exact API you describe already exists in CarPlay.
    - 3y ago
nicolaslem 3y ago
This is the kind of thing that keeps my awake at night. Does anyone here have some practical tips to turn an embedded Linux machine into an appliance? The kind of system that a botched update cannot brick but only momentarily disable until a non-technical user presses a factory reset button of some sort.
- elitepleb 3y ago
  A/B updates as implemented in android, https://bootlin.com/pub/conferences/2022/elce/opdenacker-implementing-A-B-system-updates-with-u-boot/opdenacker-implementing-A-B-system-updates-with-u-boot.pdf https://bootlin.com/pub/conferences/2022/elce/opdenacker-imp...
- hospitalJail 3y ago
  >Does anyone here have some practical tips to turn an embedded Linux machine into an appliance? Lol I suppose this is the negative about having sensors that make sure water gets hot enough to be sanitizing, but not so hot that it wastes energy. And I'm sure you can imagine 100 other uses of having a microcontroller/CPU process data and do feedback. (I'm sure there are EE only ways of doing it, but theoretically possible and useful are two different thigs)
ct0 3y ago
Will insurance carriers cover damages due to botched updates? Imagine 10 years from now the power/control that electric delivery companies would have over retailers like amazon. One botched update away from a complete backup for delivery vans.
eigenvalue 3y ago
Can’t imagine how much it would suck to be the engineer who fat fingered it and caused a huge crisis for the company, inconveniencing tons of customers and costing millions. Even if there should be processes in place to prevent it in the first place, you’d still know you were the “but for cause” of the problem.
xyst 3y ago
Looks like this car brand is circling the drain. Glad I never bought into the hype.
- seattle_spring 3y ago
  It’s circling the drain because of one bad software update? Sounds more like you’ve just bought into the doom and gloom that a few specific news outlets have been pushing.
karaterobot 3y ago
What kinds of changes are generally included in these over the air updates? I have this sudden urge to shake my fist at a cloud and tell the gods that cars shouldn't need updates in the first place, if the car was ever deemed ready for production and then sold to customers for money. But, maybe I'm wrong, and it makes perfect sense. All I can think of would be something like a periodic update to navigation data, is that it?
- ezfe 3y ago
  It’s possible to deem software ready to sell but find improvements later. Simple example: my Subaru was sold to me with an interesting design decision that caused the radio to come on whenever the car was started. This was not a bug. Every Subaru worked this way for years. A year into ownership I received an OTA update that added a “not playing” state on startup. This was never a safety issue and was likely not a defect. It was, however, stupid and needed to be changed.
  - karaterobot 3y ago
    I wish my Mazda had this option! But I would still say that I'd expect them to have included this option before selling the car, especially since radios and user preferences around radio UI are pretty well established.
    - ezfe 3y ago
      Sure, but they didn't - and it's not because the software is complicated. The Subaru headunits are very basic (and more-so a few years ago). When they make an improvement, I like getting the trivial things on my older vehicle. It's better for everyone involved, so long as they do it responsible.
cryptoegorophy 3y ago
Tesla updates are sent in batches and you can opt in for advanced updates I guess to be earlier. Normally when I see that there is an update on Reddit then it takes 1-2 weeks at least to get to my car with the “advanced” updates on.
xyst 3y ago
Tesla. Rivian. All cut from the same cloth. A car should be simple. Yet we are stuffing all of this tech junk into it and trying to repackage is as something else to pump the numbers. Car companies suck at tech. Let’s be realistic. They should stay their lane and focus on improving the car and physical aspects (safety, reducing carbon output, longevity, ease of repairability, reducing supply chain issues)
- bhauer 3y ago
  > Tesla. Rivian. All cut from the same cloth. I'm not aware of any Tesla OTA updates bricking the infotainment system. At least since I've been paying attention. I don't see them quite as similar as you suggest.
  - margalabargala 3y ago
    There have been plenty. https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_tesla_model_3_bricked_after_update/ https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_te... https://teslamotorsclub.com/tmc/threads/failed-software-update-bricked-my-2020-x-today.258670/ https://teslamotorsclub.com/tmc/threads/failed-software-upda... Two examples of many. I'm not aware of any fleet-wide issues that accidentally bricked Teslas, but as one-offs they do happen; and unlike this Rivian update, a botched Tesla OTA generally leaves the car undriveable and needing to be towed. These Rivians will at least still drive, as long as you don't need fancy extraneous luxury features like a...speedometer.
avereveard 3y ago
> In most cases, the rest of the vehicle systems are still operational Like what do you mean "in most cases" I can understand a broken infotainment needing reset but imagine if you had to tow your truck I'd be furious.
Havoc 3y ago
> the vehicle is not bricked What a time to be alive. Software updates (almost) turning cars into paper weights lol
emmelaich 3y ago
Poor title; physical repair is not required. Physical presence is required.
- Someone1234 3y ago
  The article doesn't really state what is required to repair the vehicle. I'd assume if it was as simple as loading a flash drive and plugging it in, then Rivian would have provided a way for customers to self-fix. The second a single body panel is removed to gain access to the headunit, it is a physical repair. So without more info we cannot know if it is accurate or not.
  - emmelaich 3y ago
    I don't think many people would consider removing a body panel to be a physical repair. I think the term is 'back to base' or similar. Physical repair suggests e.g. a burnt out capacitor
WirelessGigabit 3y ago
What's the impact on your insurance should you get into an accident? The speedometer screen is gone, so does that not imply the vehicle is inherently unsafe to drive?
MisterTea 3y ago
Can I please just buy a car with a motor and battery? Why does every god damn vehicle have to come littered with screens and chips all together like some tentacle monster? All I need is a gauge cluster screen that can display the normal info like stored and heading while also letting me configure the cars performance and safety features. Then let me mount a double DIN radio that isn't dog shit. I've not seen a single new car with these dumb screens with a sound system that's not tinny muddy garbage with zero adjustment save for "bass" and "treble" settings. I mean all that technology and you can't be assed to put an eq in there. HVAC never needed more than two or three knobs anyway.
easylion 3y ago
need a easy way to do restore to previous version offline. take 100 bucks extra if required to have a backup ssd. Don’t want to be camping and then realizing i’m stuck because of some junior dev not being competent enough
- seattle_spring 3y ago
  Why would you intentionally upgrade your vehicle software while camping? It’s not like this stuff installs automatically, you have to explicitly accept the installation. Waiting a few days or even a few weeks before hitting “install” is completely normal.
ralmidani 3y ago
Move fast and break things that move fast… I don’t really like or trust most (if not all) of the established automakers, but there is something to be said for having several decades (over a century in some cases) of experience building potential killing machines vs. a company that’s not even 15 years old. The established players have put out cars which suffered freak malfunctions, but Rivian (and Tesla) seem to be struggling more with QA. Non-rhetorical question: do companies have safeguards for critical components like braking systems, or are they also prone to catastrophic failure if a software engineer pushes a bad commit?
- ezfe 3y ago
  The moving fast components were unaffected by this issue…
  - ralmidani 3y ago
    I know, I just thought it was a decent pun.
Someone1234 3y ago
I wonder if the way Microsoft's XBox is designed may be something to look towards in terms of hardware reliability/fallback. Specifically they utilize a Hypervisor which rarely needs updates, running different operating environments which need frequent updates. - Better isolation of different parts of the system (e.g. infotainment unit, instrument cluster, et al). - Better isolation for updates (e.g. run a "beta" update, and a "stable" update side-by-side). - Automatic error detection and rollback (e.g. if a VM keeps restarting after an update). - Ease of offering features like rollbacks to end-users. - Rare hypervisor updates can be held to a much higher standard relative to other VM updates. The only downside of hypervisor-based systems is slightly higher hardware costs. But even that is largely mitigated by modern architectures that natively support virtualization. PS - You can also look to any containerization. I specifically brought up the XBox because it is a hardware product, just like a vehicle.
antoniuschan99 3y ago
Wondering why there isn’t an option for a factory reset (eg. press and hold with a paperclip for 10 seconds)
glonq 3y ago
As somebody who has spent many years doing embedded+iot related to remote fleet firmware updates, this is the kind of thing that lurks in my nightmares. I'd love to be a fly on the wall at Rivian engineering/operations this week!
janitor61 3y ago
This is tangentially Rivian related, but does anyone else see the inherent danger of stylized tail lights that are just a single red bar across the back of the car? Travelling on the freeway at night I can't really gauge the distance to the car in front of me if it's far ahead and if there's no discernable left and right brake lights. I'd believe Rivians and other cars like that are more at risk of high speed rear-end collisions.
- deleted 3y ago
  [deleted]
- rurp 3y ago
  This reminds of the terrible turn signals Mini used, which look like flashing arrows pointing in the opposite direction of the turn[0]. Getting cute with basic stuff like tail lights is forgettable or annoying at best, and absolutely can be dangerous. [0]https://jalopnik.com/congratulations-mini-you-made-the-stupidest-turn-sign-1847727385 https://jalopnik.com/congratulations-mini-you-made-the-stupi...
latchkey 3y ago
I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the internal auto update process. This ran in CI and would fail the build if it didn't pass. While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.
- aaronbeekay 3y ago
  As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.
  - bozhark 3y ago
    What's the priority then, telemetry data? Why is it rough out there?
    - reactordev 3y ago
      Probably due to fires, failures, and fatigue.
      - firtoz 3y ago
        Games have AAA, autos have FFF
    - jacquesm 3y ago
      Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware. I hope I didn't miss anything but I wouldn't be surprised if I did.
FireBeyond 3y ago
As annoying as this, I find this laughable, too. Rivian updated users on the situation. Then, whines Electrek: > That’s the last update we had over 10 hours after Rivian customer vehicles were fed the bad software update. "Over 10 hours"! I suppose it isn't Tesla, who yeets updates over the fence, that break new things, yeets another update that fixes that problem but introduces another one, then reverts back to two versions prior, before the issue. The Tesla that gets firmware fixes from vendors that have a test harness that should take 36+ hours to run, but says YOLO and flashes it onto a random car they have lying around and emails the vender back 3 hours later saying "LGTM, WFM, thanks!"
wnevets 3y ago
This maybe crazy but if you're writing software for hardware that cost tens of thousands of dollars it should be impossible to brick it with an update, especially if that update is OTA. The future is going great.
fsckboy 3y ago
I wish the economics of mass production didn't turn pennies into millions that need to be eliminated, because I've always thought the "don't disconnect from power" and "update bricks it" type problems could be solved by having extra EPROM to download into, the way linux keeps the previous kernel around after an update. Or at least the ability to re-init/download from scratch, like a borked macbook disk. And hey, not the extra ability to do that, make it "the way it works" so you're always testing it.
j45 3y ago
As long as they are good for fixing it, this might what being a Pioneer or Early Adopter is about.
gunapologist99 3y ago
This is why I don't really want my car to have any antenna (that receives/interprets code) or receive OTA updates, ever. I'd like to please force any attackers to at least be within 50 feet of my TPMS, instead of being literally anywhere on the planet. A car doesn't need data updates, and definitely not code updates[1] 1. source: every car built in previous century.
- pard68 3y ago
  My insistence on only driving cars made prior to 2005 keeps making more and more sense. (2005 is just an arbitrary date I settled on, nothing significant about it)
  - deleted 3y ago
    [deleted]
- sbehere 3y ago
  > A car doesn't need data updates, and definitely not code updates I don't think this is accurate. Many advanced driving assistance capabilities need access to updated map tiles, which is a data update. They may need code updates to fix errors or shortcomings that can be detected only after deployment on extensive fleets or in response to changes to the environment/infrastructure. This is just one example for why data and code updates are needed. I think it is more accurate to say that a "dumb" car with mostly electro-mechanical systems doesn't need data updates and definitely not code updates. But that isn't true for vehicles built within the last few years and definitely untrue for vehicles that will be built in the coming years.
  - gunapologist99 3y ago
    > Many advanced driving assistance capabilities need access to updated map tiles Your phone (or GPS or even a paper map) can guide you; none of the following need access to map tiles: * forward collision warning * automatic emergency braking * lane departure warning * adaptive cruise control * blind spot detection * stability control > code updates to fix errors or shortcomings That's what recalls and TSBs have traditionally been for, and the driver can refuse them if desired. I mean, actual lives are at stake here. Would we (or should we) allow 737's to get OTA updates? Of course not. The target is too valuable and surface area too vast to adequately protect it.
shoelessone 3y ago
Honestly this makes me feel good, just because it always worries me that I don't see this type of issue being resolved more often. having to physically bring in a car seems like a near worse cast situation but it's good to keep this in our minds as a possibility.
Lightbody 3y ago
I have preorders in for the R1S, the Volvo EX90, and the Kia EV9. I passed once already on buying the R1S when they had one in town available for immediate purchase, simply because they refuse to adopt CarPlay. This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlay, despite their statements otherwise. I suspect the EX90 will be what I land on eventually.
- p_j_w 3y ago
  >This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlat, I have complete faith that, 5 and maybe even 10 years from now, no auto maker will have delivered anything that can compete with either CarPlay or Android Auto. The fact that an auto maker thinks they can do better is a sign of a really high level of either arrogance or outright greed. Complete deal breaker.
collsni 3y ago
OTA on a car. What could go wrong?
sitzkrieg 3y ago
i cant believe this sort of stuff is acceptable. what a clown industry
fhub 3y ago
I'm going to have a chuckle next time I pass the Databricks billboard on 101 in San Francisco "Rivian powered by Databricks" or something to that tune.
sarchertech 3y ago
Miku baby monitors deployed an automatic firmware update that bricked nearly every monitor in use, but not for nearly a month after the update. It forced the company into bankruptcy because they had to replace all of them.
bfrog 3y ago
It’s funny I was just talking to someone about a-b images slots and boots the other day and how they had written this test suite because there were so many potential places where partial updates could be interrupted. Thousands of test points having to be verified was my understanding. That’s before even getting to the confirmed boot/watchdog aspect. What a hassle, hope they like spending money on labor because it sounds like they are going to need to.
reneberlin 3y ago
When will humans be crazy enough to update the firmware of artificial hearts OTA? Updating cars with new features OTA, even "just" an Infotainment can possibly cost lives, because the driver might get confused and isn't putting eyes on the streets. It should be forbidden and every change should be made clear to the driver, shown in detail, and should need verification twice before being accepted. There must not be any kind of surprise in a car for the driver. It should even be possible to skip an update or stop updating at all.
- rekoil 3y ago
  Not updating cars OTA (yes, even "just" the infotainment) can potentially cost lives as well, as security holes would not get patched until the next service appointment.
nunez 3y ago
/r/Rivian is a class act. I expected a wall of screaming, but instead entered a relatively calm room. People are upset, but there's no seething or flamewars, which is kind-of surprising given the cost of these trucks ($80k+, Range Rover territory).
- M3L0NM4N 3y ago
  I think the reason is because they're $80k trucks, not $400/month Tesla leases. Also, they're first generation and I think most of the buyers understand that.
AndrewKemendo 3y ago
Whomever makes the first affordable, tight tolerance electric car that doesn’t spy on you and doesn’t need special care will win the market
teeray 3y ago
An interesting thought experiment: what happens when these vehicles are out of warranty, and automakers accidentally send a vehicle-bricking OTA update? Isn’t that property damage?
- jacquesm 3y ago
  This has happened to some Apple hardware, they fixed it for free in some cases but stiffed others: https://discussions.apple.com/thread/253315438 https://discussions.apple.com/thread/253315438 With the mandatory mobile phone updates for a few years you're definitely going to see a lot more cases like that. A thread about Tesla directly related to your question: https://teslamotorsclub.com/tmc/threads/wholl-be-responsible-for-an-update-bricking-your-car-after-warranty-expires.38045/ https://teslamotorsclub.com/tmc/threads/wholl-be-responsible...
whoopsie 3y ago
Ah this is why CarPlay isn’t worth adding, right?
thumbsup-_- 3y ago
This is the new world we will be living in where you enter your car, only to find that something is broken because of OTA. While updates causing some bugs is ok in my phone but I don't want any bugs in my car. What happens if it messes up with safety systems? or what happens if OTA breaks my car that is out of warranty now? May be I'm the only one that is missing stable software in cars that once vetted, just keeps running as-is if nothing around it is ever changed (ideal scenario for an offline car).
immy 3y ago
That’s funny, I just saw a job posting for Rivian Infotainment team
Am4TIfIsER0ppos 3y ago
Look at all these commenters saying "code signing was done wrong" when the wrong part is code signing at all.
b20000 3y ago
“we use leetcode to filter out hires because it works for us”

277 points by carlivar 3y ago

This is a bit of a nightmare scenario and why when remote updating, you always test update to your own fleet first. Always.

kevin_nisbet 3y ago
Yes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.
toddmorey 3y ago
It sounds like it was tested on their own fleet but they accidentally pushed the wrong bits when deploying the update more widely out to customers.
- carlivar 3y ago
  Maybe they should have an additional phase between test deploy and customers such as "employee personal vehicles".
- eschneider 3y ago
  The usual "best practice" thing for IoT deploys, is to deploy to "your" devices, what for everyone to go green, then allow that build to deploy more widely. In a well-functioning system, it shouldn't be possible to swap bits between those stages. But who knows what these guys were doing. :/
  - bink 3y ago
    Reading between the lines of their public comments it sounds like they did run a full test through their test fleet, their employees, and then were rolling out to customers when the promotion process was "fat fingered". Maybe someone accidentally promoted the wrong release version.

baz00 3y ago

This is why I have a Dumbcar connected to a Smartphone via bluetooth.

barbazoo 3y ago
That's why _I_ have a Dumbcar connected to a smartphone via FM for audio only :)
- thatguy0900 3y ago
  I've tried that before, but it sounded terrible. What dongle do you use?
  - barbazoo 3y ago
    I mean, totally, it doesn't sound great but that's all I can do, my car doesn't have bluetooth :)
    - BobaFloutist 3y ago
      Does it not have an aux port? USB C to Aux cables are pretty cheap and simple.
      - barbazoo 3y ago
        I know I know but then there's a cable from where the phone is to the middle console where the aux port is. But true, it'd for sure be better sound quality.
        ska 3y ago
        > I know I know but then there's a cable from where the phone is to the middle console where the aux port is. This doesn't seem unmanageable, no?
        3y ago

qmarchi 3y ago

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

mlyle 3y ago
The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).
- MichaelZuo 3y ago
  The 'early release tests' weren't testing an identical copy of the actual update?
  - DannyBee 3y ago
    It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software. Or something like this. It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update. In other words, i would not assume they are idiots.
    - AlotOfReading 3y ago
      It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.

adolph 3y ago

The vehicles are drivable but software and displays go black. It appears that the 2023.42 software update hangs at 90% on the vehicle screen or 50% on the app screen and then the vehicle screens black out. All systems appear to still work except for the displays. This is what I do with my Prius to get a comfortably distraction-free driving environment. Sounds like a feature not a bug.

altairprime 3y ago
Technically, the NTSB could order an immediate recall for all Rivian vehicles due to this issue, as the disabled defroster controls are a critical safety issue in cold and/or humid environments. Tesla was forced to issue a recall notice over the controls being buried in a menu; Rivian’s ”defroster unavailable during driving due to manufacturer error” is far worse — especially given the mass and torque of their vehicles, relative to unarmored road users.
sturza 3y ago
Instrument cluster display going black is a functional safety/QM issue. No blinker, transmission direction, speed etc confirmations.
- bri3d 3y ago
  It looks like they correctly isolated the safety critical components on the instrument cluster and they are still functional without infotainment: https://twitter.com/RivianSoftware/status/1724438049675739626 https://twitter.com/RivianSoftware/status/172443804967573962...

wannacboatmovie 3y ago

Interesting to note that Ford's approach of updating software is far more conservative and car-like. It can be done fully offline via USB, but requests that you kindly upload the log files written to the memory stick back to them when complete, in the instructions as a necessary step. Presumably so they can track and stop incidents like this before they happen fleet-wide. Rivian seems more like a "ship it and we'll fix it in the next sprint!" company. How do other manufacturers handle updates?

sturza 3y ago
A/B partitions
- barryrandall 3y ago
  The last time I built something like that, it used partition 1 for the current version, 1 for the last version, 1 with the as-shipped version, and 1 that could restore A or B from the internet or USB.
post_break 3y ago
Fords approach is flawed however. You can still update sync with a bad update and bork it over usb. Ask me how I know.
- r00fus 3y ago
  Pray tell, how painful was your discovery?
  - post_break 3y ago
    Thankfully I was able to repackage it again and flash an upgrade and it unborked itself.

thrill 3y ago

Inexcusable, really.

cs702 3y ago

It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's neither easy nor cheap. All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates!

Aurornis 3y ago
> (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. Not really. Vehicle computers aren’t vastly different on every model year and every trim level or option package. These parts are standardized, tested, and carried across model years. Even with changes, the teams would be expected to have the different variants in their development and test cycles. The 2020, 2021, and 2022 model infotainment systems likely share a lot more in common than an iPhone 13, iPhone 14, and iPhone 15 with all of the non-Pro, Pro, and Max variants.
wannacboatmovie 3y ago
This isn't a bunch of Windows PCs home-built from a hodgepodge of components. They designed, built, and shipped all the hardware. There is ABSOLUTELY NO excuse for not having a database of the exact hardware configs by serial number. They have the ability to test every single shipped configuration. If they don't, they have already failed as a car company.
- AlotOfReading 3y ago
  I guarantee they have a database with the hardware configs. It's required by NHTSA to do recalls and notices. They'll undoubtedly be using that to inform the right people to come in. The update servers almost certainly don't talk to that system though.
xgbi 3y ago

samsquire 3y ago

This is actually a topic that I think about from time to time: how to do aggressive changes to software while they are running. In Ruby world you have monkeypatching. And Linux kernel has livepatching. For example, if you have a distributed system and you want to upgrade a component that every caller uses: you have a large exercise on your hands where you might have to roll out a change over time and then clean up your incremental branches where you have to handle two control flow paths through the code. It reminds me of Google's protobuf required field discussions. It reminds me of repository-per-microservice and a Java library that other microservices use and updating a dependency and having to deploy the change to every service. It's like trying to change wheels on a car while the car is moving or refueling a jet in flight. Unison lang is trying to solve this problem I think, by allowing multiple versions of a function to be available. https://www.unison-lang.org/ https://www.unison-lang.org/ Migrations in databases are painful too. One solution I've thought of which is probably overengineered is that API call sites are an abstract object and their schema and arguments is centrally deployed, I called this "protocol manager". The idea is you write all your code to use a "span" and have contextual data in a span, and you can include or exclude data in a span with a non-software rollout. Your communication schema of RPC and API calls is a runtime decided thing, not hardcoded. If you have N deployed versions of code and you want to upgrade to X, you have to test 1..N to X versions. So nobody does that.

jbott 3y ago
You might be interested in learning about Erlang – it supports hot code reloads natively: https://oozou.com/blog/understanding-elixir-otp-applications-part-4-hot-swapping-modules-151 https://oozou.com/blog/understanding-elixir-otp-applications...
fabianlindfors 3y ago
The database aspect of this problem is particularly interesting to me. I’ve previously built Reshape [0], a zero-downtime migration tool for Postgres, and am now working on ReshapeDB [1], a full database designed from the ground up to tackle this problem. [0] https://github.com/fabianlindfors/reshape https://github.com/fabianlindfors/reshape [1] https://reshapedb.com https://reshapedb.com

1970-01-01 3y ago

Lexus did the very same thing about 8 years ago: https://www.consumerreports.org/lexus/what-to-do-if-your-lexus-infotainment-system-crashed/ https://www.consumerreports.org/lexus/what-to-do-if-your-lex...

scardycat 3y ago

Bringing CI/CD mindset to cars is probably not a great idea. Software updates to commuter vehicles should have a high bar for operational standards, and a simple thing such as an expired certificate should have never been deployed. Having isolated networks in vehicles helps but doesn't prevent broken updates from, eventually, bricking the cars.

nomel 3y ago
I think this shows more of a fundamental flaw in their update mechanism, than anything. I don't think a botched update is a big deal. It happens, and should be expected, in a sane design. The fact that the customer noticed is a big deal. There are many implementations that could be used for an "auto rollback" feature. They either failed to implement that in a sane way, or they were goobers, and assumed things would always be rosy.
- babypuncher 3y ago
  I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?
  - BuckRogers 3y ago
    It doesn’t. People and these tech companies are tools. And do it largely in search of ways to take more of your money. It’s not a favor.
    - theandrewbailey 3y ago
      If ICE tech was the hot new thing in cars, things like spark plugs would have a chip so that it would fire n times then break, but don't worry, there's a subscription for new ones and they will be automatically ordered when the car says so. If the credit card on file expires, your spark plugs won't work anymore, even if you just replaced them.

qudat 3y ago

What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.

nomel 3y ago
> where a "bug" has potentially life threatening consequences. What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence. “The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...” Also, you can't do an update while driving.
- jawns 3y ago
  Based on the photo included in the article, what they're calling an infotainment system is actually two separate components, one of which appears to be taking the place of a traditional dashboard. If that's the case and there's no other way to monitor speed, fuel levels, engine temperature, warning lights, etc., I'd say that's quite a bit more worrisome than just not being able to play your favorite music while driving.
  - nomel 3y ago
    I understand, but the risk of life wouldn't be from the bug, it would be from conscious choice of driving without a speedo. There's a critical distinction there. In this case, mileage/battery are still present, and I would assume safety critical warnings would still be displayed.
    - jonplackett 3y ago
      If you need thr car in an emergency and it's unusable, that almost counts.
      - thrill 3y ago
        "almost" only counts in horseshoes and nukes.

gravitronic 3y ago

I used to work for a company that built satellite receivers that would be installed in all sorts of weird remote environments in order to pull radio or tv from satellite and rebroadcast locally. If we pushed a broken update it might mean someone from the radio company would have to make a trip to go pull the device and send it to us physically. Our upgrader did not run as root, but one time we had to move a file as root.. so I had to figure out a way to exploit our machine reliably from a local user, gain root, and move the file out of the way. We'd then deploy this over the satellite head end and N remote units would receive and run the upgrade autonomously. Fun stuff. Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick.

nomel 3y ago
> Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick. No offense, but what a shit show. It makes me assume no source control, and a really good chance that state actors made their way into your network/product. This almost happened at a communication startup I know, with three letter agencies helping resolve it. State actors really like infiltrating communication stuffs.
- gravitronic 3y ago
  oh, yeah, this place was a total shit show. BUT we were ISO9001 certified!! So we had source control (CVS) and a Process (with a capital P) to follow. In this case that code was added in a previous development iteration because someone needed to run something as root when a user pressed a certain button on the LCD panel in front and this was the decoupled solution they wrote intentionally. Somehow I feel like that makes it worse than if it was a malicious three letter agency lol.
  - nomel 3y ago
    Package this up and send it to https://thedailywtf.com https://thedailywtf.com It's beautiful.

martin8412 3y ago

Stuff like this is why I don't want OTA updates in my cars. Let the car dealership deal with it during regular maintenance. They'll be on the hook for fixing it before handing the car back to me.

ben_jones 3y ago
But what if your car doesn’t have the latest emojiset or social sharing functionality within the notes app?
galangalalgol 3y ago
Don't even need OTA. A seattle radio station bricked a bunch of mazdas. https://www.autoblog.com/2022/02/09/seattle-radio-station-breaks-mazda-infotainment-systems/ https://www.autoblog.com/2022/02/09/seattle-radio-station-br...
- cozzyd 3y ago
  Amazing. Can't wait for some car software stack to be so poorly designed that an FM transmitter can remote takeover.
  - galangalalgol 3y ago
    It quite possibly could have with a well formed digital fm payload.
    - cozzyd 3y ago
      We'll hopefully no car manufacturer is dumb enough that the FM radio software can affect vehicle control functions but I wouldn't be surprised ..
      - aftbit 3y ago
        I'm pretty sure they are all dumb enough for this. This is why people can steal (modern) cars by ripping a headlight out and accessing the CAN bus through the gap. https://kentindell.github.io/2023/04/03/can-injection/ https://kentindell.github.io/2023/04/03/can-injection/ AIUI the military uses "CAN bus firewalls" to prevent this. There is also some sort of encrypted/authenticated CAN bus protocol in the works. Neither are common in production cars as of 2022 or so, though they may be soon enough.

nicholasjarnold 3y ago

Is it possible, as a licensee of the Rivian vehicle system, to disable the automatic OTA updates without having expert-level knowledge or tooling? Also, yes, I'm specifically avoiding using the word "owner" above for obvious reasons.

55873445216111 3y ago
Rivian "licensee" here. So far all updates have required you to press a button (in the car or on the app) to launch the update installer. Not sure how many weeks you can ignore it for as I never tried.
bo1024 3y ago
Confirming that updates are not automatic, and can be ignored indefinitely. For now.

deleted 3y ago

[deleted]

kevinventullo 3y ago

My 2019 car is not connected to the internet. Instead, I use Apple CarPlay for everything. Is there any reason not to do it this way?

gmane 3y ago
Then automakers can’t sell your information for fat stacks of cash.
- lotsofpulp 3y ago
  This, and automakers do not want to become dumb appliances (even though they are for 90% of people).
- pookha 3y ago
  ..And authoritarian governments can't take complete custody of your movements. Can't shutdown your car either. Seems to be a win-win for both sides (private and public industry). https://qz.com/1522309/how-chinas-electric-car-surveillance-system-works https://qz.com/1522309/how-chinas-electric-car-surveillance-... https://apnews.com/article/4a749a4211904784826b45e812cff4ca https://apnews.com/article/4a749a4211904784826b45e812cff4ca
ChadMoran 3y ago
Quite a few reasons but primarily CarPlay doesn't know about your vehicle's charge and road trip navigation requires external inputs to determine your charge route. Apple could potentially offer an API to have "reverse" CarPlay where the car's app can feed information into iOS. I recently rented an Mercedes EV which had Apple CarPlay and it was a weird experience having to manage two sets of experiences.
- callalex 3y ago
  The exact API you describe already exists in CarPlay.
  - 3y ago

nicolaslem 3y ago

This is the kind of thing that keeps my awake at night. Does anyone here have some practical tips to turn an embedded Linux machine into an appliance? The kind of system that a botched update cannot brick but only momentarily disable until a non-technical user presses a factory reset button of some sort.

elitepleb 3y ago
A/B updates as implemented in android, https://bootlin.com/pub/conferences/2022/elce/opdenacker-implementing-A-B-system-updates-with-u-boot/opdenacker-implementing-A-B-system-updates-with-u-boot.pdf https://bootlin.com/pub/conferences/2022/elce/opdenacker-imp...
hospitalJail 3y ago
>Does anyone here have some practical tips to turn an embedded Linux machine into an appliance? Lol I suppose this is the negative about having sensors that make sure water gets hot enough to be sanitizing, but not so hot that it wastes energy. And I'm sure you can imagine 100 other uses of having a microcontroller/CPU process data and do feedback. (I'm sure there are EE only ways of doing it, but theoretically possible and useful are two different thigs)

ct0 3y ago

Will insurance carriers cover damages due to botched updates? Imagine 10 years from now the power/control that electric delivery companies would have over retailers like amazon. One botched update away from a complete backup for delivery vans.

eigenvalue 3y ago

Can’t imagine how much it would suck to be the engineer who fat fingered it and caused a huge crisis for the company, inconveniencing tons of customers and costing millions. Even if there should be processes in place to prevent it in the first place, you’d still know you were the “but for cause” of the problem.

xyst 3y ago

Looks like this car brand is circling the drain. Glad I never bought into the hype.

seattle_spring 3y ago
It’s circling the drain because of one bad software update? Sounds more like you’ve just bought into the doom and gloom that a few specific news outlets have been pushing.

karaterobot 3y ago

What kinds of changes are generally included in these over the air updates? I have this sudden urge to shake my fist at a cloud and tell the gods that cars shouldn't need updates in the first place, if the car was ever deemed ready for production and then sold to customers for money. But, maybe I'm wrong, and it makes perfect sense. All I can think of would be something like a periodic update to navigation data, is that it?

ezfe 3y ago
It’s possible to deem software ready to sell but find improvements later. Simple example: my Subaru was sold to me with an interesting design decision that caused the radio to come on whenever the car was started. This was not a bug. Every Subaru worked this way for years. A year into ownership I received an OTA update that added a “not playing” state on startup. This was never a safety issue and was likely not a defect. It was, however, stupid and needed to be changed.
- karaterobot 3y ago
  I wish my Mazda had this option! But I would still say that I'd expect them to have included this option before selling the car, especially since radios and user preferences around radio UI are pretty well established.
  - ezfe 3y ago
    Sure, but they didn't - and it's not because the software is complicated. The Subaru headunits are very basic (and more-so a few years ago). When they make an improvement, I like getting the trivial things on my older vehicle. It's better for everyone involved, so long as they do it responsible.

cryptoegorophy 3y ago

Tesla updates are sent in batches and you can opt in for advanced updates I guess to be earlier. Normally when I see that there is an update on Reddit then it takes 1-2 weeks at least to get to my car with the “advanced” updates on.

xyst 3y ago

Tesla. Rivian. All cut from the same cloth. A car should be simple. Yet we are stuffing all of this tech junk into it and trying to repackage is as something else to pump the numbers. Car companies suck at tech. Let’s be realistic. They should stay their lane and focus on improving the car and physical aspects (safety, reducing carbon output, longevity, ease of repairability, reducing supply chain issues)

bhauer 3y ago
> Tesla. Rivian. All cut from the same cloth. I'm not aware of any Tesla OTA updates bricking the infotainment system. At least since I've been paying attention. I don't see them quite as similar as you suggest.
- margalabargala 3y ago
  There have been plenty. https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_tesla_model_3_bricked_after_update/ https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_te... https://teslamotorsclub.com/tmc/threads/failed-software-update-bricked-my-2020-x-today.258670/ https://teslamotorsclub.com/tmc/threads/failed-software-upda... Two examples of many. I'm not aware of any fleet-wide issues that accidentally bricked Teslas, but as one-offs they do happen; and unlike this Rivian update, a botched Tesla OTA generally leaves the car undriveable and needing to be towed. These Rivians will at least still drive, as long as you don't need fancy extraneous luxury features like a...speedometer.

avereveard 3y ago

> In most cases, the rest of the vehicle systems are still operational Like what do you mean "in most cases" I can understand a broken infotainment needing reset but imagine if you had to tow your truck I'd be furious.

Havoc 3y ago

> the vehicle is not bricked What a time to be alive. Software updates (almost) turning cars into paper weights lol

emmelaich 3y ago

Poor title; physical repair is not required. Physical presence is required.

Someone1234 3y ago
The article doesn't really state what is required to repair the vehicle. I'd assume if it was as simple as loading a flash drive and plugging it in, then Rivian would have provided a way for customers to self-fix. The second a single body panel is removed to gain access to the headunit, it is a physical repair. So without more info we cannot know if it is accurate or not.
- emmelaich 3y ago
  I don't think many people would consider removing a body panel to be a physical repair. I think the term is 'back to base' or similar. Physical repair suggests e.g. a burnt out capacitor

WirelessGigabit 3y ago

What's the impact on your insurance should you get into an accident? The speedometer screen is gone, so does that not imply the vehicle is inherently unsafe to drive?

MisterTea 3y ago

Can I please just buy a car with a motor and battery? Why does every god damn vehicle have to come littered with screens and chips all together like some tentacle monster? All I need is a gauge cluster screen that can display the normal info like stored and heading while also letting me configure the cars performance and safety features. Then let me mount a double DIN radio that isn't dog shit. I've not seen a single new car with these dumb screens with a sound system that's not tinny muddy garbage with zero adjustment save for "bass" and "treble" settings. I mean all that technology and you can't be assed to put an eq in there. HVAC never needed more than two or three knobs anyway.

easylion 3y ago

need a easy way to do restore to previous version offline. take 100 bucks extra if required to have a backup ssd. Don’t want to be camping and then realizing i’m stuck because of some junior dev not being competent enough

seattle_spring 3y ago
Why would you intentionally upgrade your vehicle software while camping? It’s not like this stuff installs automatically, you have to explicitly accept the installation. Waiting a few days or even a few weeks before hitting “install” is completely normal.

ralmidani 3y ago

Move fast and break things that move fast… I don’t really like or trust most (if not all) of the established automakers, but there is something to be said for having several decades (over a century in some cases) of experience building potential killing machines vs. a company that’s not even 15 years old. The established players have put out cars which suffered freak malfunctions, but Rivian (and Tesla) seem to be struggling more with QA. Non-rhetorical question: do companies have safeguards for critical components like braking systems, or are they also prone to catastrophic failure if a software engineer pushes a bad commit?

ezfe 3y ago
The moving fast components were unaffected by this issue…
- ralmidani 3y ago
  I know, I just thought it was a decent pun.

Someone1234 3y ago

I wonder if the way Microsoft's XBox is designed may be something to look towards in terms of hardware reliability/fallback. Specifically they utilize a Hypervisor which rarely needs updates, running different operating environments which need frequent updates. - Better isolation of different parts of the system (e.g. infotainment unit, instrument cluster, et al). - Better isolation for updates (e.g. run a "beta" update, and a "stable" update side-by-side). - Automatic error detection and rollback (e.g. if a VM keeps restarting after an update). - Ease of offering features like rollbacks to end-users. - Rare hypervisor updates can be held to a much higher standard relative to other VM updates. The only downside of hypervisor-based systems is slightly higher hardware costs. But even that is largely mitigated by modern architectures that natively support virtualization. PS - You can also look to any containerization. I specifically brought up the XBox because it is a hardware product, just like a vehicle.

antoniuschan99 3y ago

Wondering why there isn’t an option for a factory reset (eg. press and hold with a paperclip for 10 seconds)

glonq 3y ago

As somebody who has spent many years doing embedded+iot related to remote fleet firmware updates, this is the kind of thing that lurks in my nightmares. I'd love to be a fly on the wall at Rivian engineering/operations this week!

janitor61 3y ago

This is tangentially Rivian related, but does anyone else see the inherent danger of stylized tail lights that are just a single red bar across the back of the car? Travelling on the freeway at night I can't really gauge the distance to the car in front of me if it's far ahead and if there's no discernable left and right brake lights. I'd believe Rivians and other cars like that are more at risk of high speed rear-end collisions.

deleted 3y ago
[deleted]
rurp 3y ago
This reminds of the terrible turn signals Mini used, which look like flashing arrows pointing in the opposite direction of the turn[0]. Getting cute with basic stuff like tail lights is forgettable or annoying at best, and absolutely can be dangerous. [0]https://jalopnik.com/congratulations-mini-you-made-the-stupidest-turn-sign-1847727385 https://jalopnik.com/congratulations-mini-you-made-the-stupi...

latchkey 3y ago

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the internal auto update process. This ran in CI and would fail the build if it didn't pass. While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.

aaronbeekay 3y ago
As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.
- bozhark 3y ago
  What's the priority then, telemetry data? Why is it rough out there?
  - reactordev 3y ago
    Probably due to fires, failures, and fatigue.
    - firtoz 3y ago
      Games have AAA, autos have FFF
  - jacquesm 3y ago
    Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware. I hope I didn't miss anything but I wouldn't be surprised if I did.

FireBeyond 3y ago

As annoying as this, I find this laughable, too. Rivian updated users on the situation. Then, whines Electrek: > That’s the last update we had over 10 hours after Rivian customer vehicles were fed the bad software update. "Over 10 hours"! I suppose it isn't Tesla, who yeets updates over the fence, that break new things, yeets another update that fixes that problem but introduces another one, then reverts back to two versions prior, before the issue. The Tesla that gets firmware fixes from vendors that have a test harness that should take 36+ hours to run, but says YOLO and flashes it onto a random car they have lying around and emails the vender back 3 hours later saying "LGTM, WFM, thanks!"

wnevets 3y ago

This maybe crazy but if you're writing software for hardware that cost tens of thousands of dollars it should be impossible to brick it with an update, especially if that update is OTA. The future is going great.

fsckboy 3y ago

I wish the economics of mass production didn't turn pennies into millions that need to be eliminated, because I've always thought the "don't disconnect from power" and "update bricks it" type problems could be solved by having extra EPROM to download into, the way linux keeps the previous kernel around after an update. Or at least the ability to re-init/download from scratch, like a borked macbook disk. And hey, not the extra ability to do that, make it "the way it works" so you're always testing it.

j45 3y ago

As long as they are good for fixing it, this might what being a Pioneer or Early Adopter is about.

gunapologist99 3y ago

This is why I don't really want my car to have any antenna (that receives/interprets code) or receive OTA updates, ever. I'd like to please force any attackers to at least be within 50 feet of my TPMS, instead of being literally anywhere on the planet. A car doesn't need data updates, and definitely not code updates[1] 1. source: every car built in previous century.

pard68 3y ago
My insistence on only driving cars made prior to 2005 keeps making more and more sense. (2005 is just an arbitrary date I settled on, nothing significant about it)
- deleted 3y ago
  [deleted]
sbehere 3y ago
> A car doesn't need data updates, and definitely not code updates I don't think this is accurate. Many advanced driving assistance capabilities need access to updated map tiles, which is a data update. They may need code updates to fix errors or shortcomings that can be detected only after deployment on extensive fleets or in response to changes to the environment/infrastructure. This is just one example for why data and code updates are needed. I think it is more accurate to say that a "dumb" car with mostly electro-mechanical systems doesn't need data updates and definitely not code updates. But that isn't true for vehicles built within the last few years and definitely untrue for vehicles that will be built in the coming years.
- gunapologist99 3y ago
  > Many advanced driving assistance capabilities need access to updated map tiles Your phone (or GPS or even a paper map) can guide you; none of the following need access to map tiles: * forward collision warning * automatic emergency braking * lane departure warning * adaptive cruise control * blind spot detection * stability control > code updates to fix errors or shortcomings That's what recalls and TSBs have traditionally been for, and the driver can refuse them if desired. I mean, actual lives are at stake here. Would we (or should we) allow 737's to get OTA updates? Of course not. The target is too valuable and surface area too vast to adequately protect it.

shoelessone 3y ago

Honestly this makes me feel good, just because it always worries me that I don't see this type of issue being resolved more often. having to physically bring in a car seems like a near worse cast situation but it's good to keep this in our minds as a possibility.

Lightbody 3y ago

I have preorders in for the R1S, the Volvo EX90, and the Kia EV9. I passed once already on buying the R1S when they had one in town available for immediate purchase, simply because they refuse to adopt CarPlay. This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlay, despite their statements otherwise. I suspect the EX90 will be what I land on eventually.

p_j_w 3y ago
>This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlat, I have complete faith that, 5 and maybe even 10 years from now, no auto maker will have delivered anything that can compete with either CarPlay or Android Auto. The fact that an auto maker thinks they can do better is a sign of a really high level of either arrogance or outright greed. Complete deal breaker.

collsni 3y ago

OTA on a car. What could go wrong?

sitzkrieg 3y ago

i cant believe this sort of stuff is acceptable. what a clown industry

fhub 3y ago

I'm going to have a chuckle next time I pass the Databricks billboard on 101 in San Francisco "Rivian powered by Databricks" or something to that tune.

sarchertech 3y ago

Miku baby monitors deployed an automatic firmware update that bricked nearly every monitor in use, but not for nearly a month after the update. It forced the company into bankruptcy because they had to replace all of them.

bfrog 3y ago

It’s funny I was just talking to someone about a-b images slots and boots the other day and how they had written this test suite because there were so many potential places where partial updates could be interrupted. Thousands of test points having to be verified was my understanding. That’s before even getting to the confirmed boot/watchdog aspect. What a hassle, hope they like spending money on labor because it sounds like they are going to need to.

reneberlin 3y ago

When will humans be crazy enough to update the firmware of artificial hearts OTA? Updating cars with new features OTA, even "just" an Infotainment can possibly cost lives, because the driver might get confused and isn't putting eyes on the streets. It should be forbidden and every change should be made clear to the driver, shown in detail, and should need verification twice before being accepted. There must not be any kind of surprise in a car for the driver. It should even be possible to skip an update or stop updating at all.

rekoil 3y ago
Not updating cars OTA (yes, even "just" the infotainment) can potentially cost lives as well, as security holes would not get patched until the next service appointment.

nunez 3y ago

/r/Rivian is a class act. I expected a wall of screaming, but instead entered a relatively calm room. People are upset, but there's no seething or flamewars, which is kind-of surprising given the cost of these trucks ($80k+, Range Rover territory).

M3L0NM4N 3y ago
I think the reason is because they're $80k trucks, not $400/month Tesla leases. Also, they're first generation and I think most of the buyers understand that.

AndrewKemendo 3y ago

Whomever makes the first affordable, tight tolerance electric car that doesn’t spy on you and doesn’t need special care will win the market

teeray 3y ago

An interesting thought experiment: what happens when these vehicles are out of warranty, and automakers accidentally send a vehicle-bricking OTA update? Isn’t that property damage?

jacquesm 3y ago
This has happened to some Apple hardware, they fixed it for free in some cases but stiffed others: https://discussions.apple.com/thread/253315438 https://discussions.apple.com/thread/253315438 With the mandatory mobile phone updates for a few years you're definitely going to see a lot more cases like that. A thread about Tesla directly related to your question: https://teslamotorsclub.com/tmc/threads/wholl-be-responsible-for-an-update-bricking-your-car-after-warranty-expires.38045/ https://teslamotorsclub.com/tmc/threads/wholl-be-responsible...

whoopsie 3y ago

Ah this is why CarPlay isn’t worth adding, right?

thumbsup-_- 3y ago

This is the new world we will be living in where you enter your car, only to find that something is broken because of OTA. While updates causing some bugs is ok in my phone but I don't want any bugs in my car. What happens if it messes up with safety systems? or what happens if OTA breaks my car that is out of warranty now? May be I'm the only one that is missing stable software in cars that once vetted, just keeps running as-is if nothing around it is ever changed (ideal scenario for an offline car).

immy 3y ago

That’s funny, I just saw a job posting for Rivian Infotainment team

Am4TIfIsER0ppos 3y ago

Look at all these commenters saying "code signing was done wrong" when the wrong part is code signing at all.

b20000 3y ago

“we use leetcode to filter out hires because it works for us”