5 ms·

It’s a bit troubling that automakers (and, well, anyone who is building things that have mechanisms for software updates) aren’t making software updates failsaf

It’s a bit troubling that automakers (and, well, anyone who is building things that have mechanisms for software updates) aren’t making software updates failsafe. If you have multiple copies of the image on storage, and you take care not to touch the currently-running image, then a failed software update to the alternate image shouldn’t have any impact. Think of it like blue-green deployments on the cloud; and it’s what TiVo was doing with software updates to their DVRs all the way back in the late 1990s.

variadix 3y ago
My router has dual firmware partitions, the fact a car doesn’t is inexcusable
- KennyBlanken 3y ago
  It should be mandated, but corporations dominate legislation and regulations. "You'll drive up costs for the hard-working American consumer" "We'd have to cut salaries or fire factory workers" "We wouldn't be able to compete against that car company from Nothereistan!" etc.
- g_p 3y ago
  Absolutely. One fact I recall from many years ago (so I don't know how true it is now) is that auto makers were shipping 7 year old silicon (and thus board support packages for their software). The guys working on chips and CPUs were writing software and it wouldn't hit the road for 7 years. This was why infotainment systems lagged the state of the art for so long - they were apparently shipping components that would have belonged in a smartphone from more than half a decade before. In this case, I believe we had a few phones get A/B updates by Android 7, which was late 2016. Giving it a year or so to filter to more CPU chipsets, that would mean (if the old rule held true), we would expect to see A/B hitting new cars in the next year or two. This assumes auto makers are using enough of the Android BSP to get the bootloader with A/B (which isn't a given), and then that they use it! Anyone with any experience of auto software know if the teams doing software are experienced and understand the platform side of things, or if it's a traditional front-end heavy "crunch release" type affair? I've seen some pretty poor quality infotainment systems in the past, so my assumption is inexperienced devs with limited platforms knowledge, focused on "get it working to ship it", but would be interesting to hear from anyone with experience of how this really goes down.
- justsomehnguy 3y ago
  It was a great advantage, both in marketing and the real usage, when the motherboards started to support the Dual BIOS (GIGABYTE term, AFAIR). 2003 I think?
stefan_ 3y ago
I'm sure the infotainment system does, but that is not all that is being updated; you have many more processors in ECU, assistance systems and so on that are also sometimes getting fed new firmware over various wonky buses as part of an upgrade. In some instances there might even be multiple levels involved in the firmware distribution.
- yjftsjthsd-h 3y ago
  So? Everything that can update firmware should do the same thing.
  - stefan_ 3y ago
    And then what? A/B update isn't magic dust, if some sub component ends up with an incompatible firmware because it had to boot to the alternate partition, you might still have an incompatible overall system. (Not like the subsubsupplier of that module even specced the extra flash memory for A/B update, hah)
    - mistercheph 3y ago
      Retain the known good firmware for each controller on master disk A, if the update fails, reflash all components.
- gpderetta 3y ago
  The real question is why would they be doing OTA upgrades of such critical systems.
cedws 3y ago
This is more commonly known as A/B partitioning and as you say, is standard practice in embedded devices. It is indeed disturbing if automakers have not adopted this, nor a rollback mechanism.
iknowstuff 3y ago
Tesla is. Failed updates are non-destructive, they also have an RTOS gateway isolating entertainment software from the software needed to drive the vehicle. https://www.pentestpartners.com/security-blog/reverse-engineering-tesla-hardware/ https://www.pentestpartners.com/security-blog/reverse-engine... And this is their decade old architecture. If you watch some cybertruck videos you’ll see it is much more elegant now.
- rpmisms 3y ago
  Tesla is very very good at software engineering. They still need to find the balance in their software and process engineering, though.
  - stefan_ 3y ago
    You are talking about the company that wrote logs to the fully read-write filesystem of their infotainment system and killed the EMMC from raw flash cycles within as few as 2 years. Recalled 10 years later.
    - rpmisms 3y ago
      Yes, this happened to me, although it took 10 years. Free replacement, and the car still worked fine.
      - FirmwareBurner 3y ago
        It's still a major embedded engineering oversight on their part. We had fallback developed for such cases in $5 embedded devices. Calling out Tesla as a model for excellent software is insulting to SW that's actually quality. What Tesla SW has and others lack, is a great "wow" factor, which consumers who don't get to see how the sausage is made, mistakenly associate with 'quality' because it behaves like their iPad, but don't know all the skeletons in that closet.
        rpmisms 3y ago
j45 3y ago
It's really interesting that updates can't revert, or enable some limp mode, or disable the offending functionality.
superkuh 3y ago
Anyone on the 'net back in the 1990s will remember the image macro, email forwards, and usenet posts about "If cars were made by software companies". It looks like they're finally coming true.
- soganess 3y ago
  Except the price part. We have an inverse Miley Cyrus situation on our hands.
amelius 3y ago
We should have a law for this. In most countries, new vehicles need to be tested by govt agencies before they are allowed to hit the road. The same should hold after software updates. Ideally, the update should be done by independent agencies, so they can make sure the software is tested and they can properly inform the user (something which we already know companies don't always do). The way it is now, it is all too easy for a company to decide to push a quick fix for a problem that really needs a more careful approach (but which might cost the company money).
- andsoitis 3y ago
  You want an agency to validate every single software updates to vehicles?
  - amelius 3y ago
    At the very least, I want the agency to keep a hold on the update for $N days. That way, I am more convinced that the software engineers of the car manufacturer won't push any quick fixes that might cause bigger problems later. In the case of urgent problems, the correct response is to ground the fleet. This will cost the manufacturer money, but I'm a software engineer for long enough to know that a quick fix won't be a safer solution.
    - throwaway2037 3y ago
      > At the very least, I want the agency to keep a hold on the update for $N days. What is the difference between this and "No Deploy Fridays"?
  - norir 3y ago
    I actually think this isn't a bad idea. I really don't want an automaker treating consumer updates like a developer repeatedly pushing nearly empty commits to github to get ci green.
  - Klonoar 3y ago
    At the very least, if it touches anything that impacts the driving… maybe? It’s a 3 ton (+) machine with people paying attention less each passing day. It should be taken very seriously.

variadix 3y ago

My router has dual firmware partitions, the fact a car doesn’t is inexcusable

KennyBlanken 3y ago
It should be mandated, but corporations dominate legislation and regulations. "You'll drive up costs for the hard-working American consumer" "We'd have to cut salaries or fire factory workers" "We wouldn't be able to compete against that car company from Nothereistan!" etc.
g_p 3y ago
Absolutely. One fact I recall from many years ago (so I don't know how true it is now) is that auto makers were shipping 7 year old silicon (and thus board support packages for their software). The guys working on chips and CPUs were writing software and it wouldn't hit the road for 7 years. This was why infotainment systems lagged the state of the art for so long - they were apparently shipping components that would have belonged in a smartphone from more than half a decade before. In this case, I believe we had a few phones get A/B updates by Android 7, which was late 2016. Giving it a year or so to filter to more CPU chipsets, that would mean (if the old rule held true), we would expect to see A/B hitting new cars in the next year or two. This assumes auto makers are using enough of the Android BSP to get the bootloader with A/B (which isn't a given), and then that they use it! Anyone with any experience of auto software know if the teams doing software are experienced and understand the platform side of things, or if it's a traditional front-end heavy "crunch release" type affair? I've seen some pretty poor quality infotainment systems in the past, so my assumption is inexperienced devs with limited platforms knowledge, focused on "get it working to ship it", but would be interesting to hear from anyone with experience of how this really goes down.
justsomehnguy 3y ago
It was a great advantage, both in marketing and the real usage, when the motherboards started to support the Dual BIOS (GIGABYTE term, AFAIR). 2003 I think?

stefan_ 3y ago

I'm sure the infotainment system does, but that is not all that is being updated; you have many more processors in ECU, assistance systems and so on that are also sometimes getting fed new firmware over various wonky buses as part of an upgrade. In some instances there might even be multiple levels involved in the firmware distribution.

yjftsjthsd-h 3y ago
So? Everything that can update firmware should do the same thing.
- stefan_ 3y ago
  And then what? A/B update isn't magic dust, if some sub component ends up with an incompatible firmware because it had to boot to the alternate partition, you might still have an incompatible overall system. (Not like the subsubsupplier of that module even specced the extra flash memory for A/B update, hah)
  - mistercheph 3y ago
    Retain the known good firmware for each controller on master disk A, if the update fails, reflash all components.
gpderetta 3y ago
The real question is why would they be doing OTA upgrades of such critical systems.

cedws 3y ago

This is more commonly known as A/B partitioning and as you say, is standard practice in embedded devices. It is indeed disturbing if automakers have not adopted this, nor a rollback mechanism.

iknowstuff 3y ago

Tesla is. Failed updates are non-destructive, they also have an RTOS gateway isolating entertainment software from the software needed to drive the vehicle. https://www.pentestpartners.com/security-blog/reverse-engineering-tesla-hardware/ https://www.pentestpartners.com/security-blog/reverse-engine... And this is their decade old architecture. If you watch some cybertruck videos you’ll see it is much more elegant now.

rpmisms 3y ago
Tesla is very very good at software engineering. They still need to find the balance in their software and process engineering, though.
- stefan_ 3y ago
  You are talking about the company that wrote logs to the fully read-write filesystem of their infotainment system and killed the EMMC from raw flash cycles within as few as 2 years. Recalled 10 years later.
  - rpmisms 3y ago
    Yes, this happened to me, although it took 10 years. Free replacement, and the car still worked fine.
    - FirmwareBurner 3y ago
      It's still a major embedded engineering oversight on their part. We had fallback developed for such cases in $5 embedded devices. Calling out Tesla as a model for excellent software is insulting to SW that's actually quality. What Tesla SW has and others lack, is a great "wow" factor, which consumers who don't get to see how the sausage is made, mistakenly associate with 'quality' because it behaves like their iPad, but don't know all the skeletons in that closet.
      - rpmisms 3y ago

j45 3y ago

It's really interesting that updates can't revert, or enable some limp mode, or disable the offending functionality.

superkuh 3y ago

Anyone on the 'net back in the 1990s will remember the image macro, email forwards, and usenet posts about "If cars were made by software companies". It looks like they're finally coming true.

soganess 3y ago
Except the price part. We have an inverse Miley Cyrus situation on our hands.

amelius 3y ago

We should have a law for this. In most countries, new vehicles need to be tested by govt agencies before they are allowed to hit the road. The same should hold after software updates. Ideally, the update should be done by independent agencies, so they can make sure the software is tested and they can properly inform the user (something which we already know companies don't always do). The way it is now, it is all too easy for a company to decide to push a quick fix for a problem that really needs a more careful approach (but which might cost the company money).

andsoitis 3y ago
You want an agency to validate every single software updates to vehicles?
- amelius 3y ago
  At the very least, I want the agency to keep a hold on the update for $N days. That way, I am more convinced that the software engineers of the car manufacturer won't push any quick fixes that might cause bigger problems later. In the case of urgent problems, the correct response is to ground the fleet. This will cost the manufacturer money, but I'm a software engineer for long enough to know that a quick fix won't be a safer solution.
  - throwaway2037 3y ago
    > At the very least, I want the agency to keep a hold on the update for $N days. What is the difference between this and "No Deploy Fridays"?
- norir 3y ago
  I actually think this isn't a bad idea. I really don't want an automaker treating consumer updates like a developer repeatedly pushing nearly empty commits to github to get ci green.
- Klonoar 3y ago
  At the very least, if it touches anything that impacts the driving… maybe? It’s a 3 ton (+) machine with people paying attention less each passing day. It should be taken very seriously.