6 ms·

A/B boot is old, good, and proven, but what do you do when you have dozens of separate microcontrollers, each governing a semi-independent subsystem, all runnin

A/B boot is old, good, and proven, but what do you do when you have dozens of separate microcontrollers, each governing a semi-independent subsystem, all running their own software that needs its own updates? It would make sense from safety/reliability PoV when a crash/bug in a slightly less critical system (like adjusting mirrors) does not affect a slightly more critical system (like indicators). What do you do when updating only one of the components fails? Do you rollback everything? Do you continue with the rest? Just to be clear, I'm firmly in the "less smart = more reliable" camp, but even a SoC resembles a distributed system nowadays, and this architecture has its merits. The GPU on my M1 Mac crashed, rebooted itself, and all I saw was 1-2s of glitched graphics - no application noticed. Of course ideally GPUs wouldn't crash, but the same could be said of cars.

onetimeuse92304 3y ago
I don't think "more smart = less reliable" is inevitable. The goal should be designing it in a way that failure of the more smart features does not spill to critical functionality. Or that you can use it in a degraded state sort of like fly by wire planes do. I have no car design experience, but here is my 5-minute idea: For starters, you build a basic car. It drives, it has basic indicators, it has ABS, etc. It has some APIs that can be connected to but there is limited ability to break the basic car through those APIs. Then you build your smart car around it by adding to the basic car. You add on more computers, electronics, etc. But any time you rip off the smart car functionality, you still have the basic car working. If there is a need to update software, this should really only be needed for the smart car part. But even if it fails completely, the basic car underneath should be functional and you should be able to use the car albeit without infotainment and navigation and other "stuff". You should be able to do whatever you need to do, call your mechanic and limp there to get them to sort the problem out for you.
- AlotOfReading 3y ago
  That's not how vehicles are designed for many reasons. For one, there's no option where critical components aren't designed with updateability in mind. It's simply a fact of life that code has bugs, and the only way to fix those bugs is updates. It's valid to feel that that those updates should applied at a dealership where the system can be tested afterwards, but that has a lot of implications of its own.
  - onetimeuse92304 3y ago
    You simply do not understand the point of extracting "basic car" from "smart car". The point isn't that the software will have no bugs. The point is that the basic car will have much less software and much more basic software and that it is much easier to test and ensure this software is in working condition. Also, it is absolutely not true that the basic software has to be updateable. With care, basic functionality can absolutely be completely finished. Imagine you have been in the market with ABS controller for a decade or more. It has really simple functionality, it should be possible to just bake it and never touch it again. Or maybe a controller that takes care of basic battery protection. Or motor ESC. When the components are simple enough and well defined enough, it should be possible to "finish" the software to a state where it never again needs to be updated. Or updated only in exceptional circumstances.
    - AlotOfReading 3y ago
      I have a very deep appreciation for simpler automotive components. Please don't take anything I'm saying as discounting those very real benefits. What I'm saying is that no practical amount of testing and verification can eliminate the need for updates. Let me give some practical examples I've seen: a bug in the silicon vendor BSP that prevented programs from accessing half of the physical memory. In another case the vendor provided optimistic battery curves that caused safety issues in certain conditions, so changes were needed after units were already in the field. In another case, a factory was taking the stuff we were giving them and using it to build weird Frankenstein images from various binary blobs, so nothing that had been produced for those weeks corresponded to anything that had been tested.
      - shiroiuma 3y ago
        Cars have been built for decades without needing any updates at all. Microcontrollers were first put in cars in the 1970s, for fuel injection control, and later for ABS and SRS. No one ever got a software update for their ABS controller in the 1990s. They've been happily running non-updateable code for many decades now, and all of a sudden you think it's normal to update embedded software? No, it really isn't. If there was a real problem, there was a manufacturer recall and they replaced parts.
- aendruk 3y ago
  This sounds like the concept of progressive enhancement.
intern4tional 3y ago
So this is mostly a solved problem at other OEMs, and I do not know how Ford does it. I know that other OEMs do variant testing, have complex SILS (software in the loop) test systems so that all potential failure scenarios are tested in software prior to update. The downside is that updates are slow to release then, with some other OEMs only putting out an update once or twice a year, but they avoid this scenario. Software wise, the industry is moving towards defined standardized interfaces for sensors that are versioned. Example: https://blackberry.qnx.com/en/ultimate-guides/software-defined-vehicle/sensor-fusion https://blackberry.qnx.com/en/ultimate-guides/software-defin... Back to the topic though, all major domain ECUs will have an A/B partition. Usually one of these is the OTA master, and has the capability to update the other microcontrollers via UDS. For safety critical microcontrollers that do not have a A/B partition, about half will not support OTA (this is just a thing for small MCUs), and the other half are flashable completely. So something like this is rare and should only happen on a secure boot failure or some other catastrophic scenario, which ideally your SILS system will have tested (SILS won't test all scenarios but will definitely test all failure cases). Larger OEMs may even have HILS (hardware in the loop systems) where these things are also tested on physical hardware prior to launch, but with software defined vehicles this is slowly going away.
blep-arsh 3y ago
Regarding M1 Macs: Apple bundles all the firmware with the OS and initializes every auxiliary core by uploading the corresponding firmware into its RAM on system boot (as far as I understand it). Not sure if that's something that can be done with a car (e.g. what if a controller loses power physically and reboots from a clean slate?), but that's one way of keeping firmware in sync with the OS release.

onetimeuse92304 3y ago

I don't think "more smart = less reliable" is inevitable. The goal should be designing it in a way that failure of the more smart features does not spill to critical functionality. Or that you can use it in a degraded state sort of like fly by wire planes do. I have no car design experience, but here is my 5-minute idea: For starters, you build a basic car. It drives, it has basic indicators, it has ABS, etc. It has some APIs that can be connected to but there is limited ability to break the basic car through those APIs. Then you build your smart car around it by adding to the basic car. You add on more computers, electronics, etc. But any time you rip off the smart car functionality, you still have the basic car working. If there is a need to update software, this should really only be needed for the smart car part. But even if it fails completely, the basic car underneath should be functional and you should be able to use the car albeit without infotainment and navigation and other "stuff". You should be able to do whatever you need to do, call your mechanic and limp there to get them to sort the problem out for you.

AlotOfReading 3y ago
That's not how vehicles are designed for many reasons. For one, there's no option where critical components aren't designed with updateability in mind. It's simply a fact of life that code has bugs, and the only way to fix those bugs is updates. It's valid to feel that that those updates should applied at a dealership where the system can be tested afterwards, but that has a lot of implications of its own.
- onetimeuse92304 3y ago
  You simply do not understand the point of extracting "basic car" from "smart car". The point isn't that the software will have no bugs. The point is that the basic car will have much less software and much more basic software and that it is much easier to test and ensure this software is in working condition. Also, it is absolutely not true that the basic software has to be updateable. With care, basic functionality can absolutely be completely finished. Imagine you have been in the market with ABS controller for a decade or more. It has really simple functionality, it should be possible to just bake it and never touch it again. Or maybe a controller that takes care of basic battery protection. Or motor ESC. When the components are simple enough and well defined enough, it should be possible to "finish" the software to a state where it never again needs to be updated. Or updated only in exceptional circumstances.
  - AlotOfReading 3y ago
    I have a very deep appreciation for simpler automotive components. Please don't take anything I'm saying as discounting those very real benefits. What I'm saying is that no practical amount of testing and verification can eliminate the need for updates. Let me give some practical examples I've seen: a bug in the silicon vendor BSP that prevented programs from accessing half of the physical memory. In another case the vendor provided optimistic battery curves that caused safety issues in certain conditions, so changes were needed after units were already in the field. In another case, a factory was taking the stuff we were giving them and using it to build weird Frankenstein images from various binary blobs, so nothing that had been produced for those weeks corresponded to anything that had been tested.
    - shiroiuma 3y ago
      Cars have been built for decades without needing any updates at all. Microcontrollers were first put in cars in the 1970s, for fuel injection control, and later for ABS and SRS. No one ever got a software update for their ABS controller in the 1990s. They've been happily running non-updateable code for many decades now, and all of a sudden you think it's normal to update embedded software? No, it really isn't. If there was a real problem, there was a manufacturer recall and they replaced parts.
aendruk 3y ago
This sounds like the concept of progressive enhancement.

intern4tional 3y ago

So this is mostly a solved problem at other OEMs, and I do not know how Ford does it. I know that other OEMs do variant testing, have complex SILS (software in the loop) test systems so that all potential failure scenarios are tested in software prior to update. The downside is that updates are slow to release then, with some other OEMs only putting out an update once or twice a year, but they avoid this scenario. Software wise, the industry is moving towards defined standardized interfaces for sensors that are versioned. Example: https://blackberry.qnx.com/en/ultimate-guides/software-defined-vehicle/sensor-fusion https://blackberry.qnx.com/en/ultimate-guides/software-defin... Back to the topic though, all major domain ECUs will have an A/B partition. Usually one of these is the OTA master, and has the capability to update the other microcontrollers via UDS. For safety critical microcontrollers that do not have a A/B partition, about half will not support OTA (this is just a thing for small MCUs), and the other half are flashable completely. So something like this is rare and should only happen on a secure boot failure or some other catastrophic scenario, which ideally your SILS system will have tested (SILS won't test all scenarios but will definitely test all failure cases). Larger OEMs may even have HILS (hardware in the loop systems) where these things are also tested on physical hardware prior to launch, but with software defined vehicles this is slowly going away.

blep-arsh 3y ago

Regarding M1 Macs: Apple bundles all the firmware with the OS and initializes every auxiliary core by uploading the corresponding firmware into its RAM on system boot (as far as I understand it). Not sure if that's something that can be done with a car (e.g. what if a controller loses power physically and reboots from a clean slate?), but that's one way of keeping firmware in sync with the OS release.