5 ms·

They later posted on their Twitter feed that the problem was caused by SiriusXM, and as a stopgap fix they'd be disabling that. But this mostly sounds like a ma

They later posted on their Twitter feed that the problem was caused by SiriusXM, and as a stopgap fix they'd be disabling that. But this mostly sounds like a massive failure to follow software-industry best practices. In particular: * They released the update a Friday (bad; you need staff ready to respond to problems * They released the update to everyone at once (bad; you want to deploy to a small subset first, monitor that subset for problems, then deploy to the rest); * They didn't have a rollback option ready * They didn't isolate software components well enough (SiriusXM shouldn't be able to crash the whole OS) * They didn't have good instrumentation - it took until Monday for them to find out there was a problem, and two days after that to find out which component was causing it. The center-console software is mostly not safety critical, but there is some possibility for a problem there to be a safety issue, which makes these issues quite concerning.

floatingatoll 9y ago
SiriusXM may be responsible for transmitting the OS update for their hardware stack over the satellite, but the chances are vanishingly small that SiriusXM is responsible for designing, testing, and queuing for delivery OS updates to a car manufacturer's hardware stack. Blaming SiriusXM for the issue masks that, but they can be called to question more accurately as a result by the press: * Did Fiat-Chrysler negotiate phased-deployment and day-of-week criteria with SiriusXM, or was SiriusXM alone responsible for those choices? * Can SiriusXM release OTA firmware updates for the hardware stack in these vehicles without the cooperation and signoff of Fiat? * Why did the firmware's watchdog process fail to detect the crash-reboot loop (or is no watchdog present), and do they intend to correct that failing in the future so that such crashes result in an automatic firmware rollback in the future?
- t1o5 9y ago
  FCA ISOs are bundled from various vendors. The QNX OS from Harman, some of the uconnect apps and the data connectivity is from Sprint. * FCA is an automotive company, I presume their s/w release management dont follow best practices. * SiriusXM cannot push OTA updates. It has to be provisioned from FCA and pushed through their uconnect DRM process to the headunit. * There is a watchdog in FCA UConnect but for the apps. Im not sure they have it for the OS. I know for sure that the watchdog restarts daemon apps when they crash.
  - bonestamp2 9y ago
    > The QNX OS from Harman Is it true that Panasonic makes some head units for FCA? Are all UConnect head units Harman?
    - t1o5 9y ago
      I think the 6.5” screensizes and MY13 or older ones were from Panasonic. Harman took over with the new touch enabled screen models. The consumer facing apps run on J2ME LWUIT on top of JamaicaVM in QNX.
    - luma 9y ago
      Mitsubishi made a handful of UConnect head units (specifically the RHB and RBZ models) in the early 2010s as well.
vvanders 9y ago
The fact that they didn't do 5%/10%/25%/50%/100% rollout over a week or two is just pure amateur hour. It's like OTA 101 for this exact reason. People always say the big 3 are going to catch up with Tesla but if anything this shows they have a heck of a long way to go.
- username223 9y ago
  > People always say the big 3 are going to catch up with Tesla... Who are "people?" I don't own a Tesla, but from what I've read, they are not great at manufacturing, and that's something that takes awhile to fix. My guess is that companies with decades of experience in building physical things, but little in software, have a decent chance.
  - aje403 9y ago
    Agreed, like Blockbuster
    - username223 9y ago
      Or Netflix. They got a whole lot worse as they started failing at shipping DVDs. Then BitTorrent ate their lunch. EDIT: Unfortunately, Tesla is in the business of moving humans around, and they are physical things with mass and volume.
    - PhasmaFelis 9y ago
      I don't believe Blockbuster manufactured anything.
bo1024 9y ago
Your list of mistakes seem to me like (to borrow from Fight Club) a list of unpolished brass on the Titanic. There is no right way to implement a software system integral to a car that auto-updates over the air at the whim of the manufacturer. And I'm uneasy about recommending for a safety-critical system any "best practices" from the industry that regularly brings us blue screens of death (or modern equivalents), gaping security holes, megabytes-stuffed webpages full of coin miners and ad tracking, etc.
- tehlike 9y ago
  A staged roll out (with a/b test manner with equal control/experiment arms) and monitoring of health metrics would totally catch this.
  - michaelt 9y ago
    A staged roll out would mean only 5% of customers' cars lost their radio, navigation, climate control and rear-view camera. I'll admit that's better than 100%, but it's still garbage. Where was the pre-release testing?
    - tehlike 9y ago
      Why not start with 1 car, 10 cars, 0.1% and beyond. My point is prerelease testing can only catch so much. you can think of many bugs that affected only people with certain settings etc. It is hard to replicate all the conditions appearing in the wild. This could be as simple as "last software/firmware version", or as complex as "car stalls as it makes a u turn". Hard to test all these.

floatingatoll 9y ago

SiriusXM may be responsible for transmitting the OS update for their hardware stack over the satellite, but the chances are vanishingly small that SiriusXM is responsible for designing, testing, and queuing for delivery OS updates to a car manufacturer's hardware stack. Blaming SiriusXM for the issue masks that, but they can be called to question more accurately as a result by the press: * Did Fiat-Chrysler negotiate phased-deployment and day-of-week criteria with SiriusXM, or was SiriusXM alone responsible for those choices? * Can SiriusXM release OTA firmware updates for the hardware stack in these vehicles without the cooperation and signoff of Fiat? * Why did the firmware's watchdog process fail to detect the crash-reboot loop (or is no watchdog present), and do they intend to correct that failing in the future so that such crashes result in an automatic firmware rollback in the future?

t1o5 9y ago
FCA ISOs are bundled from various vendors. The QNX OS from Harman, some of the uconnect apps and the data connectivity is from Sprint. * FCA is an automotive company, I presume their s/w release management dont follow best practices. * SiriusXM cannot push OTA updates. It has to be provisioned from FCA and pushed through their uconnect DRM process to the headunit. * There is a watchdog in FCA UConnect but for the apps. Im not sure they have it for the OS. I know for sure that the watchdog restarts daemon apps when they crash.
- bonestamp2 9y ago
  > The QNX OS from Harman Is it true that Panasonic makes some head units for FCA? Are all UConnect head units Harman?
  - t1o5 9y ago
    I think the 6.5” screensizes and MY13 or older ones were from Panasonic. Harman took over with the new touch enabled screen models. The consumer facing apps run on J2ME LWUIT on top of JamaicaVM in QNX.
  - luma 9y ago
    Mitsubishi made a handful of UConnect head units (specifically the RHB and RBZ models) in the early 2010s as well.

vvanders 9y ago

The fact that they didn't do 5%/10%/25%/50%/100% rollout over a week or two is just pure amateur hour. It's like OTA 101 for this exact reason. People always say the big 3 are going to catch up with Tesla but if anything this shows they have a heck of a long way to go.

username223 9y ago
> People always say the big 3 are going to catch up with Tesla... Who are "people?" I don't own a Tesla, but from what I've read, they are not great at manufacturing, and that's something that takes awhile to fix. My guess is that companies with decades of experience in building physical things, but little in software, have a decent chance.
- aje403 9y ago
  Agreed, like Blockbuster
  - username223 9y ago
    Or Netflix. They got a whole lot worse as they started failing at shipping DVDs. Then BitTorrent ate their lunch. EDIT: Unfortunately, Tesla is in the business of moving humans around, and they are physical things with mass and volume.
  - PhasmaFelis 9y ago
    I don't believe Blockbuster manufactured anything.

bo1024 9y ago

Your list of mistakes seem to me like (to borrow from Fight Club) a list of unpolished brass on the Titanic. There is no right way to implement a software system integral to a car that auto-updates over the air at the whim of the manufacturer. And I'm uneasy about recommending for a safety-critical system any "best practices" from the industry that regularly brings us blue screens of death (or modern equivalents), gaping security holes, megabytes-stuffed webpages full of coin miners and ad tracking, etc.

tehlike 9y ago
A staged roll out (with a/b test manner with equal control/experiment arms) and monitoring of health metrics would totally catch this.
- michaelt 9y ago
  A staged roll out would mean only 5% of customers' cars lost their radio, navigation, climate control and rear-view camera. I'll admit that's better than 100%, but it's still garbage. Where was the pre-release testing?
  - tehlike 9y ago
    Why not start with 1 car, 10 cars, 0.1% and beyond. My point is prerelease testing can only catch so much. you can think of many bugs that affected only people with certain settings etc. It is hard to replicate all the conditions appearing in the wild. This could be as simple as "last software/firmware version", or as complex as "car stalls as it makes a u turn". Hard to test all these.