4 ms·
This is a bit of a nightmare scenario and why when remote updating, you always test update to your own fleet first. Always.
by eschneider 3y ago
This is a bit of a nightmare scenario and why when remote updating, you always test update to your own fleet first. Always.
- kevin_nisbet 3y agoYes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.
- toddmorey 3y agoIt sounds like it was tested on their own fleet but they accidentally pushed the wrong bits when deploying the update more widely out to customers.
- carlivar 3y agoMaybe they should have an additional phase between test deploy and customers such as "employee personal vehicles".
- eschneider 3y agoThe usual "best practice" thing for IoT deploys, is to deploy to "your" devices, what for everyone to go green, then allow that build to deploy more widely. In a well-functioning system, it shouldn't be possible to swap bits between those stages. But who knows what these guys were doing. :/
- bink 3y agoReading between the lines of their public comments it sounds like they did run a full test through their test fleet, their employees, and then were rolling out to customers when the promotion process was "fat fingered". Maybe someone accidentally promoted the wrong release version.