3 ms·

I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our

I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our servers and updating. As well as an onion roll-out process, with exponentially sized groups prioritized by distance from our office, and risk of customer loss due to downtime.

dividuum 9y ago
I think this all is a fascinating problem. I'm running a digital signage service for the Raspberry Pi (see profile) and also offer OTA updates. I've initially built my OTA process around the same idea: Keep a bare minimum bootable kernel and minimal userland that is always able to connect to the update servers. In case of a broken update I would simply rely on this part to fetch a fixed version. Unfortunately it turned out that most updates I did also updated the kernel (for new features/bug fixes) and it felt like I superfluously split my system into two parts that both need maintenance and testing. I've since changed to an A/B booting system that automatically falls back to the previous version if it isn't able to reach the servers within a few minutes. It feels way more unified and all I need for it to fall back to the previous version is a working kernel (which, on the Pi, is easy to test), the busybox shell and the first couple of lines of my init script. Everything else might break and the system will revert. It wasn't easy to reach this point but thinking about OTA updates doesn't feel so scary now.