6 ms·

I've pushed dozens of OTA updates to WiFi devices (telepresence robots), and it's always terrifying. We'd test thoroughly, and we'd always start with devices in

by tlb 9y ago

I've pushed dozens of OTA updates to WiFi devices (telepresence robots), and it's always terrifying. We'd test thoroughly, and we'd always start with devices in our office, and wait to see them reboot before pushing to customers. In addition, we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again. WCGW? We never bricked one in the field, but I can think of a million ways it could have happened. The scariest times were pushing updates that affected WiFi connection for customers with unusual network configurations. In particular, we had one with a multi-campus Cisco WiFi system with WPA-Enterprise, which requires logging into a RADIUS server to get WiFi. We were running on FreeBSD which had a half-assed port of the Linux WPA authenticator. We never managed to replicate their network perfectly for in-house testing, so pushing updates to them was always nerve-wracking.

joshribakoff 9y ago
our app updates via the app itself. This gets regularly bricked but we have a Cronjob where we can put bash commands in a .txt file and upload that to our server, and devices will execute it regularly. This has always saved the day but apparently the past developers before me used to drive around with a USB stick on a regular basis, to re-image bricked devices. I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are
- mysterydip 9y ago
  Your end devices execute arbitrary code put in a specific url on your website? I assume this is a purely in-house device and not one in the hands of consumers?
  - agwa 9y ago
    Executing arbitrary code put at a specific URL on a website is exactly how software updates work. As long as it's properly authenticated, it's fine.
    - com2kid 9y ago
      Normally there is a bit of code signing involved. Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)
      - joshribakoff 9y ago
        If someone is not competent enough to keep their domain renewed, what's to say they're competent enough to keep private keys secured, and not lose or leak them. Then again I saw something on here recently about someone hijacking the whole .io TLD, so what you're saying is a real issue.
    - ramses0 9y ago
      Slightly, but not exactly. Any remote code execution could lead to "example.com/run_this.txt" being filled with "evil_hacker.sh". However if "run_this.txt" were somehow signed and verified prior to downloading (with signing keys kept off the distribution box) then it's much safer. Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.
      - joshribakoff 9y ago
        Our main issue is if someone tampers with content stored on the server, which gets copied to devices. For example someone replaces a JPG with the wrong JPG that is something offensive. If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry. To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.
cperciva 9y ago
we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)? I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.
- tlb 9y ago
  Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this file> else cd /home/dist && git checkout STABLE && make install reboot fi (We used git to distribute binaries, so we could checkout and install a previous version easily). This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted. An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
  - cperciva 9y ago
    I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
    - srcmap 9y ago
      This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
makmanalp 9y ago
Gradual rollout also helps (whenever possible), especially with stuff that has to work on many hardware platforms etc, so you can roll out only a few to a new hardware platform, see that it works, then roll out the rest.
frenchie4111 9y ago
I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our servers and updating. As well as an onion roll-out process, with exponentially sized groups prioritized by distance from our office, and risk of customer loss due to downtime.
- dividuum 9y ago
  I think this all is a fascinating problem. I'm running a digital signage service for the Raspberry Pi (see profile) and also offer OTA updates. I've initially built my OTA process around the same idea: Keep a bare minimum bootable kernel and minimal userland that is always able to connect to the update servers. In case of a broken update I would simply rely on this part to fetch a fixed version. Unfortunately it turned out that most updates I did also updated the kernel (for new features/bug fixes) and it felt like I superfluously split my system into two parts that both need maintenance and testing. I've since changed to an A/B booting system that automatically falls back to the previous version if it isn't able to reach the servers within a few minutes. It feels way more unified and all I need for it to fall back to the previous version is a working kernel (which, on the Pi, is easy to test), the busybox shell and the first couple of lines of my init script. Everything else might break and the system will revert. It wasn't easy to reach this point but thinking about OTA updates doesn't feel so scary now.
t0mbstone 9y ago
All hardware devices that allow firmware updates should have two things: 1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds) 2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it. Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
t0mbstone 9y ago
All hardware devices that allow firmware updates should have two things: 1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds) 2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it. Real time flashing over spotty WiFi (without any sort of pre-flashing validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
michaeloblak 9y ago
At RentingLock.com we solved such issue by... not using internet connection, thus not relaying on something that might fail anytime.

by tlb 9y ago

joshribakoff 9y ago

our app updates via the app itself. This gets regularly bricked but we have a Cronjob where we can put bash commands in a .txt file and upload that to our server, and devices will execute it regularly. This has always saved the day but apparently the past developers before me used to drive around with a USB stick on a regular basis, to re-image bricked devices. I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are

mysterydip 9y ago
Your end devices execute arbitrary code put in a specific url on your website? I assume this is a purely in-house device and not one in the hands of consumers?
- agwa 9y ago
  Executing arbitrary code put at a specific URL on a website is exactly how software updates work. As long as it's properly authenticated, it's fine.
  - com2kid 9y ago
    Normally there is a bit of code signing involved. Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)
    - joshribakoff 9y ago
      If someone is not competent enough to keep their domain renewed, what's to say they're competent enough to keep private keys secured, and not lose or leak them. Then again I saw something on here recently about someone hijacking the whole .io TLD, so what you're saying is a real issue.
  - ramses0 9y ago
    Slightly, but not exactly. Any remote code execution could lead to "example.com/run_this.txt" being filled with "evil_hacker.sh". However if "run_this.txt" were somehow signed and verified prior to downloading (with signing keys kept off the distribution box) then it's much safer. Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.
    - joshribakoff 9y ago
      Our main issue is if someone tampers with content stored on the server, which gets copied to devices. For example someone replaces a JPG with the wrong JPG that is something offensive. If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry. To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.

cperciva 9y ago

we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)? I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.

tlb 9y ago
Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this file> else cd /home/dist && git checkout STABLE && make install reboot fi (We used git to distribute binaries, so we could checkout and install a previous version easily). This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted. An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
- cperciva 9y ago
  I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
  - srcmap 9y ago
    This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.

makmanalp 9y ago

Gradual rollout also helps (whenever possible), especially with stuff that has to work on many hardware platforms etc, so you can roll out only a few to a new hardware platform, see that it works, then roll out the rest.

frenchie4111 9y ago

I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our servers and updating. As well as an onion roll-out process, with exponentially sized groups prioritized by distance from our office, and risk of customer loss due to downtime.

dividuum 9y ago
I think this all is a fascinating problem. I'm running a digital signage service for the Raspberry Pi (see profile) and also offer OTA updates. I've initially built my OTA process around the same idea: Keep a bare minimum bootable kernel and minimal userland that is always able to connect to the update servers. In case of a broken update I would simply rely on this part to fetch a fixed version. Unfortunately it turned out that most updates I did also updated the kernel (for new features/bug fixes) and it felt like I superfluously split my system into two parts that both need maintenance and testing. I've since changed to an A/B booting system that automatically falls back to the previous version if it isn't able to reach the servers within a few minutes. It feels way more unified and all I need for it to fall back to the previous version is a working kernel (which, on the Pi, is easy to test), the busybox shell and the first couple of lines of my init script. Everything else might break and the system will revert. It wasn't easy to reach this point but thinking about OTA updates doesn't feel so scary now.

t0mbstone 9y ago

All hardware devices that allow firmware updates should have two things: 1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds) 2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it. Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.

t0mbstone 9y ago

All hardware devices that allow firmware updates should have two things: 1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds) 2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it. Real time flashing over spotty WiFi (without any sort of pre-flashing validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.

michaeloblak 9y ago

At RentingLock.com we solved such issue by... not using internet connection, thus not relaying on something that might fail anytime.