5 ms·

we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot

we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)? I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.

tlb 9y ago
Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this file> else cd /home/dist && git checkout STABLE && make install reboot fi (We used git to distribute binaries, so we could checkout and install a previous version easily). This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted. An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
- cperciva 9y ago
  I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
  - srcmap 9y ago
    This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
    - cperciva 9y ago
      Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.
- joezydeco 9y ago
  U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM). If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.
swiley 9y ago
It's fun to poke around in these embeded firmwares. Most of them are strung together with shell scripts for everything but the actual application, this kind of stuff is generally just another script.

tlb 9y ago

Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this file> else cd /home/dist && git checkout STABLE && make install reboot fi (We used git to distribute binaries, so we could checkout and install a previous version easily). This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted. An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.

cperciva 9y ago
I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
- srcmap 9y ago
  This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
  - cperciva 9y ago
    Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.
joezydeco 9y ago
U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM). If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.

swiley 9y ago

It's fun to poke around in these embeded firmwares. Most of them are strung together with shell scripts for everything but the actual application, this kind of stuff is generally just another script.