3 ms·

Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this fil

by tlb 9y ago

Userland. Basically during upgrades we installed a shell script in /etc/init.d: sleep 1200 if ping -c 5 central-services.anybots.com; then rm <this file> else cd /home/dist && git checkout STABLE && make install reboot fi (We used git to distribute binaries, so we could checkout and install a previous version easily). This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted. An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.

cperciva 9y ago
I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
- srcmap 9y ago
  This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
  - cperciva 9y ago
    Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.
joezydeco 9y ago
U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM). If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.

by tlb 9y ago

cperciva 9y ago

I believe GRUB can be told to fallback to a safe mode. I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.

srcmap 9y ago
This will be the time where setup HW watch dog should help. After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader. If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader. As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
- cperciva 9y ago
  Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.

joezydeco 9y ago

U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM). If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.