5 ms·

Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old

Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old partition? For example, if the new partition is corrupt, or if there is a software bug that causes one of the following to happen: - the partition is unbootable, or - the device is able to begin the boot, even starting to run the new kernel, but somewhere along the way it locks up. Whether that be a kernel panic early on, or later on when systemd has started but one or more important services fail to start, or - The device finishes the initial boot successfully, including all of the services managed by systemd, but after say a few minutes, some services are exhibiting unstable behavior (like services crashing and starting and crashing and starting), or the kernel panics after everything had been up and running for a while

planewave 2y ago
Many embedded systems use SWupdate to manage this, which can hook into uboot.
opello 2y ago
For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html
- tkfu 2y ago
  There's a very useful framework called greenboot[0] for the latter case. [0] https://github.com/fedora-iot/greenboot https://github.com/fedora-iot/greenboot
msarnoff 2y ago
Use the hardware watchdog timer. When the machine resets, the bootloader can check a hardware register to determine the reset cause, and then boot into a different OS partition. (note: not every SoC has a register indicating reset cause, I’ve worked with Allwinner chips that don’t, so alternate methods may be required) Enable the watchdog timer at the start of the bootloader. (I work with arm systems so this would be in uboot). If no bootable OS is found, or the system crashes before the kernel fully boots, the watchdog times out. In Linux, enable the watchdog timer at boot in your kconfig. Then have a userspace service that periodically pings /dev/watchdog to keep the system alive. Have your panic handler just hang, and the watchdog will reset the machine. Another helpful trick to save panic logs is to use pstore/ramoops to keep a portion of the kernel logs across resets, and then save the pstore buffer out to disk after booting into the recovery system.

planewave 2y ago

Many embedded systems use SWupdate to manage this, which can hook into uboot.

opello 2y ago

For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html

tkfu 2y ago
There's a very useful framework called greenboot[0] for the latter case. [0] https://github.com/fedora-iot/greenboot https://github.com/fedora-iot/greenboot

msarnoff 2y ago

Use the hardware watchdog timer. When the machine resets, the bootloader can check a hardware register to determine the reset cause, and then boot into a different OS partition. (note: not every SoC has a register indicating reset cause, I’ve worked with Allwinner chips that don’t, so alternate methods may be required) Enable the watchdog timer at the start of the bootloader. (I work with arm systems so this would be in uboot). If no bootable OS is found, or the system crashes before the kernel fully boots, the watchdog times out. In Linux, enable the watchdog timer at boot in your kconfig. Then have a userspace service that periodically pings /dev/watchdog to keep the system alive. Have your panic handler just hang, and the watchdog will reset the machine. Another helpful trick to save panic logs is to use pstore/ramoops to keep a portion of the kernel logs across resets, and then save the pstore buffer out to disk after booting into the recovery system.