4 ms·

How are kernel or application updates applied to buildroot-installed systems?

leonheld 2y ago
Well, buildroot is mainly for embedded applications, so whatever applies to that: OTA updates with A/B partitioning are common. OStree-based are getting a lot of traction as well. But technically you could build something with a package manager etc and update as usual. It's just a build system.
montecarl 2y ago
I would love to know. I currently have an embedded product using buildroot and as it is not exposed to a network at all, I don't have any worries about security. However, I'd love to hear of a nice mechanism to basically upgrade the system image in place. I imagine you could use something like a pair of partitions and just change the kernel boot parameters to point at the most recent one, but I'm curious about what solutions people use.
- lawik 2y ago
  That's the general wisdom I've seen. I work on the Nerves project which does Elixir on top of Buildroot and there we use fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) which does a very nice job of a lot of this. Including signing, hashing and more. This is a real example of a config: https://github.com/nerves-project/nerves_system_rpi4/blob/main/fwup.conf https://github.com/nerves-project/nerves_system_rpi4/blob/ma...
- jameshilliard 2y ago
  I use swupdate myself with buildroot. https://sbabic.github.io/swupdate/swupdate.html https://sbabic.github.io/swupdate/swupdate.html
jakogut 2y ago
I've previously developed a commercial product that used buildroot. In that case, it used an A/B partition layout like Android, along with a custom update script and kernel-based bootloader to implement atomic updates and fallback capability. The rootfs image was built by buildroot and devices would query an API and download the image, then write it to the inactive root partition. You can do effectively the same thing with GRUB, but the project I worked on also supported devices that booted using u-boot. The kexec bootloader allowed a single implementation for different architectures and platforms. So, the answer is you can update however you want, but there are a few common ways it's done in embedded.
- codetrotter 2y ago
  Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old partition? For example, if the new partition is corrupt, or if there is a software bug that causes one of the following to happen: - the partition is unbootable, or - the device is able to begin the boot, even starting to run the new kernel, but somewhere along the way it locks up. Whether that be a kernel panic early on, or later on when systemd has started but one or more important services fail to start, or - The device finishes the initial boot successfully, including all of the services managed by systemd, but after say a few minutes, some services are exhibiting unstable behavior (like services crashing and starting and crashing and starting), or the kernel panics after everything had been up and running for a while
  - planewave 2y ago
    Many embedded systems use SWupdate to manage this, which can hook into uboot.
  - opello 2y ago
    For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html
    - tkfu 2y ago
      There's a very useful framework called greenboot[0] for the latter case. [0] https://github.com/fedora-iot/greenboot https://github.com/fedora-iot/greenboot
  - msarnoff 2y ago
    Use the hardware watchdog timer. When the machine resets, the bootloader can check a hardware register to determine the reset cause, and then boot into a different OS partition. (note: not every SoC has a register indicating reset cause, I’ve worked with Allwinner chips that don’t, so alternate methods may be required) Enable the watchdog timer at the start of the bootloader. (I work with arm systems so this would be in uboot). If no bootable OS is found, or the system crashes before the kernel fully boots, the watchdog times out. In Linux, enable the watchdog timer at boot in your kconfig. Then have a userspace service that periodically pings /dev/watchdog to keep the system alive. Have your panic handler just hang, and the watchdog will reset the machine. Another helpful trick to save panic logs is to use pstore/ramoops to keep a portion of the kernel logs across resets, and then save the pstore buffer out to disk after booting into the recovery system.
stefan_ 2y ago
It's sort of out of the scope of Buildroot, but there are different systems like mender or swupdate that cover updates. They tend to need some level of platform specific changes because embedded systems these days have 7 levels of bootloaders.
- msarnoff 2y ago
  I’ve been using fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) in embedded Linux consumer products for almost 10 years and it has Just Worked. Supports A/B updates, uboot integration, and signed update files. Not sure about buildroot, but in Yocto it’s fairly straightforward to write an image creation recipe that bundles your ext4 filesystem image(s) into an update file you can deploy.
thebeardisred 2y ago
You build a new filesystem image and apply the new filesystem to the system.
andoma 2y ago
At work I've developed a buildroot based "embedded OS" för intel machines. Our buildroot system gets bundled into the kernels initramfs and we boot straight into that from UEFI. For upgrades we kexec() into a new kernel-image so generally we never write anything to disk (The original image can still be upgraded if deemed necessary). All applications are then regular docker images. All this is managed by an in-house developed deamon. Works quite well. That said, we're not really using buildroot for anything other than building a few libs + this in-house daemon so I'm leaning towards getting rid of it and just rely on busybox+musl+a few other deps built using a simple script.

Well, buildroot is mainly for embedded applications, so whatever applies to that: OTA updates with A/B partitioning are common. OStree-based are getting a lot of traction as well. But technically you could build something with a package manager etc and update as usual. It's just a build system.

montecarl 2y ago

I would love to know. I currently have an embedded product using buildroot and as it is not exposed to a network at all, I don't have any worries about security. However, I'd love to hear of a nice mechanism to basically upgrade the system image in place. I imagine you could use something like a pair of partitions and just change the kernel boot parameters to point at the most recent one, but I'm curious about what solutions people use.

lawik 2y ago
That's the general wisdom I've seen. I work on the Nerves project which does Elixir on top of Buildroot and there we use fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) which does a very nice job of a lot of this. Including signing, hashing and more. This is a real example of a config: https://github.com/nerves-project/nerves_system_rpi4/blob/main/fwup.conf https://github.com/nerves-project/nerves_system_rpi4/blob/ma...
jameshilliard 2y ago
I use swupdate myself with buildroot. https://sbabic.github.io/swupdate/swupdate.html https://sbabic.github.io/swupdate/swupdate.html

jakogut 2y ago

I've previously developed a commercial product that used buildroot. In that case, it used an A/B partition layout like Android, along with a custom update script and kernel-based bootloader to implement atomic updates and fallback capability. The rootfs image was built by buildroot and devices would query an API and download the image, then write it to the inactive root partition. You can do effectively the same thing with GRUB, but the project I worked on also supported devices that booted using u-boot. The kexec bootloader allowed a single implementation for different architectures and platforms. So, the answer is you can update however you want, but there are a few common ways it's done in embedded.

codetrotter 2y ago
Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old partition? For example, if the new partition is corrupt, or if there is a software bug that causes one of the following to happen: - the partition is unbootable, or - the device is able to begin the boot, even starting to run the new kernel, but somewhere along the way it locks up. Whether that be a kernel panic early on, or later on when systemd has started but one or more important services fail to start, or - The device finishes the initial boot successfully, including all of the services managed by systemd, but after say a few minutes, some services are exhibiting unstable behavior (like services crashing and starting and crashing and starting), or the kernel panics after everything had been up and running for a while
- planewave 2y ago
  Many embedded systems use SWupdate to manage this, which can hook into uboot.
- opello 2y ago
  For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html
  - tkfu 2y ago
    There's a very useful framework called greenboot[0] for the latter case. [0] https://github.com/fedora-iot/greenboot https://github.com/fedora-iot/greenboot
- msarnoff 2y ago
  Use the hardware watchdog timer. When the machine resets, the bootloader can check a hardware register to determine the reset cause, and then boot into a different OS partition. (note: not every SoC has a register indicating reset cause, I’ve worked with Allwinner chips that don’t, so alternate methods may be required) Enable the watchdog timer at the start of the bootloader. (I work with arm systems so this would be in uboot). If no bootable OS is found, or the system crashes before the kernel fully boots, the watchdog times out. In Linux, enable the watchdog timer at boot in your kconfig. Then have a userspace service that periodically pings /dev/watchdog to keep the system alive. Have your panic handler just hang, and the watchdog will reset the machine. Another helpful trick to save panic logs is to use pstore/ramoops to keep a portion of the kernel logs across resets, and then save the pstore buffer out to disk after booting into the recovery system.

stefan_ 2y ago

It's sort of out of the scope of Buildroot, but there are different systems like mender or swupdate that cover updates. They tend to need some level of platform specific changes because embedded systems these days have 7 levels of bootloaders.

msarnoff 2y ago
I’ve been using fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) in embedded Linux consumer products for almost 10 years and it has Just Worked. Supports A/B updates, uboot integration, and signed update files. Not sure about buildroot, but in Yocto it’s fairly straightforward to write an image creation recipe that bundles your ext4 filesystem image(s) into an update file you can deploy.

thebeardisred 2y ago

You build a new filesystem image and apply the new filesystem to the system.

andoma 2y ago

At work I've developed a buildroot based "embedded OS" för intel machines. Our buildroot system gets bundled into the kernels initramfs and we boot straight into that from UEFI. For upgrades we kexec() into a new kernel-image so generally we never write anything to disk (The original image can still be upgraded if deemed necessary). All applications are then regular docker images. All this is managed by an in-house developed deamon. Works quite well. That said, we're not really using buildroot for anything other than building a few libs + this in-house daemon so I'm leaning towards getting rid of it and just rely on busybox+musl+a few other deps built using a simple script.