5 ms·

Buildroot

81 points by jakogut 2y ago

metadat 2y ago
How are kernel or application updates applied to buildroot-installed systems?
- leonheld 2y ago
  Well, buildroot is mainly for embedded applications, so whatever applies to that: OTA updates with A/B partitioning are common. OStree-based are getting a lot of traction as well. But technically you could build something with a package manager etc and update as usual. It's just a build system.
- montecarl 2y ago
  I would love to know. I currently have an embedded product using buildroot and as it is not exposed to a network at all, I don't have any worries about security. However, I'd love to hear of a nice mechanism to basically upgrade the system image in place. I imagine you could use something like a pair of partitions and just change the kernel boot parameters to point at the most recent one, but I'm curious about what solutions people use.
  - lawik 2y ago
    That's the general wisdom I've seen. I work on the Nerves project which does Elixir on top of Buildroot and there we use fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) which does a very nice job of a lot of this. Including signing, hashing and more. This is a real example of a config: https://github.com/nerves-project/nerves_system_rpi4/blob/main/fwup.conf https://github.com/nerves-project/nerves_system_rpi4/blob/ma...
  - jameshilliard 2y ago
    I use swupdate myself with buildroot. https://sbabic.github.io/swupdate/swupdate.html https://sbabic.github.io/swupdate/swupdate.html
- jakogut 2y ago
  I've previously developed a commercial product that used buildroot. In that case, it used an A/B partition layout like Android, along with a custom update script and kernel-based bootloader to implement atomic updates and fallback capability. The rootfs image was built by buildroot and devices would query an API and download the image, then write it to the inactive root partition. You can do effectively the same thing with GRUB, but the project I worked on also supported devices that booted using u-boot. The kexec bootloader allowed a single implementation for different architectures and platforms. So, the answer is you can update however you want, but there are a few common ways it's done in embedded.
  - codetrotter 2y ago
    Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old partition? For example, if the new partition is corrupt, or if there is a software bug that causes one of the following to happen: - the partition is unbootable, or - the device is able to begin the boot, even starting to run the new kernel, but somewhere along the way it locks up. Whether that be a kernel panic early on, or later on when systemd has started but one or more important services fail to start, or - The device finishes the initial boot successfully, including all of the services managed by systemd, but after say a few minutes, some services are exhibiting unstable behavior (like services crashing and starting and crashing and starting), or the kernel panics after everything had been up and running for a while
    - planewave 2y ago
      Many embedded systems use SWupdate to manage this, which can hook into uboot.
    - opello 2y ago
      For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html
- stefan_ 2y ago
  It's sort of out of the scope of Buildroot, but there are different systems like mender or swupdate that cover updates. They tend to need some level of platform specific changes because embedded systems these days have 7 levels of bootloaders.
  - msarnoff 2y ago
    I’ve been using fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) in embedded Linux consumer products for almost 10 years and it has Just Worked. Supports A/B updates, uboot integration, and signed update files. Not sure about buildroot, but in Yocto it’s fairly straightforward to write an image creation recipe that bundles your ext4 filesystem image(s) into an update file you can deploy.
- thebeardisred 2y ago
  You build a new filesystem image and apply the new filesystem to the system.
- andoma 2y ago
  At work I've developed a buildroot based "embedded OS" för intel machines. Our buildroot system gets bundled into the kernels initramfs and we boot straight into that from UEFI. For upgrades we kexec() into a new kernel-image so generally we never write anything to disk (The original image can still be upgraded if deemed necessary). All applications are then regular docker images. All this is managed by an in-house developed deamon. Works quite well. That said, we're not really using buildroot for anything other than building a few libs + this in-house daemon so I'm leaning towards getting rid of it and just rely on busybox+musl+a few other deps built using a simple script.
kajkojednojajko 2y ago
Me: I couldn't write anything longer than 20 lines in make - there's just one type (string), no debugger, no data structures, no standard library, and not even modules. Buildroot developers: let's write a build system in make!
- squarefoot 2y ago
  I wonder if one day we'll see Zig used for system development. Among its other peculiarities, it also use itself as build language. https://ziglang.org/learn/build-system/ https://ziglang.org/learn/build-system/
  - tazjin 2y ago
    I really want to like Zig, but man, an unsafe language in 2024 is a hard sell (even to myself).
- ARob109 2y ago
  I found the learning curve to be much easier with buildroot and compared to yocto whose build system is python but uses mixed shell/python to describe the build. Builroot errors seemed to happen much less, and easier to understand compared to Yocto errors. Also seemed easier to corrupt the Yocto environment to the point of having to wipe and start fresh Buildroot kconfig and TUI menuconfig was super simple to find and add packages to the rootfs, even custom ones patched in to tree to out of tree directories that could be added in with a config option. Yocto AFAICT does not have a package discovery via TUI. You have to guess at package names, add them to you config and wait for build error to say not found. It will at least sometimes recommend similar named packages or the actual package name that eg provides the library you wanted. Otherwise it’s a search on openembedded to find package recipes that are not in the Yocto poky reference. Adding OOT tree code to Yocto is pretty simple, once you figure out to write shell/python recipe The makefile/Kconfig way of buildroot seemed much more intuitive especially given the kernel, busybox and uboot all use kconfig. So if you’re used to those, buildroot just seemed a natural extension to the rootfs.
  - 2y ago
IshKebab 2y ago
Coincidentally I tried this recently. I dunno if I liked it. It uses the kernel's `make menuconfig` system which seems like it occupies a bad place between "easy usable GUI" and "robust config file you can check in to git". Once you've configured it it then does a lot of downloading and compilation of random stuff and it's all strung together with janky shell scripts which means it's almost guaranteed to break. I got some weird errors that returned zero google results and the source of the error was a long awful shell pipeline involving `sed`, `tr` etc. Doesn't scream quality. I gave up and switched to manually compiling Linux and OpenSBI (which is really all I needed so buildroot was overkill anyway). Went much smoother and I actually vaguely understood what was happening. I haven't tried Yocto, maybe that is better engineered.
- dezgeg 2y ago
  I've generally found buildroot work pretty well. One common problem causing weird issues however is lots of tools from the host system are used (without any sandboxing). E.g building old releases of buildroot on newer Ubuntu will break. Its biggest Achilles heel IMO is the need to always compile everything from scratch with no possibility of caching. Also the fact that incremental rebuilds after changing config options isn't guaranteed to work unless you manually know to know which packages to force nuke and force to recompile (this is intentional design choice for implementation simplicity and well documented). Also having two different boards that inherit from common shared config isn't really supported, everyone just rolls their own custom config splicer at that point. Yocto fixes those issues, but IMO the most other things are much less ergonomic. While buildroot is extremely simple, yocto is total opposite to that, an architecture astronaut's dream. There is custom language to learn, ton of functionality, config and documentation split in tiny files in different repositories. Of which everyone using slightly different subset to accomplish almost the same thing.
- ahepp 2y ago
  > It uses the kernel's `make menuconfig` system which seems like it occupies a bad place between "easy usable GUI" and "robust config file you can check in to git". Are you aware you can generate a sparse config file with `make savedefconfig`? Maybe you're aware of that and it doesn't meet your definition of robust, which is reasonable. I think Buildroot is much better engineered than Yocto, in the sense of simplicity. Buildroot is fundamentally just a bunch of make and kconfig. Yocto has some powerful features but I find it to be considerably more difficult to penetrate "what is actually happening here". I haven't had as negative an experience with Buildroot error messages, but if you think those are bad, Yocto errors are like g++ template errors circa 2010.
throwaway9364 2y ago
We’ve built something that goes well with buildroot for installing and updating apps during development or in production. Does not require cloud or internet access. https://www.voyonic-systems.de/products/voyonic-fieldkit https://www.voyonic-systems.de/products/voyonic-fieldkit
- marbm 2y ago
  Not very sophisticated to use a throwaway account to advertise your product.
lrvick 2y ago
For those that want to see full source bootstrapped, deterministic, and container native alternative to buildroot check out https://codeberg.org/stagex https://codeberg.org/stagex Can make a bootable OS with just a containerfile, for example: https://git.distrust.co/public/airgap/src/branch/main/Containerfile https://git.distrust.co/public/airgap/src/branch/main/Contai... https://git.distrust.co/public/enclaveos/src/branch/master/Containerfile https://git.distrust.co/public/enclaveos/src/branch/master/C...
- vilunov 2y ago
  This is beautiful. Deterministic, bootstrapped, and doesn't break the established Linux conventions! Shame it's musl-only, I would love to use something like that to build glibc-based apps for future redistribution as proper Linux binaries.
  - lrvick 2y ago
    Musl is -much- easier to bootstrap than glibc with far fewer dependencies, that said, we are adding a x86_64-linux-gnu (glibc) cross compiler right now, among other target combinations. That will make it possible to make packages for any other linux distro as you like.
    - lrvick 2y ago
      Done.

Buildroot

81 points by jakogut 2y ago

metadat 2y ago

How are kernel or application updates applied to buildroot-installed systems?

leonheld 2y ago
Well, buildroot is mainly for embedded applications, so whatever applies to that: OTA updates with A/B partitioning are common. OStree-based are getting a lot of traction as well. But technically you could build something with a package manager etc and update as usual. It's just a build system.
montecarl 2y ago
I would love to know. I currently have an embedded product using buildroot and as it is not exposed to a network at all, I don't have any worries about security. However, I'd love to hear of a nice mechanism to basically upgrade the system image in place. I imagine you could use something like a pair of partitions and just change the kernel boot parameters to point at the most recent one, but I'm curious about what solutions people use.
- lawik 2y ago
  That's the general wisdom I've seen. I work on the Nerves project which does Elixir on top of Buildroot and there we use fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) which does a very nice job of a lot of this. Including signing, hashing and more. This is a real example of a config: https://github.com/nerves-project/nerves_system_rpi4/blob/main/fwup.conf https://github.com/nerves-project/nerves_system_rpi4/blob/ma...
- jameshilliard 2y ago
  I use swupdate myself with buildroot. https://sbabic.github.io/swupdate/swupdate.html https://sbabic.github.io/swupdate/swupdate.html
jakogut 2y ago
I've previously developed a commercial product that used buildroot. In that case, it used an A/B partition layout like Android, along with a custom update script and kernel-based bootloader to implement atomic updates and fallback capability. The rootfs image was built by buildroot and devices would query an API and download the image, then write it to the inactive root partition. You can do effectively the same thing with GRUB, but the project I worked on also supported devices that booted using u-boot. The kexec bootloader allowed a single implementation for different architectures and platforms. So, the answer is you can update however you want, but there are a few common ways it's done in embedded.
- codetrotter 2y ago
  Is there a common way with Buildroot and A/B partitioning to make it so that if the reboot into the updated partition fails to boot, it will reboot into the old partition? For example, if the new partition is corrupt, or if there is a software bug that causes one of the following to happen: - the partition is unbootable, or - the device is able to begin the boot, even starting to run the new kernel, but somewhere along the way it locks up. Whether that be a kernel panic early on, or later on when systemd has started but one or more important services fail to start, or - The device finishes the initial boot successfully, including all of the services managed by systemd, but after say a few minutes, some services are exhibiting unstable behavior (like services crashing and starting and crashing and starting), or the kernel panics after everything had been up and running for a while
  - planewave 2y ago
    Many embedded systems use SWupdate to manage this, which can hook into uboot.
  - opello 2y ago
    For systems using U-Boot you can use the bootcount[1] feature to detect the failure and react by booting into the old environment. Edit: This doesn't cover your running instability case, but that seems more like a feature that should be implemented using a user space watchdog. [1] https://docs.u-boot.org/en/latest/api/bootcount.html https://docs.u-boot.org/en/latest/api/bootcount.html
stefan_ 2y ago
It's sort of out of the scope of Buildroot, but there are different systems like mender or swupdate that cover updates. They tend to need some level of platform specific changes because embedded systems these days have 7 levels of bootloaders.
- msarnoff 2y ago
  I’ve been using fwup (https://github.com/fwup-home/fwup https://github.com/fwup-home/fwup) in embedded Linux consumer products for almost 10 years and it has Just Worked. Supports A/B updates, uboot integration, and signed update files. Not sure about buildroot, but in Yocto it’s fairly straightforward to write an image creation recipe that bundles your ext4 filesystem image(s) into an update file you can deploy.
thebeardisred 2y ago
You build a new filesystem image and apply the new filesystem to the system.
andoma 2y ago
At work I've developed a buildroot based "embedded OS" för intel machines. Our buildroot system gets bundled into the kernels initramfs and we boot straight into that from UEFI. For upgrades we kexec() into a new kernel-image so generally we never write anything to disk (The original image can still be upgraded if deemed necessary). All applications are then regular docker images. All this is managed by an in-house developed deamon. Works quite well. That said, we're not really using buildroot for anything other than building a few libs + this in-house daemon so I'm leaning towards getting rid of it and just rely on busybox+musl+a few other deps built using a simple script.

kajkojednojajko 2y ago

Me: I couldn't write anything longer than 20 lines in make - there's just one type (string), no debugger, no data structures, no standard library, and not even modules. Buildroot developers: let's write a build system in make!

squarefoot 2y ago
I wonder if one day we'll see Zig used for system development. Among its other peculiarities, it also use itself as build language. https://ziglang.org/learn/build-system/ https://ziglang.org/learn/build-system/
- tazjin 2y ago
  I really want to like Zig, but man, an unsafe language in 2024 is a hard sell (even to myself).
ARob109 2y ago
I found the learning curve to be much easier with buildroot and compared to yocto whose build system is python but uses mixed shell/python to describe the build. Builroot errors seemed to happen much less, and easier to understand compared to Yocto errors. Also seemed easier to corrupt the Yocto environment to the point of having to wipe and start fresh Buildroot kconfig and TUI menuconfig was super simple to find and add packages to the rootfs, even custom ones patched in to tree to out of tree directories that could be added in with a config option. Yocto AFAICT does not have a package discovery via TUI. You have to guess at package names, add them to you config and wait for build error to say not found. It will at least sometimes recommend similar named packages or the actual package name that eg provides the library you wanted. Otherwise it’s a search on openembedded to find package recipes that are not in the Yocto poky reference. Adding OOT tree code to Yocto is pretty simple, once you figure out to write shell/python recipe The makefile/Kconfig way of buildroot seemed much more intuitive especially given the kernel, busybox and uboot all use kconfig. So if you’re used to those, buildroot just seemed a natural extension to the rootfs.
- 2y ago

IshKebab 2y ago

Coincidentally I tried this recently. I dunno if I liked it. It uses the kernel's `make menuconfig` system which seems like it occupies a bad place between "easy usable GUI" and "robust config file you can check in to git". Once you've configured it it then does a lot of downloading and compilation of random stuff and it's all strung together with janky shell scripts which means it's almost guaranteed to break. I got some weird errors that returned zero google results and the source of the error was a long awful shell pipeline involving `sed`, `tr` etc. Doesn't scream quality. I gave up and switched to manually compiling Linux and OpenSBI (which is really all I needed so buildroot was overkill anyway). Went much smoother and I actually vaguely understood what was happening. I haven't tried Yocto, maybe that is better engineered.

dezgeg 2y ago
I've generally found buildroot work pretty well. One common problem causing weird issues however is lots of tools from the host system are used (without any sandboxing). E.g building old releases of buildroot on newer Ubuntu will break. Its biggest Achilles heel IMO is the need to always compile everything from scratch with no possibility of caching. Also the fact that incremental rebuilds after changing config options isn't guaranteed to work unless you manually know to know which packages to force nuke and force to recompile (this is intentional design choice for implementation simplicity and well documented). Also having two different boards that inherit from common shared config isn't really supported, everyone just rolls their own custom config splicer at that point. Yocto fixes those issues, but IMO the most other things are much less ergonomic. While buildroot is extremely simple, yocto is total opposite to that, an architecture astronaut's dream. There is custom language to learn, ton of functionality, config and documentation split in tiny files in different repositories. Of which everyone using slightly different subset to accomplish almost the same thing.
ahepp 2y ago
> It uses the kernel's `make menuconfig` system which seems like it occupies a bad place between "easy usable GUI" and "robust config file you can check in to git". Are you aware you can generate a sparse config file with `make savedefconfig`? Maybe you're aware of that and it doesn't meet your definition of robust, which is reasonable. I think Buildroot is much better engineered than Yocto, in the sense of simplicity. Buildroot is fundamentally just a bunch of make and kconfig. Yocto has some powerful features but I find it to be considerably more difficult to penetrate "what is actually happening here". I haven't had as negative an experience with Buildroot error messages, but if you think those are bad, Yocto errors are like g++ template errors circa 2010.

throwaway9364 2y ago

We’ve built something that goes well with buildroot for installing and updating apps during development or in production. Does not require cloud or internet access. https://www.voyonic-systems.de/products/voyonic-fieldkit https://www.voyonic-systems.de/products/voyonic-fieldkit

marbm 2y ago
Not very sophisticated to use a throwaway account to advertise your product.

lrvick 2y ago

For those that want to see full source bootstrapped, deterministic, and container native alternative to buildroot check out https://codeberg.org/stagex https://codeberg.org/stagex Can make a bootable OS with just a containerfile, for example: https://git.distrust.co/public/airgap/src/branch/main/Containerfile https://git.distrust.co/public/airgap/src/branch/main/Contai... https://git.distrust.co/public/enclaveos/src/branch/master/Containerfile https://git.distrust.co/public/enclaveos/src/branch/master/C...

vilunov 2y ago
This is beautiful. Deterministic, bootstrapped, and doesn't break the established Linux conventions! Shame it's musl-only, I would love to use something like that to build glibc-based apps for future redistribution as proper Linux binaries.
- lrvick 2y ago
  Musl is -much- easier to bootstrap than glibc with far fewer dependencies, that said, we are adding a x86_64-linux-gnu (glibc) cross compiler right now, among other target combinations. That will make it possible to make packages for any other linux distro as you like.
  - lrvick 2y ago
    Done.