5 ms·

Embedded Linux systems supporting OTA usually employs a dual root file system (RFS) approach where the upgrade is placed onto the currently not used partition a

Embedded Linux systems supporting OTA usually employs a dual root file system (RFS) approach where the upgrade is placed onto the currently not used partition and then after successful upgrade the RFS to boot into is replaced. It is an easy solutions which ensures integrity and has few drawbacks for typical embedded systems. The output of a typical Embedded Linux CI build is a complete RFS, having to care about individual packages would just be a headache, when you can replace the entire RFS and be done with it.

xyzzy_plugh 10y ago
Exactly. I wish Mender was around 4 years ago. I built exactly this!
- padelt 10y ago
  Interesting! How did you go about falling back to an older version if the update was bad? Is there a nice way to do this automatically? Say I update to a really botched version with the kernel panicing before it reaches userland. Does this need manual intervention?
  - mpasinski 10y ago
    disclaimer: I work for Mender It is possible to make rollback fully automatic. In order to do so you need some integration with bootloader. It needs to be configured so that it can roll back to the previously working partition if update is broken. What is more, you can add some user space runtime checks that can verify the update and if those are not passing (updated image is broken) you can rollback to the previous one as well.
    - padelt 10y ago
      (Way) back when I was working with OpenEmbedded/Ångström I dreamt up sth like this: make uboot/whatever set up a HW watchdog that is retriggered once from kernel mode and then in the userspace as ususal. The daunting task would've been to implement the (platform specific) watchdog in the bootloader. Looking now, there is support for at least some platforms - nice! Good luck with Mender - I'm sure you are badly needed!
    - dividuum 10y ago
      As someone that also implemented A/B booting for the Pi: I wonder how you roll back fully automated? I read a bit of the code but wasn't able to find that. Or is that already handled by u-boot? In my case, the first thing I do once an unverified version boots is to switch back to the other partition (so the known good version is active during the next boot), then run a detached reboot process that forces a reboot in 5 minutes. Once the system is up and it verified that everything is ok, it commits the next version (by switching back to the partition that booted and marking it as confirmed) so it is now active by default. Finally it kills the still running 'reboot' process. As far as I understand your update process: You download a complete new version for every update and are able to stream that directly to the new partition? Is there any way to do delta updates? In my experience, most of the disk content is unchanged, unless you do major updates. In my case I download the new version using zsync, verify the downloaded/updated `install.zip` (which is kept on the volatile data partition), then extract that to the new partition. I make sure that `install.zip` is created in a way that it is rsyncable, so updates are pretty small that way. Of course you lose the streaming feature, unless you modify zsync somehow to support that.
      - eystein 10y ago
        I work on Mender, so I can tell you how automated rollback works there. The update is written to the inactive rootfs partition, uboot is configured to boot from it and the device is rebooted. Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Only after successfully reporting the success to the server Mender will "commit" the update, meaning configuring uboot to persistently boot from this updated partition. Mender already does compression, but you are right that there are optimizations that can be made for application updates, e.g. delta or other types of updates. We are planning to implement this as well. The first priority for Mender is to make it robust, i.e. make sure the update is atomic and that you can always roll back.
        dividuum 10y ago
        > Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure. > Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Is there any deadline at all for that? I explicitly spawn a reboot command that ensures that even if everything gets stuck (in software, not in hardware) for whatever reason, the system falls back to the previous version (unless the reboot command gets killed too, in which case a manual restart is required). Any thoughts on that?
Matthias247 10y ago
That's true. I'm currently also involved in the development of an embedded linux project for which we yet have to find out the perfect update story. We thought about replacing individual packets too, but it looks like a hard way from various perspectives: - Build system: Would need to figure out how to build all these packets (with their requirements) in a reliable way. - Deployment: Deployments should be reproducable. All devices should get the same version of all subcomponents. And a rollback to an older version (with older versions of subcomponents) should be possible. Having some devices floating around with untested configurations (because the device decided to update on package but not the remaining ones) is not desirable, because you want to only handover fully tested/qualified software versions to the customer. Building the whole RFS at once, storing it and flashing it at once currently seems like the way to go. Will take a look at mender if it could help.
- jononor 10y ago
  Yes, I did package-based updates on a embedded system (not enough time to go dual-RFS), and it was painful. The statefulness of such update means that all transition paths must be tested for each release, which adds up very quickly. This effectively reduced how often and how timely we shipped updates. Package systems are also quite fragile, ie if a post-trigger/script fails the package install fails. And sometimes not cleanly either, where the partially-installed package blocks install of a new fixed package. In the worst case the system is left in an in-between broken state.
- karmicthreat 10y ago
  I'm kind of in this boat now with a couple of products. The catch is I need to do both local and OTA updates. This seems to be a rare feature that nobody is doing. I also am trying to get Resin.io like containers working. It seems like it would be an easier way to test and deploy.
  - Matthias247 10y ago
    That's also a requirement for me - local updates (via connected USB stick, triggered via reboot or web interface) should work also in addition to OTA. _In fact local updates would have an even higher priority. We fiddled around also a little bit with the container route. I liked it for quick iteration times (rebuild a docker container, pull it from local image registry to device and test it there). But we found out that we won't get by with just container updates, and that most updates would also need to contain a new kernel or kernel module versions.

xyzzy_plugh 10y ago

Exactly. I wish Mender was around 4 years ago. I built exactly this!

padelt 10y ago
Interesting! How did you go about falling back to an older version if the update was bad? Is there a nice way to do this automatically? Say I update to a really botched version with the kernel panicing before it reaches userland. Does this need manual intervention?
- mpasinski 10y ago
  disclaimer: I work for Mender It is possible to make rollback fully automatic. In order to do so you need some integration with bootloader. It needs to be configured so that it can roll back to the previously working partition if update is broken. What is more, you can add some user space runtime checks that can verify the update and if those are not passing (updated image is broken) you can rollback to the previous one as well.
  - padelt 10y ago
    (Way) back when I was working with OpenEmbedded/Ångström I dreamt up sth like this: make uboot/whatever set up a HW watchdog that is retriggered once from kernel mode and then in the userspace as ususal. The daunting task would've been to implement the (platform specific) watchdog in the bootloader. Looking now, there is support for at least some platforms - nice! Good luck with Mender - I'm sure you are badly needed!
  - dividuum 10y ago
    As someone that also implemented A/B booting for the Pi: I wonder how you roll back fully automated? I read a bit of the code but wasn't able to find that. Or is that already handled by u-boot? In my case, the first thing I do once an unverified version boots is to switch back to the other partition (so the known good version is active during the next boot), then run a detached reboot process that forces a reboot in 5 minutes. Once the system is up and it verified that everything is ok, it commits the next version (by switching back to the partition that booted and marking it as confirmed) so it is now active by default. Finally it kills the still running 'reboot' process. As far as I understand your update process: You download a complete new version for every update and are able to stream that directly to the new partition? Is there any way to do delta updates? In my experience, most of the disk content is unchanged, unless you do major updates. In my case I download the new version using zsync, verify the downloaded/updated `install.zip` (which is kept on the volatile data partition), then extract that to the new partition. I make sure that `install.zip` is created in a way that it is rsyncable, so updates are pretty small that way. Of course you lose the streaming feature, unless you modify zsync somehow to support that.
    - eystein 10y ago
      I work on Mender, so I can tell you how automated rollback works there. The update is written to the inactive rootfs partition, uboot is configured to boot from it and the device is rebooted. Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Only after successfully reporting the success to the server Mender will "commit" the update, meaning configuring uboot to persistently boot from this updated partition. Mender already does compression, but you are right that there are optimizations that can be made for application updates, e.g. delta or other types of updates. We are planning to implement this as well. The first priority for Mender is to make it robust, i.e. make sure the update is atomic and that you can always roll back.
      - dividuum 10y ago
        > Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure. > Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Is there any deadline at all for that? I explicitly spawn a reboot command that ensures that even if everything gets stuck (in software, not in hardware) for whatever reason, the system falls back to the previous version (unless the reboot command gets killed too, in which case a manual restart is required). Any thoughts on that?

Matthias247 10y ago

That's true. I'm currently also involved in the development of an embedded linux project for which we yet have to find out the perfect update story. We thought about replacing individual packets too, but it looks like a hard way from various perspectives: - Build system: Would need to figure out how to build all these packets (with their requirements) in a reliable way. - Deployment: Deployments should be reproducable. All devices should get the same version of all subcomponents. And a rollback to an older version (with older versions of subcomponents) should be possible. Having some devices floating around with untested configurations (because the device decided to update on package but not the remaining ones) is not desirable, because you want to only handover fully tested/qualified software versions to the customer. Building the whole RFS at once, storing it and flashing it at once currently seems like the way to go. Will take a look at mender if it could help.

jononor 10y ago
Yes, I did package-based updates on a embedded system (not enough time to go dual-RFS), and it was painful. The statefulness of such update means that all transition paths must be tested for each release, which adds up very quickly. This effectively reduced how often and how timely we shipped updates. Package systems are also quite fragile, ie if a post-trigger/script fails the package install fails. And sometimes not cleanly either, where the partially-installed package blocks install of a new fixed package. In the worst case the system is left in an in-between broken state.
karmicthreat 10y ago
I'm kind of in this boat now with a couple of products. The catch is I need to do both local and OTA updates. This seems to be a rare feature that nobody is doing. I also am trying to get Resin.io like containers working. It seems like it would be an easier way to test and deploy.
- Matthias247 10y ago
  That's also a requirement for me - local updates (via connected USB stick, triggered via reboot or web interface) should work also in addition to OTA. _In fact local updates would have an even higher priority. We fiddled around also a little bit with the container route. I liked it for quick iteration times (rebuild a docker container, pull it from local image registry to device and test it there). But we found out that we won't get by with just container updates, and that most updates would also need to contain a new kernel or kernel module versions.