12 ms·

Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

84 points by ralphmender 9y ago

jon-wood 9y ago
This is wonderfully timed, I’m currently looking for options when it comes to doing firmware updates on Linux based IoT devices. Does anyone have any recommendations?
- bgentry 9y ago
  The source of this article is Mender.io, which is an open source embedded linux distro that includes built-in OTA updates: http://mender.io/ http://mender.io/ I can't speak to their quality as I haven't tried it yet. I have used Resin.io quite a bit and it's great if it fits your use case (and the pricing model makes sense for you).
- sjmulder 9y ago
  A friend who is building an Linux appliance is using Yocto: https://www.yoctoproject.org/ https://www.yoctoproject.org/. He also talked about Mender, mentioned in the other comment. I don't know how much overlap there is between these projects.
  - ralphmender 9y ago
    Disclaimer: I'm with Mender.io and author of the article. The Yocto Project is a popular build system for your own embedded Linux distribution. Mender integrates with the Yocto Project with a layer (https://github.com/mendersoftware/meta-mender https://github.com/mendersoftware/meta-mender), but is a separate project for end-to-end OTA, which includes the client and the management server. Both are licensed under Apache 2.0 thus it is freely available and you're not locked into a hosted-only backend. Although this is an older blog post, here is how you can port the Mender client to a non-Yocto build system: https://mender.io/blog/porting-mender-to-a-non-yocto-build-system https://mender.io/blog/porting-mender-to-a-non-yocto-build-s...
  - kbaker 9y ago
    This page on the Yocto wiki https://wiki.yoctoproject.org/wiki/System_Update https://wiki.yoctoproject.org/wiki/System_Update provides an overview of many of the current offerings for software update systems.
- subway 9y ago
  Since Mender and Yocto have already been mentioned, I have to bring up Buildroot with SWupdate. Batteries aren't included like with Mender, but you also don't have to assemble layers from the far reaches of the internet like Yocto.
- SEJeff 9y ago
  I occasionally contribute to a fairly large home automation project named Home Assistant (https://home-assistant.io/ https://home-assistant.io/) They have an all in one operating system perconfigured for a raspberry pi or intel nuc dubbed "hass.io" that uses ResinOS under the covers: https://home-assistant.io/hassio/ https://home-assistant.io/hassio/ https://resinos.io/ https://resinos.io/ Under the covers, ResinOS is a minimal Yocto embedded linux + docker + some ota stuff. Consider looking into that before building your own.
  - jon-wood 9y ago
    Resin is definitely quite high up on my list of options, I've done some prototyping with their SaaS service as well, my only hesitation is that the pricing is pretty prohibitive for the sort of volumes we're looking at in the medium term. Like you, my initial contact with them was through Hassio, which has been fantastic. Definitely going to do some digging into Yocto and Mender though.
tomc1985 9y ago
This is FUD. Partition twice the amount of space you need on IOT's fixed media, and trickle-download your update to the empty partition. Once that's finished, verify the download then switch your bootloader. I have to update 500+ field installations with a new OS and this is the approach I'd take if we had the space and bandwidth (Instead we're sending out an army of techs and account reps armed with USB sticks)
- geofft 9y ago
  I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact that this means your root partition changes. Either you want to structure your system with separate read and write partitions and bind-mount the relevant directories from the write partition, or you want to make it completely read-only / stateless. Remember that /var/log is traditionally on your local disk, so if you don't do anything special, you'll even get two /var/logs on each device, which may or may not be what you want. - You do want a management server, as this document suggests, to track which devices have actually updated and which haven't, so you can manually send people after devices that are just behind a terrible internet connection. - You want some mechanism for detecting if the new version doesn't work and rolling back; this is basically as simple as setting a "I just tried partition X, if it doesn't work don't try it again" flag in the bootloader on boot, and clearing it once userspace is up (and when the partition gets rewritten with a new version). - The updates should be signed etc. as described in the document. Depending on your threat model, you might want to prevent replay attacks that cause an attacker-controlled downgrade by giving it a higher-versioned filename; either use HTTPS to an update server you control, or use signed metadata files with timestamps. The fact that you get image-based deployments instead of dealing with apt upgrades from arbitrarily-old versions (and thus inevitably slightly drifting configs on devices installed at different times) is fantastic.
  - Cyph0n 9y ago
msarnoff 9y ago
It's not a full OTA solution, but fwup (https://github.com/fhunleth/fwup https://github.com/fhunleth/fwup) handles the packaging and application of Linux firmware update images quite well. Apache licensed, supports A/B updates (see tomc1985's comment), Ed25519 digital signature verification, and u-boot support.
kahlonel 9y ago
I wonder how will a Git based client-side agent fit in. Why not use an already proven tech for the rollout, and then use custom installation scripts (also within the git repo) for doing software setup? The "server" here will just be normal git server, with commit ID serving as version number.
- wmf 9y ago
  That's addressed by the article. What about a management server, power/network loss, atomic updates, validation, etc? If you try to write custom installation scripts to do all that it's going to be a lot of work.
seanalltogether 9y ago
I've been contracting with an IoT company for 3 years now. It's interesting to see guys who coded exclusively for radio or infrared based remotes get pulled into the software development world. V1: 5 years ago everything was raw sockets and custom messaging formats with hand coded firmware and all data stored in a custom vector format, builds were distributed on google drive and flashed by hand. V2: 3 years ago we dragged them kicking and screaming into http and hand coded json apis, firmware was still custom, data still stored in custom vector format but updates were now done on a non secure server with a hash check. V3: this past year they started on a small box with a micro linux distro, apis are provided by standardized library, data now stored in sql, updates done over https. Things are better now, except they still expect to sell and support those first 2 options for the next 10 years.
- nitrogen 9y ago
  Part of what you are describing is exactly why IoT is sometimes called IoS. We've moved from solid, fast, low-latency, low-power hardware to unpredictable, slow, jittery, high-latency crap. Take the Philips Hue bridge for example. Raw binary protocol lighting tech from the 1980s can outperform it in terms of latency, throughput, and jitter. It's unaccpetable for a button to take action after some random delay between 100ms and 5s. It's even worse if there's a remote https round trip required, as network lag adds another layer of unpredictability.
  - seanalltogether 9y ago
    Raw binary protocol lighting tech from the 1980 was stateless, but nobody is willing to accept that nowadays for home automation. "Turn device on" - Great I can do that fast "Turn device off" - Great I can do that fast "Is device on or off?" - Hold on while I poll a serial rf signal device by device while I determine state. All of the slowdown coming from our tech and others that I've seen is because hardware guys still think these old stateless solution are acceptable and then have to hack something dirty on top to turn it into a stateful solution.
    - ianhowson 9y ago
keithnz 9y ago
We have 1000s and 1000s of devices and can easily Update them. It's not hard. The devices also has multiple micros and they can individually be updated and rolled back. It's not really hard to implement. Though in our case we had to build a lot of the infrastructure anyways for other reasons.
karmicthreat 9y ago
I like mender. It gives me a cheap 80% solution that covers updating and management. That said I would really like something as easy as docker for building devices. Yocto has a learning curve like a cliff. I really like resin.io's container system, but I want to self-host.
paulgerhardt 9y ago
If you're starting a new IoT project, build the updater first and use that for pushing new builds. By the time you're ready for production it will be rock solid and quick to boot.

84 points by ralphmender 9y ago

This is wonderfully timed, I’m currently looking for options when it comes to doing firmware updates on Linux based IoT devices. Does anyone have any recommendations?

bgentry 9y ago
The source of this article is Mender.io, which is an open source embedded linux distro that includes built-in OTA updates: http://mender.io/ http://mender.io/ I can't speak to their quality as I haven't tried it yet. I have used Resin.io quite a bit and it's great if it fits your use case (and the pricing model makes sense for you).
sjmulder 9y ago
A friend who is building an Linux appliance is using Yocto: https://www.yoctoproject.org/ https://www.yoctoproject.org/. He also talked about Mender, mentioned in the other comment. I don't know how much overlap there is between these projects.
- ralphmender 9y ago
  Disclaimer: I'm with Mender.io and author of the article. The Yocto Project is a popular build system for your own embedded Linux distribution. Mender integrates with the Yocto Project with a layer (https://github.com/mendersoftware/meta-mender https://github.com/mendersoftware/meta-mender), but is a separate project for end-to-end OTA, which includes the client and the management server. Both are licensed under Apache 2.0 thus it is freely available and you're not locked into a hosted-only backend. Although this is an older blog post, here is how you can port the Mender client to a non-Yocto build system: https://mender.io/blog/porting-mender-to-a-non-yocto-build-system https://mender.io/blog/porting-mender-to-a-non-yocto-build-s...
- kbaker 9y ago
  This page on the Yocto wiki https://wiki.yoctoproject.org/wiki/System_Update https://wiki.yoctoproject.org/wiki/System_Update provides an overview of many of the current offerings for software update systems.
subway 9y ago
Since Mender and Yocto have already been mentioned, I have to bring up Buildroot with SWupdate. Batteries aren't included like with Mender, but you also don't have to assemble layers from the far reaches of the internet like Yocto.
SEJeff 9y ago
I occasionally contribute to a fairly large home automation project named Home Assistant (https://home-assistant.io/ https://home-assistant.io/) They have an all in one operating system perconfigured for a raspberry pi or intel nuc dubbed "hass.io" that uses ResinOS under the covers: https://home-assistant.io/hassio/ https://home-assistant.io/hassio/ https://resinos.io/ https://resinos.io/ Under the covers, ResinOS is a minimal Yocto embedded linux + docker + some ota stuff. Consider looking into that before building your own.
- jon-wood 9y ago
  Resin is definitely quite high up on my list of options, I've done some prototyping with their SaaS service as well, my only hesitation is that the pricing is pretty prohibitive for the sort of volumes we're looking at in the medium term. Like you, my initial contact with them was through Hassio, which has been fantastic. Definitely going to do some digging into Yocto and Mender though.

tomc1985 9y ago

This is FUD. Partition twice the amount of space you need on IOT's fixed media, and trickle-download your update to the empty partition. Once that's finished, verify the download then switch your bootloader. I have to update 500+ field installations with a new OS and this is the approach I'd take if we had the space and bandwidth (Instead we're sending out an army of techs and account reps armed with USB sticks)

geofft 9y ago
I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact that this means your root partition changes. Either you want to structure your system with separate read and write partitions and bind-mount the relevant directories from the write partition, or you want to make it completely read-only / stateless. Remember that /var/log is traditionally on your local disk, so if you don't do anything special, you'll even get two /var/logs on each device, which may or may not be what you want. - You do want a management server, as this document suggests, to track which devices have actually updated and which haven't, so you can manually send people after devices that are just behind a terrible internet connection. - You want some mechanism for detecting if the new version doesn't work and rolling back; this is basically as simple as setting a "I just tried partition X, if it doesn't work don't try it again" flag in the bootloader on boot, and clearing it once userspace is up (and when the partition gets rewritten with a new version). - The updates should be signed etc. as described in the document. Depending on your threat model, you might want to prevent replay attacks that cause an attacker-controlled downgrade by giving it a higher-versioned filename; either use HTTPS to an update server you control, or use signed metadata files with timestamps. The fact that you get image-based deployments instead of dealing with apt upgrades from arbitrarily-old versions (and thus inevitably slightly drifting configs on devices installed at different times) is fantastic.
- Cyph0n 9y ago

msarnoff 9y ago

It's not a full OTA solution, but fwup (https://github.com/fhunleth/fwup https://github.com/fhunleth/fwup) handles the packaging and application of Linux firmware update images quite well. Apache licensed, supports A/B updates (see tomc1985's comment), Ed25519 digital signature verification, and u-boot support.

kahlonel 9y ago

I wonder how will a Git based client-side agent fit in. Why not use an already proven tech for the rollout, and then use custom installation scripts (also within the git repo) for doing software setup? The "server" here will just be normal git server, with commit ID serving as version number.

wmf 9y ago
That's addressed by the article. What about a management server, power/network loss, atomic updates, validation, etc? If you try to write custom installation scripts to do all that it's going to be a lot of work.

seanalltogether 9y ago

I've been contracting with an IoT company for 3 years now. It's interesting to see guys who coded exclusively for radio or infrared based remotes get pulled into the software development world. V1: 5 years ago everything was raw sockets and custom messaging formats with hand coded firmware and all data stored in a custom vector format, builds were distributed on google drive and flashed by hand. V2: 3 years ago we dragged them kicking and screaming into http and hand coded json apis, firmware was still custom, data still stored in custom vector format but updates were now done on a non secure server with a hash check. V3: this past year they started on a small box with a micro linux distro, apis are provided by standardized library, data now stored in sql, updates done over https. Things are better now, except they still expect to sell and support those first 2 options for the next 10 years.

nitrogen 9y ago
Part of what you are describing is exactly why IoT is sometimes called IoS. We've moved from solid, fast, low-latency, low-power hardware to unpredictable, slow, jittery, high-latency crap. Take the Philips Hue bridge for example. Raw binary protocol lighting tech from the 1980s can outperform it in terms of latency, throughput, and jitter. It's unaccpetable for a button to take action after some random delay between 100ms and 5s. It's even worse if there's a remote https round trip required, as network lag adds another layer of unpredictability.
- seanalltogether 9y ago
  Raw binary protocol lighting tech from the 1980 was stateless, but nobody is willing to accept that nowadays for home automation. "Turn device on" - Great I can do that fast "Turn device off" - Great I can do that fast "Is device on or off?" - Hold on while I poll a serial rf signal device by device while I determine state. All of the slowdown coming from our tech and others that I've seen is because hardware guys still think these old stateless solution are acceptable and then have to hack something dirty on top to turn it into a stateful solution.
  - ianhowson 9y ago

keithnz 9y ago

We have 1000s and 1000s of devices and can easily Update them. It's not hard. The devices also has multiple micros and they can individually be updated and rolled back. It's not really hard to implement. Though in our case we had to build a lot of the infrastructure anyways for other reasons.

karmicthreat 9y ago

I like mender. It gives me a cheap 80% solution that covers updating and management. That said I would really like something as easy as docker for building devices. Yocto has a learning curve like a cliff. I really like resin.io's container system, but I want to self-host.

paulgerhardt 9y ago

If you're starting a new IoT project, build the updater first and use that for pushing new builds. By the time you're ready for production it will be rock solid and quick to boot.