5 ms·

This is FUD. Partition twice the amount of space you need on IOT's fixed media, and trickle-download your update to the empty partition. Once that's finished,

by tomc1985 9y ago

This is FUD. Partition twice the amount of space you need on IOT's fixed media, and trickle-download your update to the empty partition. Once that's finished, verify the download then switch your bootloader. I have to update 500+ field installations with a new OS and this is the approach I'd take if we had the space and bandwidth (Instead we're sending out an army of techs and account reps armed with USB sticks)

geofft 9y ago
I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact that this means your root partition changes. Either you want to structure your system with separate read and write partitions and bind-mount the relevant directories from the write partition, or you want to make it completely read-only / stateless. Remember that /var/log is traditionally on your local disk, so if you don't do anything special, you'll even get two /var/logs on each device, which may or may not be what you want. - You do want a management server, as this document suggests, to track which devices have actually updated and which haven't, so you can manually send people after devices that are just behind a terrible internet connection. - You want some mechanism for detecting if the new version doesn't work and rolling back; this is basically as simple as setting a "I just tried partition X, if it doesn't work don't try it again" flag in the bootloader on boot, and clearing it once userspace is up (and when the partition gets rewritten with a new version). - The updates should be signed etc. as described in the document. Depending on your threat model, you might want to prevent replay attacks that cause an attacker-controlled downgrade by giving it a higher-versioned filename; either use HTTPS to an update server you control, or use signed metadata files with timestamps. The fact that you get image-based deployments instead of dealing with apt upgrades from arbitrarily-old versions (and thus inevitably slightly drifting configs on devices installed at different times) is fantastic.
- Cyph0n 9y ago
  How do you handle key and/or certificate storage at the client side? Depending on the threat model, the update verification step can be subverted.
  - geofft 9y ago
    You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long). Once the device has been installed, there is always at least one working partition on the device - the partition that was last booted. So you don't need a minimal recovery partition or anything. (You could build a recovery command-line option in, if you want, but it's just a custom way of booting the normal partition.)
    - Cyph0n 9y ago
      How do you protect keys and/or certificates stored on the device from being exfiltrated or replaced by an adversary?
      - wmf 9y ago
        Unless you have a TPM or something you just don't; this is an unsolvable problem in general.
      - geofft 9y ago
        For all of my use cases, either physical access counts as game over and is out of the threat model, or we're using Secure Boot for verifying the bootloader and/or TPMs for keeping secrets, at which point this is a problem with known solutions that aren't specific to image-based updates. If you're using Secure Boot with read-only images, one thing to try is dm_verity, which is how Chrome OS solves this problem - it's a Merkle hash of the entire block device that's checked lazily as blocks are accessed. If a block has been tampered with, you can configure dm_verity to either panic the system or return an I/O error for that block. (Or you can just read the entire image and verify it up front at the cost of slower boot time.) In particular, for all my use cases, there's already some mechanism for the device gaining a secure channel to the rest of the infrastructure, so if you're worried about keeping updates secret (which you may or may not be!), just protect updates by that mechanism.
bigiain 9y ago
Heh - I've mentioned this here before - I got very lucky a while back, when between finalising the design and BOM of our IoT widget and ordering parts for the first production run - the price of 4G micro SD cards dropped below the price of 2G ones. That gave me an entire full sized extra partition to play with for updates, which reduced a _lot_ of my headaches...
nickpsecurity 9y ago
All of you interested in embedded tips might find Jack Ganssle's The Embedded Muse interesting. The current and back issues are here: http://www.ganssle.com/tem-subunsub.html http://www.ganssle.com/tem-subunsub.html Relevant to tomc1985's comment, Jack surveyed how embedded developers were handling updates with their methods in these two articles: http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem289.html http://www.ganssle.com/tem/tem289.html
- IndrekR 9y ago
  Second that! Have been reading TEM since 2004. Great fortnightly source of wisdom for anyone dealing with microcontrollers!
euyyn 9y ago
That's precisely what Android Things does for you for updates :) Plus automatic rollback if the new version is problematic.
kabdib 9y ago
A short tale of three firmware updaters. Device #1: A "root boot" loader in flash, very simple, loaded by a boot ROM. Its only job is to find the first good "large" chunk of boot code, get it into RAM and run it. The "large" chunks are in one of two slots. The update process is simple (update and test each slot independently), and if the update is interrupted there's always an image to boot from. It takes the developer responsible for FW update about a day to write the code and get it rock solid. An update takes less than about five seconds. Device #2: Implements a complicated file system. Some files need to be updated, some are untouchable, some need other special treatment (and the documentation is, unsurprisingly, dead wrong about things). The update process is very slow (tens of minutes) and if it is interrupted you wind up with a brick. It takes about two man-years to get the update process stable enough to ship. Device #3: Implements the standard USB firmware update protocol (DFU). There's a mountain of host-side code to deal with the edge conditions, and firmware updates are unpredictable. Devices get bricked pretty often. Takes maybe six months to get the update process stable enough to ship. Thinking of #2 and #3 still makes me mad. This stuff just isn't hard.
teekert 9y ago
All built into https://www.ubuntu.com/core https://www.ubuntu.com/core

geofft 9y ago

I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact that this means your root partition changes. Either you want to structure your system with separate read and write partitions and bind-mount the relevant directories from the write partition, or you want to make it completely read-only / stateless. Remember that /var/log is traditionally on your local disk, so if you don't do anything special, you'll even get two /var/logs on each device, which may or may not be what you want. - You do want a management server, as this document suggests, to track which devices have actually updated and which haven't, so you can manually send people after devices that are just behind a terrible internet connection. - You want some mechanism for detecting if the new version doesn't work and rolling back; this is basically as simple as setting a "I just tried partition X, if it doesn't work don't try it again" flag in the bootloader on boot, and clearing it once userspace is up (and when the partition gets rewritten with a new version). - The updates should be signed etc. as described in the document. Depending on your threat model, you might want to prevent replay attacks that cause an attacker-controlled downgrade by giving it a higher-versioned filename; either use HTTPS to an update server you control, or use signed metadata files with timestamps. The fact that you get image-based deployments instead of dealing with apt upgrades from arbitrarily-old versions (and thus inevitably slightly drifting configs on devices installed at different times) is fantastic.

Cyph0n 9y ago
How do you handle key and/or certificate storage at the client side? Depending on the threat model, the update verification step can be subverted.
- geofft 9y ago
  You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long). Once the device has been installed, there is always at least one working partition on the device - the partition that was last booted. So you don't need a minimal recovery partition or anything. (You could build a recovery command-line option in, if you want, but it's just a custom way of booting the normal partition.)
  - Cyph0n 9y ago
    How do you protect keys and/or certificates stored on the device from being exfiltrated or replaced by an adversary?
    - wmf 9y ago
      Unless you have a TPM or something you just don't; this is an unsolvable problem in general.
    - geofft 9y ago
      For all of my use cases, either physical access counts as game over and is out of the threat model, or we're using Secure Boot for verifying the bootloader and/or TPMs for keeping secrets, at which point this is a problem with known solutions that aren't specific to image-based updates. If you're using Secure Boot with read-only images, one thing to try is dm_verity, which is how Chrome OS solves this problem - it's a Merkle hash of the entire block device that's checked lazily as blocks are accessed. If a block has been tampered with, you can configure dm_verity to either panic the system or return an I/O error for that block. (Or you can just read the entire image and verify it up front at the cost of slower boot time.) In particular, for all my use cases, there's already some mechanism for the device gaining a secure channel to the rest of the infrastructure, so if you're worried about keeping updates secret (which you may or may not be!), just protect updates by that mechanism.

bigiain 9y ago

Heh - I've mentioned this here before - I got very lucky a while back, when between finalising the design and BOM of our IoT widget and ordering parts for the first production run - the price of 4G micro SD cards dropped below the price of 2G ones. That gave me an entire full sized extra partition to play with for updates, which reduced a _lot_ of my headaches...

nickpsecurity 9y ago

All of you interested in embedded tips might find Jack Ganssle's The Embedded Muse interesting. The current and back issues are here: http://www.ganssle.com/tem-subunsub.html http://www.ganssle.com/tem-subunsub.html Relevant to tomc1985's comment, Jack surveyed how embedded developers were handling updates with their methods in these two articles: http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem288.html http://www.ganssle.com/tem/tem289.html http://www.ganssle.com/tem/tem289.html

IndrekR 9y ago
Second that! Have been reading TEM since 2004. Great fortnightly source of wisdom for anyone dealing with microcontrollers!

euyyn 9y ago

That's precisely what Android Things does for you for updates :) Plus automatic rollback if the new version is problematic.

kabdib 9y ago

A short tale of three firmware updaters. Device #1: A "root boot" loader in flash, very simple, loaded by a boot ROM. Its only job is to find the first good "large" chunk of boot code, get it into RAM and run it. The "large" chunks are in one of two slots. The update process is simple (update and test each slot independently), and if the update is interrupted there's always an image to boot from. It takes the developer responsible for FW update about a day to write the code and get it rock solid. An update takes less than about five seconds. Device #2: Implements a complicated file system. Some files need to be updated, some are untouchable, some need other special treatment (and the documentation is, unsurprisingly, dead wrong about things). The update process is very slow (tens of minutes) and if it is interrupted you wind up with a brick. It takes about two man-years to get the update process stable enough to ship. Device #3: Implements the standard USB firmware update protocol (DFU). There's a mountain of host-side code to deal with the edge conditions, and firmware updates are unpredictable. Devices get bricked pretty often. Takes maybe six months to get the update process stable enough to ship. Thinking of #2 and #3 still makes me mad. This stuff just isn't hard.

teekert 9y ago

All built into https://www.ubuntu.com/core https://www.ubuntu.com/core