5 ms·

My biggest pain point with Balena in my previous job was that it used Docker; it's unfortunate that most of the solutions in this space (including Deviceplane)

My biggest pain point with Balena in my previous job was that it used Docker; it's unfortunate that most of the solutions in this space (including Deviceplane) seem to do so. It's not really ideal in a number of installations to be pulling down 1GB Docker images simultaneously from 20 devices when you push an update. I'd be really interested in seeing a system based around Git and something like buildpacks instead. IoT definitely feels like the right space for pushing small diffs for deployments.

yjftsjthsd-h 7y ago
> It's not really ideal in a number of installations to be pulling down 1GB Docker images simultaneously from 20 devices when you push an update. That seems like a really big image? Alpine-based images in my experience tend to be under 100MB. But I would be fascinated to use p2p for distributing image layers; it feels like that shouldn't be too hard? EDIT: it's been done, apparently:) https://blog.bonner.is/docker-registry-for-ipfs/ https://blog.bonner.is/docker-registry-for-ipfs/
- zerotolerance 7y ago
  IPFS and decentralization don't really address artifact size issues.
  - yjftsjthsd-h 7y ago
    It certainly improves the practical outcome; replicating layers between nodes in a rack/datacenter is way faster than pulling it down 20x from one external upstream.
    - zerotolerance 7y ago
      Sure, but latency is not the same as size. You're talking about latency optimization via local caching. IPFS is a bit of a heavy solution compared to a simple cache. IPFS solves a totally different problem.
- joshwget 7y ago
  We see Alpine used in production less than most people think. People usually feel more comfortable writing a Dockerfile based on Debian or Ubuntu because those are what they're familiar with. In most cases it's worth the one-time cost of pulling the bigger base image, but it all depends on the use case. Another common reason for large images is the Nvidia ecosystem. Most of the images that Nvidia publish can get close to 1GB. However, these should only need to be pulled to devices once until you need to upgrade the base image.
zerotolerance 7y ago
Docker images are designed for minimal artifact transport while avoiding the need for assembly from source at the edge. If anyone is pushing around 1GB images for iterative deployments they might be able to do better with a few tweaks to their image structure and delivery pipeline.
deleted 7y ago
[deleted]
joshwget 7y ago
Can you talk more about what kind of use case you had where the average update took 1GB? I can certainly see how pulling a base image could require that kind of download size, but subsequent updates should only need to pull the layers that have changed.
nickysielicki 7y ago
There are two existing update frameworks that work on this model, one based on ostree and another from Intel called swupd.
eystein 7y ago
I'm involved with Mender which focuses on OTA updates and does not limit the type of update you can deploy, some popular choices: * full system image (with robust a/b layout for rollback support, can also do delta updates to reduce bandwidth with 70-90%) * docker containers * single files or directories * deb packages You can see the ones OOTB here: https://hub.mender.io/c/update-modules https://hub.mender.io/c/update-modules Also, you can write your own update type if you have a more custom use case: https://docs.mender.io/2.2/devices/update-modules https://docs.mender.io/2.2/devices/update-modules Does that make sense? Git and buildpack support is not yet OOTB but should be fairly simple to add with a new update module. Note however, that those type of updates may not be easy to roll back (or make atomic - meaning you can get partially installed updates and bricked devices/applications if you lose power).

yjftsjthsd-h 7y ago

> It's not really ideal in a number of installations to be pulling down 1GB Docker images simultaneously from 20 devices when you push an update. That seems like a really big image? Alpine-based images in my experience tend to be under 100MB. But I would be fascinated to use p2p for distributing image layers; it feels like that shouldn't be too hard? EDIT: it's been done, apparently:) https://blog.bonner.is/docker-registry-for-ipfs/ https://blog.bonner.is/docker-registry-for-ipfs/

zerotolerance 7y ago
IPFS and decentralization don't really address artifact size issues.
- yjftsjthsd-h 7y ago
  It certainly improves the practical outcome; replicating layers between nodes in a rack/datacenter is way faster than pulling it down 20x from one external upstream.
  - zerotolerance 7y ago
    Sure, but latency is not the same as size. You're talking about latency optimization via local caching. IPFS is a bit of a heavy solution compared to a simple cache. IPFS solves a totally different problem.
joshwget 7y ago
We see Alpine used in production less than most people think. People usually feel more comfortable writing a Dockerfile based on Debian or Ubuntu because those are what they're familiar with. In most cases it's worth the one-time cost of pulling the bigger base image, but it all depends on the use case. Another common reason for large images is the Nvidia ecosystem. Most of the images that Nvidia publish can get close to 1GB. However, these should only need to be pulled to devices once until you need to upgrade the base image.

zerotolerance 7y ago

Docker images are designed for minimal artifact transport while avoiding the need for assembly from source at the edge. If anyone is pushing around 1GB images for iterative deployments they might be able to do better with a few tweaks to their image structure and delivery pipeline.

deleted 7y ago

[deleted]

joshwget 7y ago

Can you talk more about what kind of use case you had where the average update took 1GB? I can certainly see how pulling a base image could require that kind of download size, but subsequent updates should only need to pull the layers that have changed.

nickysielicki 7y ago

There are two existing update frameworks that work on this model, one based on ostree and another from Intel called swupd.

eystein 7y ago

I'm involved with Mender which focuses on OTA updates and does not limit the type of update you can deploy, some popular choices: * full system image (with robust a/b layout for rollback support, can also do delta updates to reduce bandwidth with 70-90%) * docker containers * single files or directories * deb packages You can see the ones OOTB here: https://hub.mender.io/c/update-modules https://hub.mender.io/c/update-modules Also, you can write your own update type if you have a more custom use case: https://docs.mender.io/2.2/devices/update-modules https://docs.mender.io/2.2/devices/update-modules Does that make sense? Git and buildpack support is not yet OOTB but should be fairly simple to add with a new update module. Note however, that those type of updates may not be easy to roll back (or make atomic - meaning you can get partially installed updates and bricked devices/applications if you lose power).