4 ms·

Device fleets are incredibly hard to get right, and if you have no updateability, you have to nail them the first time. For someone coming from a cloud backgrou

Device fleets are incredibly hard to get right, and if you have no updateability, you have to nail them the first time. For someone coming from a cloud background, it's a jungle out there, where all sorts of "easy" or "solved" problems are nothing of the sort. As a sort of PSA, I want to plug the work that we have done at balena.io (formerly known as resin.io) to solve problems like these for everyone deploying Linux IoT devices, including open source / non-paying users. We've built the open source balenaOS[1] and the (newly) open sourced openBalena[2][3] management server that anyone can use without paying us a penny. If you're about to manufacture a fleet of linux devices, and are about to put them out there, and don't want to pay for a service (so you're not using balenaCloud) but also don't want to solve problems we've spent years already finetuning, (such as ntp, dns, cellular modem support, read-only filesystem, host updates, etc etc) use the above projects and save yourself a mountain of pain in the ensuing years. I hate being alarmist, but working with device fleets over the last 7 years, I've seen things you wouldn't believe. I've seen devices catch on fire, drown in storms, launch DDoS attacks, and everything in between. It's incredibly hard to underestimate how bad things can get when you put your devices out there with no ability to find them and update them if push comes to shove. [1] balenaOS: https://www.balena.io/os/ https://www.balena.io/os/ [2] openBalena: https://www.balena.io/open/ https://www.balena.io/open/ [3] openBalena github: https://github.com/balena-io/open-balena https://github.com/balena-io/open-balena

pdimitar 8y ago
What kinds of algorigthms you guys use to coordinate fleets? Also, your story is quite fascinating, can you tell us more? I am very, and I mean VERY interested in working on something like that, but not sure how to transition out of the generic CRUD web programmer to your area... :(
- alexandros 8y ago
  To be honest, with fleets in the wild, simple works. We do deploy containers to these devices, but rather than kubernetes-ing the whole thing, it's more like "every device in the fleet must run this container". In IoT/Edge affinity is a big deal, so the payloads are pretty uniform. You don't want your container to migrate away to another device while you're trying to use the device to e.g. fly a drone or open a lock. The difficulty comes in configuration, where things are much more complex than the cloud, since every device is "special". Different customers, keys, settings of all sorts. So we allow individual environment variable settings for each device. The real complexity is in ensuring you can reach every device as long as possible, and that the devices behave well when they can't reach the cloud. We set up a VPN to ensure the former, and have a pull-based architecture where the device is responsible for "catching up" to the fleet when it comes back online to handle the latter. There's a bunch of other things we've solved, like container deltas to improve download times by something like 10-70 times (and reduce bandwidth costs) but honestly the biggest difficulty has been in integrating cloud/hardware/developer/network/OS etc seamlessly, so people can succeed at building fleets, and fast. It's like there's 10,000 papercuts moreso than one or two big problems to solve. It's a long chain of things that all have to work right to make the system work, and each one has to be done in a way so that it doesn't break after some time passes (see the OP for an example). To answer your last question with (yet another) plug, you can probably run through our getting started guide in an hour and have your first device that you can deploy a js project to pretty fast. If you spend a few days, you can start to have some real accomplishments. It's not that hard to make the jump these days if you have a web background (hell, most of the founders didn't have IoT experience when we started) but there are things you need to learn that relate to hardware and linux if you want to do something more advanced. You can learn at your pace though, and I thoroughly recommend starting with a project you enjoy, something like, but not necessarily https://www.balena.io/blog/make-a-web-frame-with-raspberry-pi-in-30-minutes/ https://www.balena.io/blog/make-a-web-frame-with-raspberry-p...
zestyping 8y ago
This looks really cool. I may be deploying a fleet of Raspberry Pi or similar servers in the near future and this might save us a ton of headaches. The target context is a medical facility in a remote location without Internet (or extremely limited Internet), though. Is there a way to distribute updates by USB stick? What's the best place to read stories of people who have tried deploying with Balena on RasPi-type servers and/or contexts without Internet access, and learn about their successes and difficulties? Thanks!
- walrus01 8y ago
  please do yourself a favor and don't remotely deploy rpi or anything that uses microsd cards for OS image and boot. The write lifetime of those, even using "industrial" cards, will mean a huge failure rate even with very light usage. Only use some sort of real sata3 or nvme bus SSD.
  - sdenton4 8y ago
    There are ways to reduce the risk, though, depending on your workload. It is possible to use the sd card just to boot, with all actual file systems in a USB attached device. You can also use some of the (very limited) memory as a ram disk for things with lots of writes...
    - walrus01 8y ago
      Yes, but even so, the write wear leveling algorithm in a $50 consumer class SSD and dwpd (drive writes per day) are vastly better than microsd media.
      - milesvp 8y ago
        Will second this. A friend of mine had to write his own wear leveling software to make use of SD cards in his hardware, sadly bom made anything better way too expensive to be commercially viable.
Game_Ender 8y ago
I have looked at your stack and it’s very cool. One thing I was most curious about is the long term plan for your docker fork. Are you going to upstream changes or APIs so you don’t have to maintain the fork? The other question I have is about the stability of docker itself at scale. I worry about the reliability of essentially developer/server tooling in an embedded space where there is no ops team to help it along. Have you had any issues?
- alexandros 8y ago
  Hey there -- so this is a bit of a terminological thing, but we don't consider balenaEngine to be a docker fork -- it's a sibling project descending from the Moby Project, just like Docker, and of course highly copmpatible with Docker. The things we've done in balenaEngine aren't things that Docker would do. Shrinking the size to a quarter, removing Swarm, adding container deltas, changing the tradeoffs around durability -- most of the changes we have made relate to the embedded use case. As such, we intend to keep evolving balenaEngine alongside Docker, also because it's important for us to control the size of the binary. It goes into our root partition, and if Docker keeps adding things to their engine, we can't have that variable affect our ability to update devices. We've done a lot of things to make sure balenaEngine is rock solid in an embedded environment, and our fleetOps team is pretty hardcore in helping customers get out of hard situations (which help us further evolve the OS and the engine to avoid those by default).
solarkraft 8y ago
> If you have no updateability, you have to nail them the first time I would go a bit more radical: I'd say you'd have a problem.
- mikepurvis 8y ago
  Seriously. I do robots (OTTO Motors) and I just can't imagine shipping _anything_ to the field that I didn't have at least some plan for how to keep up to date. Even if the plan was the Panavision model of only ever leasing hardware and having all hardware periodically make its way back to the factory for updates and refurbishing, that's still more of a plan than a bunch of unidentifiable devices floating around at customer sites.
antonvs 8y ago
> I've seen things you wouldn't believe. I've seen devices catch on fire ...off the shoulder of Orion?
- Terr_ 8y ago
  Like bad enclosures in rain.
voltagex_ 8y ago
Hey, I wish the Pi foundation would take up https://github.com/balena-io/wifi-connect https://github.com/balena-io/wifi-connect - I thought about doing something like this ages ago but never got around to it. Thanks for open sourcing it.
- alexandros 8y ago
  This one is one of those things that took us so many iterations to nail. Very happy it finally got to a good place. We'll integrate it into balenaOS soon by default, hopefully.

pdimitar 8y ago

What kinds of algorigthms you guys use to coordinate fleets? Also, your story is quite fascinating, can you tell us more? I am very, and I mean VERY interested in working on something like that, but not sure how to transition out of the generic CRUD web programmer to your area... :(

alexandros 8y ago
To be honest, with fleets in the wild, simple works. We do deploy containers to these devices, but rather than kubernetes-ing the whole thing, it's more like "every device in the fleet must run this container". In IoT/Edge affinity is a big deal, so the payloads are pretty uniform. You don't want your container to migrate away to another device while you're trying to use the device to e.g. fly a drone or open a lock. The difficulty comes in configuration, where things are much more complex than the cloud, since every device is "special". Different customers, keys, settings of all sorts. So we allow individual environment variable settings for each device. The real complexity is in ensuring you can reach every device as long as possible, and that the devices behave well when they can't reach the cloud. We set up a VPN to ensure the former, and have a pull-based architecture where the device is responsible for "catching up" to the fleet when it comes back online to handle the latter. There's a bunch of other things we've solved, like container deltas to improve download times by something like 10-70 times (and reduce bandwidth costs) but honestly the biggest difficulty has been in integrating cloud/hardware/developer/network/OS etc seamlessly, so people can succeed at building fleets, and fast. It's like there's 10,000 papercuts moreso than one or two big problems to solve. It's a long chain of things that all have to work right to make the system work, and each one has to be done in a way so that it doesn't break after some time passes (see the OP for an example). To answer your last question with (yet another) plug, you can probably run through our getting started guide in an hour and have your first device that you can deploy a js project to pretty fast. If you spend a few days, you can start to have some real accomplishments. It's not that hard to make the jump these days if you have a web background (hell, most of the founders didn't have IoT experience when we started) but there are things you need to learn that relate to hardware and linux if you want to do something more advanced. You can learn at your pace though, and I thoroughly recommend starting with a project you enjoy, something like, but not necessarily https://www.balena.io/blog/make-a-web-frame-with-raspberry-pi-in-30-minutes/ https://www.balena.io/blog/make-a-web-frame-with-raspberry-p...

zestyping 8y ago

This looks really cool. I may be deploying a fleet of Raspberry Pi or similar servers in the near future and this might save us a ton of headaches. The target context is a medical facility in a remote location without Internet (or extremely limited Internet), though. Is there a way to distribute updates by USB stick? What's the best place to read stories of people who have tried deploying with Balena on RasPi-type servers and/or contexts without Internet access, and learn about their successes and difficulties? Thanks!

walrus01 8y ago
please do yourself a favor and don't remotely deploy rpi or anything that uses microsd cards for OS image and boot. The write lifetime of those, even using "industrial" cards, will mean a huge failure rate even with very light usage. Only use some sort of real sata3 or nvme bus SSD.
- sdenton4 8y ago
  There are ways to reduce the risk, though, depending on your workload. It is possible to use the sd card just to boot, with all actual file systems in a USB attached device. You can also use some of the (very limited) memory as a ram disk for things with lots of writes...
  - walrus01 8y ago
    Yes, but even so, the write wear leveling algorithm in a $50 consumer class SSD and dwpd (drive writes per day) are vastly better than microsd media.
    - milesvp 8y ago
      Will second this. A friend of mine had to write his own wear leveling software to make use of SD cards in his hardware, sadly bom made anything better way too expensive to be commercially viable.

Game_Ender 8y ago

I have looked at your stack and it’s very cool. One thing I was most curious about is the long term plan for your docker fork. Are you going to upstream changes or APIs so you don’t have to maintain the fork? The other question I have is about the stability of docker itself at scale. I worry about the reliability of essentially developer/server tooling in an embedded space where there is no ops team to help it along. Have you had any issues?

alexandros 8y ago
Hey there -- so this is a bit of a terminological thing, but we don't consider balenaEngine to be a docker fork -- it's a sibling project descending from the Moby Project, just like Docker, and of course highly copmpatible with Docker. The things we've done in balenaEngine aren't things that Docker would do. Shrinking the size to a quarter, removing Swarm, adding container deltas, changing the tradeoffs around durability -- most of the changes we have made relate to the embedded use case. As such, we intend to keep evolving balenaEngine alongside Docker, also because it's important for us to control the size of the binary. It goes into our root partition, and if Docker keeps adding things to their engine, we can't have that variable affect our ability to update devices. We've done a lot of things to make sure balenaEngine is rock solid in an embedded environment, and our fleetOps team is pretty hardcore in helping customers get out of hard situations (which help us further evolve the OS and the engine to avoid those by default).

solarkraft 8y ago

> If you have no updateability, you have to nail them the first time I would go a bit more radical: I'd say you'd have a problem.

mikepurvis 8y ago
Seriously. I do robots (OTTO Motors) and I just can't imagine shipping _anything_ to the field that I didn't have at least some plan for how to keep up to date. Even if the plan was the Panavision model of only ever leasing hardware and having all hardware periodically make its way back to the factory for updates and refurbishing, that's still more of a plan than a bunch of unidentifiable devices floating around at customer sites.

antonvs 8y ago

> I've seen things you wouldn't believe. I've seen devices catch on fire ...off the shoulder of Orion?

Terr_ 8y ago
Like bad enclosures in rain.

voltagex_ 8y ago

Hey, I wish the Pi foundation would take up https://github.com/balena-io/wifi-connect https://github.com/balena-io/wifi-connect - I thought about doing something like this ages ago but never got around to it. Thanks for open sourcing it.

alexandros 8y ago
This one is one of those things that took us so many iterations to nail. Very happy it finally got to a good place. We'll integrate it into balenaOS soon by default, hopefully.