9 ms·

Serverless Raspberry Pi Cluster with Docker

247 points by alexellisuk 9y ago

zxcmx 9y ago
This is super awesome and fun but personally I had to migrate my micro datacenter from pis to nucs. The "armhf tax" is that you tend to have to build your own images for stuff :( Then you need your own build infra (or "heath robinson" qemu builds) because pis run out of memory building a lot of stuff... but mainly if C++ is involved so ymmv. That said, I got a rack of 8 pis doing nothing right now, so... (unrelated http://www.bitscope.com/product/BB04/ http://www.bitscope.com/product/BB04/ is handy if you want to rack a lotta pis, not affliated...) There is probably a micro business for someone running a slick docker build system for armhf handling the qemu emulation or toolchain dirtiness "under the hood" in the cloud somewhere, on x86-64 boxes with a lot more than 1GB of RAM.
- rubenbe 9y ago
  Can you elaborate on your NUC setup? Which NUC did you choose? I have a similar RPI rack collecting dust for the same reason, hence the question.
- jacobush 9y ago
  You could try running a Raspberry but add a Network Block device from another machine and. (xNBD: https://bitbucket.org/hirofuchi/xnbd/wiki/Home https://bitbucket.org/hirofuchi/xnbd/wiki/Home ) Then export a RAM disk from that machine and add the NBD disk as swap on the Raspberry. It would be slow, but builds would complete. Then you'd need only one low-to-moderate power machine (a PC presumably) in your Raspberry cluster, just with lots of RAM in the one PC.
  - Artemis2 9y ago
    Scaleway has a good array of ARM cloud offerings: https://www.scaleway.com/armv8-cloud-servers/ https://www.scaleway.com/armv8-cloud-servers/ Could this work for speeding up builds of ARM images and then deploying locally?
    - alexellisuk 9y ago
      Packet.net can go one further - take those 8 cores and upgrade to 96 cores and 120GB RAM.. it's ARMv8 which will be next for OpenFaaS when the Docker support catches up :-)
- omgwtfbyobbq 9y ago
  I've built ABS (archlinux) packages by having some swap space on my pogoplug mobile's boot drive (1TB USB). It's a slog, but stuff will finish sooner or later, and by that I mean later or really late. Now that I'm thinking about it, I'd like to see if going from the RPI's USB3->SATA adapter->M.2 adapter->16GB of Optane ($40+tax locally) would work, and if it did work (a big if), what performance is like. Edit - Scratch that, I just remembered the Pi 3 is still USB 2.
syvanen 9y ago
So this blog instructs to install commercial docker instead of moby. I wonder how the licensing goes if you were to use this.
- filipn 9y ago
  I believe it actually installs the Docker community edition (CE) which is under the Apache 2.0 license. Docker (the product) is assembled using the Moby libraries and components.
  - alexellisuk 9y ago
    Correct, Moby is not a distribution of a container runtime.. it's not the "docker" you are looking for. Docker CE is and that's what's used in the guide.
  - maniktan 9y ago
    Full Disclosure: I, Manik Taneja, am the Product Manager for all the open source efforts at Docker and work on the Moby Project. As suggested here, the blog post only talks about installing the Docker Community Edition (CE) that is published under Apache 2.0 License. This is the official incarnation of the Docker Product and is provided to have: - a consistent user experience across different linux distributions - strong security guarantees - regular bug fixes and updates Moby Project serves as the upstream for the entire Docker Product and includes all open source components that make up Docker, such as runc, containerd, notary, moby, infrakit, linuxkit, libnetwork, hyperkit, vpnkit, datakit, etc.
- zeta0134 9y ago
  I imagine the licensing goes something like this: https://www.docker.com/components-licenses https://www.docker.com/components-licenses It's my understanding that the bulk of the components that make Docker work are fully open source, and some extra support and deployment related things are the only things that are commercially licensed. This would include Docker Swarm, as it's a part of Docker itself and not something separate. IANAL though. I'm pretty sure the reason they go for plain Docker over Moby is sheer ease of use. Despite being a bit weird to understand under the hood, Docker is just dead simple to get up and running with, and using the clustering mode that's built right in is easier to teach new readers than Moby, which from its Github page is obviously designed for folks that are already rather comfortable with Docker. Straight from Moby's Github page: "Moby is NOT recommended for: Application developers looking for an easy way to run their applications in containers. We recommend Docker CE instead." https://github.com/moby/moby https://github.com/moby/moby
elcapitan 9y ago
This may be a stupid question, but is there an actual use case for building clusters of raspberry pi's?
- zeta0134 9y ago
  Part of me wants to believe that a solar powered raspberry pi cluster has a legitimate use case. The things use like ~1W of power, so the board itself is surprisingly efficient compared to a "real" server. But really, I think the primary use case is cost. Actually having access to that many physical machines to play with in a classroom or home learning environment is sort of new! The market hasn't really had such accessible linux computers at "Ehh, if it breaks I'll just buy a new one, no big deal" prices. It's educational, and the more stable the ARM support is, the better a student's skills will transfer over into the real world of systems administration.
  - creshal 9y ago
    > The things use like ~1W of power, so the board itself is surprisingly efficient compared to a "real" server. Try 3.5 watts[1], not counting overhead of most USB power bricks being incredibly inefficient. A current-gen 35W laptop CPU will be some 10 times faster[2] as a RasPi, have much faster storage available (SATA3 or NVMe versus… USB2), much faster I/O (GBit LAN and GBit Wifi versus… USB2), and a lot of other benefits. (Like an integrated screen and battery and keyboard and …) It also won't need external hardware to communicate with other cluster members – that 10-port ethernet switch will need power, too. One RasPi is relatively energy efficient; RasPi clusters… not so much. > But really, I think the primary use case is cost. Indeed. [1] http://raspi.tv/2016/how-much-power-does-raspberry-pi3b-use-how-fast-is-it-compared-to-pi2b http://raspi.tv/2016/how-much-power-does-raspberry-pi3b-use-... , see the numbers for "Multi-threaded CPU Tests", which is the most applicable for server workloads [2] Running that script manages ~9 runs/second on an i7-6700HQ, vs. ~0.9 run/second on a RPi3.
    - marmaduke 9y ago
      If you had ten of them, ideally you'd use a USB power supply where that overhead is a much lower percentage.
mrmondo 9y ago
> 'Serverless is an architectural pattern resulting in: Functions as a Service, or FaaS' Then call it FaaS - it's not serverless and that term is misleading, marketing bunk. Even the title of the post describes the servers used in it's deployment, I'm quite sick of this term - I don't find it at all helpful when describing application architecture.
- alexellisuk 9y ago
  It is what it is - and I put a paragraph at the start of the blog for folks who are still confused. It explains that Serverless is a pattern and not a literal term. I even wrote a blog post about it - we've done such a poor job in this industry of explaining it. https://blog.alexellis.io/introducing-functions-as-a-service/ https://blog.alexellis.io/introducing-functions-as-a-service...
  - senko 9y ago
    Maybe if a better term is used it wouldn't need so much explaining. To me, "Functions as a Service" is massively more obvious than "Serverless".
    - xupybd 9y ago
      And the cloud would be better as a server somewhere on the internet but terms like this get created because it's easier to sell a one word concept.
      - alexellisuk 9y ago
        Yes... can we all start berating every blog post that uses the word "cloud"? There are no clouds in this server! :-D
        deleted 9y ago
        [deleted]
    - DJHenk 9y ago
metakermit 9y ago
If you still like going "serverful", but want an easy method of deploying to your Raspberry Pi using Docker, check out this open source Docker Hub for ARM alternative https://marina.io/ https://marina.io/ (Full disclosure: I am one of the authors)
- alexellisuk 9y ago
  Marina looks very cool and I'll take a look at it soon. Let's get in contact somehow.
  - metakermit 9y ago
    Sure, that would be great. Yeah you have our email and other contacts on – https://cloudfleet.io/ https://cloudfleet.io/ or mine directly on https://metakermit.com/ https://metakermit.com/
imtringued 9y ago
I'm surprised that the raspberry pi monoculture prevailed even until today.
gkya 9y ago
Does anybody really use this FaaS thing? This is the weirdest of the New Things I've seen recently.
- bananarepdev 9y ago
  Like most new tech, it has it's applicability. Depending on your needs, and the characteristics of your application load, it can save you money.
  - dvfjsdhgfv 9y ago
    Could you give an example of a scenario where it can actually save me money?
    - jclulow 9y ago
      I suspect it makes economic sense only when your workload is relatively elastic, having a relatively low duty cycle. The ability to pay less when you aren't actually using any resources is likely of more economic benefit as it becomes more fine-grained. If you aren't in that position, other models of lease or ownership of computing resources probably merit consideration.
      - dvfjsdhgfv 9y ago
        Well, to be honest I pay 40€/month for a baremetal server (i7 SkyLake, 64 GB RAM, 4TB HDD). It's a powerful machine running many services, including a few virtual machines. I consider whatever it does would qualify as low-duty. Now, I know that each month I'm paying 40€ for all this. The last time about I read about serverless was when someone directly discovered the money saving part is quite tricky: https://news.ycombinator.com/item?id=14982220 https://news.ycombinator.com/item?id=14982220.
    - Prefinem 9y ago
      When the cost of a Dev Ops or Sys Admin is more than the service.
lngnmn 9y ago
Serverless meme means that all I got is a chrooted directory with only systemd and /lib but without signals, similar to an apache vhost with a cgi-bin, so I could run my full-stack crap and it is supposed to be so damn cool because actual server maintenance is now someone else's problem?
Hortinstein 9y ago
great article, been a huge fan of your stuff in the past. Helped me get some good ideas of things to use my Pi and Clusterhat with. https://clusterhat.com https://clusterhat.com

247 points by alexellisuk 9y ago

This is super awesome and fun but personally I had to migrate my micro datacenter from pis to nucs. The "armhf tax" is that you tend to have to build your own images for stuff :( Then you need your own build infra (or "heath robinson" qemu builds) because pis run out of memory building a lot of stuff... but mainly if C++ is involved so ymmv. That said, I got a rack of 8 pis doing nothing right now, so... (unrelated http://www.bitscope.com/product/BB04/ http://www.bitscope.com/product/BB04/ is handy if you want to rack a lotta pis, not affliated...) There is probably a micro business for someone running a slick docker build system for armhf handling the qemu emulation or toolchain dirtiness "under the hood" in the cloud somewhere, on x86-64 boxes with a lot more than 1GB of RAM.

rubenbe 9y ago
Can you elaborate on your NUC setup? Which NUC did you choose? I have a similar RPI rack collecting dust for the same reason, hence the question.
jacobush 9y ago
You could try running a Raspberry but add a Network Block device from another machine and. (xNBD: https://bitbucket.org/hirofuchi/xnbd/wiki/Home https://bitbucket.org/hirofuchi/xnbd/wiki/Home ) Then export a RAM disk from that machine and add the NBD disk as swap on the Raspberry. It would be slow, but builds would complete. Then you'd need only one low-to-moderate power machine (a PC presumably) in your Raspberry cluster, just with lots of RAM in the one PC.
- Artemis2 9y ago
  Scaleway has a good array of ARM cloud offerings: https://www.scaleway.com/armv8-cloud-servers/ https://www.scaleway.com/armv8-cloud-servers/ Could this work for speeding up builds of ARM images and then deploying locally?
  - alexellisuk 9y ago
    Packet.net can go one further - take those 8 cores and upgrade to 96 cores and 120GB RAM.. it's ARMv8 which will be next for OpenFaaS when the Docker support catches up :-)
omgwtfbyobbq 9y ago
I've built ABS (archlinux) packages by having some swap space on my pogoplug mobile's boot drive (1TB USB). It's a slog, but stuff will finish sooner or later, and by that I mean later or really late. Now that I'm thinking about it, I'd like to see if going from the RPI's USB3->SATA adapter->M.2 adapter->16GB of Optane ($40+tax locally) would work, and if it did work (a big if), what performance is like. Edit - Scratch that, I just remembered the Pi 3 is still USB 2.

syvanen 9y ago

So this blog instructs to install commercial docker instead of moby. I wonder how the licensing goes if you were to use this.

filipn 9y ago
I believe it actually installs the Docker community edition (CE) which is under the Apache 2.0 license. Docker (the product) is assembled using the Moby libraries and components.
- alexellisuk 9y ago
  Correct, Moby is not a distribution of a container runtime.. it's not the "docker" you are looking for. Docker CE is and that's what's used in the guide.
- maniktan 9y ago
  Full Disclosure: I, Manik Taneja, am the Product Manager for all the open source efforts at Docker and work on the Moby Project. As suggested here, the blog post only talks about installing the Docker Community Edition (CE) that is published under Apache 2.0 License. This is the official incarnation of the Docker Product and is provided to have: - a consistent user experience across different linux distributions - strong security guarantees - regular bug fixes and updates Moby Project serves as the upstream for the entire Docker Product and includes all open source components that make up Docker, such as runc, containerd, notary, moby, infrakit, linuxkit, libnetwork, hyperkit, vpnkit, datakit, etc.
zeta0134 9y ago
I imagine the licensing goes something like this: https://www.docker.com/components-licenses https://www.docker.com/components-licenses It's my understanding that the bulk of the components that make Docker work are fully open source, and some extra support and deployment related things are the only things that are commercially licensed. This would include Docker Swarm, as it's a part of Docker itself and not something separate. IANAL though. I'm pretty sure the reason they go for plain Docker over Moby is sheer ease of use. Despite being a bit weird to understand under the hood, Docker is just dead simple to get up and running with, and using the clustering mode that's built right in is easier to teach new readers than Moby, which from its Github page is obviously designed for folks that are already rather comfortable with Docker. Straight from Moby's Github page: "Moby is NOT recommended for: Application developers looking for an easy way to run their applications in containers. We recommend Docker CE instead." https://github.com/moby/moby https://github.com/moby/moby

elcapitan 9y ago

This may be a stupid question, but is there an actual use case for building clusters of raspberry pi's?

zeta0134 9y ago
Part of me wants to believe that a solar powered raspberry pi cluster has a legitimate use case. The things use like ~1W of power, so the board itself is surprisingly efficient compared to a "real" server. But really, I think the primary use case is cost. Actually having access to that many physical machines to play with in a classroom or home learning environment is sort of new! The market hasn't really had such accessible linux computers at "Ehh, if it breaks I'll just buy a new one, no big deal" prices. It's educational, and the more stable the ARM support is, the better a student's skills will transfer over into the real world of systems administration.
- creshal 9y ago
  > The things use like ~1W of power, so the board itself is surprisingly efficient compared to a "real" server. Try 3.5 watts[1], not counting overhead of most USB power bricks being incredibly inefficient. A current-gen 35W laptop CPU will be some 10 times faster[2] as a RasPi, have much faster storage available (SATA3 or NVMe versus… USB2), much faster I/O (GBit LAN and GBit Wifi versus… USB2), and a lot of other benefits. (Like an integrated screen and battery and keyboard and …) It also won't need external hardware to communicate with other cluster members – that 10-port ethernet switch will need power, too. One RasPi is relatively energy efficient; RasPi clusters… not so much. > But really, I think the primary use case is cost. Indeed. [1] http://raspi.tv/2016/how-much-power-does-raspberry-pi3b-use-how-fast-is-it-compared-to-pi2b http://raspi.tv/2016/how-much-power-does-raspberry-pi3b-use-... , see the numbers for "Multi-threaded CPU Tests", which is the most applicable for server workloads [2] Running that script manages ~9 runs/second on an i7-6700HQ, vs. ~0.9 run/second on a RPi3.
  - marmaduke 9y ago
    If you had ten of them, ideally you'd use a USB power supply where that overhead is a much lower percentage.

mrmondo 9y ago

> 'Serverless is an architectural pattern resulting in: Functions as a Service, or FaaS' Then call it FaaS - it's not serverless and that term is misleading, marketing bunk. Even the title of the post describes the servers used in it's deployment, I'm quite sick of this term - I don't find it at all helpful when describing application architecture.

alexellisuk 9y ago
It is what it is - and I put a paragraph at the start of the blog for folks who are still confused. It explains that Serverless is a pattern and not a literal term. I even wrote a blog post about it - we've done such a poor job in this industry of explaining it. https://blog.alexellis.io/introducing-functions-as-a-service/ https://blog.alexellis.io/introducing-functions-as-a-service...
- senko 9y ago
  Maybe if a better term is used it wouldn't need so much explaining. To me, "Functions as a Service" is massively more obvious than "Serverless".
  - xupybd 9y ago
    And the cloud would be better as a server somewhere on the internet but terms like this get created because it's easier to sell a one word concept.
    - alexellisuk 9y ago
      Yes... can we all start berating every blog post that uses the word "cloud"? There are no clouds in this server! :-D
      - deleted 9y ago
        [deleted]
  - DJHenk 9y ago

metakermit 9y ago

If you still like going "serverful", but want an easy method of deploying to your Raspberry Pi using Docker, check out this open source Docker Hub for ARM alternative https://marina.io/ https://marina.io/ (Full disclosure: I am one of the authors)

alexellisuk 9y ago
Marina looks very cool and I'll take a look at it soon. Let's get in contact somehow.
- metakermit 9y ago
  Sure, that would be great. Yeah you have our email and other contacts on – https://cloudfleet.io/ https://cloudfleet.io/ or mine directly on https://metakermit.com/ https://metakermit.com/

imtringued 9y ago

I'm surprised that the raspberry pi monoculture prevailed even until today.

gkya 9y ago

Does anybody really use this FaaS thing? This is the weirdest of the New Things I've seen recently.

bananarepdev 9y ago
Like most new tech, it has it's applicability. Depending on your needs, and the characteristics of your application load, it can save you money.
- dvfjsdhgfv 9y ago
  Could you give an example of a scenario where it can actually save me money?
  - jclulow 9y ago
    I suspect it makes economic sense only when your workload is relatively elastic, having a relatively low duty cycle. The ability to pay less when you aren't actually using any resources is likely of more economic benefit as it becomes more fine-grained. If you aren't in that position, other models of lease or ownership of computing resources probably merit consideration.
    - dvfjsdhgfv 9y ago
      Well, to be honest I pay 40€/month for a baremetal server (i7 SkyLake, 64 GB RAM, 4TB HDD). It's a powerful machine running many services, including a few virtual machines. I consider whatever it does would qualify as low-duty. Now, I know that each month I'm paying 40€ for all this. The last time about I read about serverless was when someone directly discovered the money saving part is quite tricky: https://news.ycombinator.com/item?id=14982220 https://news.ycombinator.com/item?id=14982220.
  - Prefinem 9y ago
    When the cost of a Dev Ops or Sys Admin is more than the service.

lngnmn 9y ago

Serverless meme means that all I got is a chrooted directory with only systemd and /lib but without signals, similar to an apache vhost with a cgi-bin, so I could run my full-stack crap and it is supposed to be so damn cool because actual server maintenance is now someone else's problem?

Hortinstein 9y ago

great article, been a huge fan of your stuff in the past. Helped me get some good ideas of things to use my Pi and Clusterhat with. https://clusterhat.com https://clusterhat.com