10 ms·

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

470 points by alexellisuk 7y ago

SirMonkey 7y ago
I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ https://cilium.io/ (Networking) https://rook.io/ https://rook.io/ (Persistent Storage)
- salamander014 7y ago
  I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router (https://kube-router.io https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (currently running Canal). Not sure what we will decide on when we go to production.
  - fyfy18 7y ago
    I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device. Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It hasn't lost data though, even though one of the SSDs died and was unrecoverable, so props for that. To be fair I haven't tried upgrading since I installed it so maybe my issues would be resolved by that. Overall Rook feels somewhat overkill for a homelab user, but I haven't seen any recommended alternatives.
    - Nullabillity 7y ago
      You need to set a memory target for your OSDs, or they will gobble up as much as possible for their cache. In Rook you do that by specifying the resource limits on your CephCluster[0]. Sadly, Rook enforces some pretty high minimum values (2GiB per OSD, 1 GiB per mon), which is fine for production but can make homelab experimentation annoying. [0]: https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster-wide-resources-configuration-settings https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster...
- bogomipz 7y ago
  Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is that correct?
  - moondev 7y ago
    Not directly, rook provides a storage backend for in-cluster workload persistent volumes. In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.
    - ownagefool 7y ago
      I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume. You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.
      - moondev 7y ago
        The parent comment was asking about root volumes for kubernetes nodes > Does Rook give you the equivalent of EBS root volumes for your nodes then? I think we are saying the same thing if i'm not mistaken?
        ownagefool 7y ago
        I doubt the parent really cares about whether or not it's a root volume, but for the record you can mount it to root. It'll just essentially be empty, which means you have to add a step of populating root with something useful. You'd have to either try and use an initContainer to copy in a filesystem at runtime, have ceph give you a pre-populated directory, probably via thin provisioning, or do something out of band (I've did this in k8s). This is usually more effort than it's worth though, as container runtimes populate the root for you with whatever you want anyway. Plus, if you start treating containers as blobs like VMs, you'll end up in the situation where you don't know where your important variable data is, which leads to situations where people forget to back it up and test it.
pojntfx 7y ago
@alexellisuk almost all of my interest in Kubernetes is due to your work. Thank you for everything you do!
ricardbejarano 7y ago
Everything I know about Kubernetes I learned from a single-node (pseudo)clustered refurbished ThinkPad X201. It cost me 90€ at an IBM refurbished sell. It is downstairs by the router, and it has been hosting everything for me except my email and my blog (which I want to host but I'm not sure about the reliability of my ISP's service, this morning it went down for 5h12m, without prior notice or anything). It is amazing how much you learn from doing stuff. I'm currently on my 3rd year of univerity for CS, so I've tried the academia-style learning process, reading books on my own and doing stuff on my own. The latter is the best method by far.
- mpfundstein 7y ago
  I only really learn by doing stuff. I read books usually to get started but mostly quickly have to turn to building something. Then revisiting the book on the go. If I only read the book/paper/whatever, I usually ‘think’ I get it but nearly always that isn’t the case :-)
tyingq 7y ago
Thankfully these days, places like Digital Ocean and Linode have managed K8s where you only pay for the compute nodes, for $5/month each. So it's fairly cheap and easy to learn on a "real" cluster without having to build one.
- gatherhunterer 7y ago
  Building a cluster is a fun and relatively easy project. You learn much more by having the hardware at your fingertips. You can simulate network failures and power failures or you can crash an important daemon. By causing problems you can see how K8s responds and manages itself when it does not have the nodes it expects. It is important to know these things because, for example, if you create a pod instead of a replica set then the loss of a node will mean the inability to use the pod assigned to that node. You need to know how a pod and a replica set differ in order to create a self-healing stack. You can learn all of that with the cloud solutions but the ability to answer your own questions will always be the superior means of learning. The Agile movement has convinced many that the solution that gets you up and running today is the best. A good engineer is not afraid of working and learning instead of just buying a pre-baked implementation and calling themself an experienced user. The cluster that I know like the back of my hand is my “real” cluster. The production cluster is someone else’s that I just rent.
  - geerlingguy 7y ago
    Through my work on this project, I've learned a ton about kubectl, kubeadm, the control plane, etc. that I never learned when managing EKS clusters at a previous job. It's good to know more of the 'guts' of an insanely complex ecosystem like Kubernetes, which is why I often recommend people follow Kelsey Hightower's "Kubernetes the Hard Way" guide to install Kubernetes from primitives at least once.
  - tyingq 7y ago
    Sure. It's not necessarily skipping all those things you mention that's the motivation. Just the desired order and upfront effort to learn things about K8S.
  - 7y ago
raesene9 7y ago
A good method for learning how clusters work and playing with them without having to spend a lot of time re-building when you break things is kind (https://kind.sigs.k8s.io/ https://kind.sigs.k8s.io/). Each node is a Docker container, but the version of Kubernetes running inside it is vanilla Kubeadm, so it's quite representative of what "real" clusters would look like. The great thing about it is you can spin up a cluster in < 2 mins locally to try things out, then it's a single command to delete.
- geerlingguy 7y ago
  I use Kind (and Minikube, and a number of other solutions too), but this is kind of my "Kubernetes-the-really-hard-way" fun project. Note that I maintain a parallel configuration that runs a multi-node cluster on Vagrant for local development [1] as well as a docker environment built for CI purposes[2] using almost all the same images, with the same Ansible playbooks to configure them all across platforms. [1] https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/vagrant https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas... [2] https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/docker https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas...
  - raesene9 7y ago
    yeah I think this approach will have fewer limitations than kind, and also it's fun to try different approaches! BTW I think your ansible book is great, really helped me get to grips with it.
  - vlahmot 7y ago
    Thank you for all of your great Ansible work! I leverage a lot of them at work and they have been great! (I’ll be playing around with these!)
  - fizixer 7y ago
    Sorry, late to the party. Thanks for sharing. I fully agree with the fun, and the really-hard-way part. However, if we ignore that part for the moment, would you think having an R-Pi cluster handy is worth it for training/sandboxing/testing with the aim of applying the skills in your day job? Are there any advantages to having access to your own R-Pi kubernetes cluster, despite having stuff like Kind and Minikube available online?
jptoto 7y ago
I LOVE Jeff's work and I own his Ansible book. The setup is pretty awesome. FWIW I think you could do this with local VMs a little more easily and provision with Vagrant. Having said that, a cluster of Pis is super fun!
- geerlingguy 7y ago
  What do you know, that's exactly how I do local development work for the cluster ;) https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/vagrant https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas...
  - jptoto 7y ago
    Aha! Makes sense.
- JeremyNT 7y ago
  Yeah, using local vms is going to greatly expand access to this kind of learning. A current developer laptop is more than beefy enough, and I'd suggest that there aren't many useful lessons to learn from the pi hardware that can't be learned fully virtualized. But if you're having fun, of course, more power to you :)
Mountain_Skies 7y ago
Wonder if some old netbooks could be used for this purpose. Doubt I am the only one with a small pile of them laying around.
- ricardbejarano 7y ago
  From experience, anything with 2GB or more RAM can be a master node. Workers can even have 1GB and work just fine. Be warned, though, in my experience etcd requires a somewhat decent read/write latency, or else it's going to fail, and when etcd fails everything fails. Your changes don't apply, etc.
  - geerlingguy 7y ago
    _Technically_ 1GB is the limit, but yeah, at 1GB I was hitting OOM errors every day or so (this was on the Pi 3 B+). I was very happy when I found out the Pi 4 had 2 or 4GB models! No matter what, you should also make sure to not schedule Pods on the master node(s) when you have limited memory (or in most cases, really...).
    - ricardbejarano 7y ago
      Or set tight resource limits. Setting resource requests also enables the use of HorizontalPodAutoscalers.
      - geerlingguy 7y ago
        Very true. The Pi Dramble's web pods are set to scale using the HPA.
sogubsys 7y ago
Thanks Jeff!
vyshane 7y ago
3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-tower/ https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.
- geerlingguy 7y ago
  Very nice; I've considered doing something similar and running some production sites on it from my home, but the limitation has always been my terrible Internet bandwidth through Spectrum. We almost got Verizon gigabit fiber a few years ago... then AT&T ran fiber to the front of my neighborhood last year, and then never ran it to the houses. As it is, I'm stuck with 10 mbps uplink, which is not enough to be able to do most of what I would want to do with a more powerful local cluster.
  - deleted 7y ago
    [deleted]
- yumraj 7y ago
  This is very cool. Curious, roughly, how much did this setup cost?
- rwmj 7y ago
  Yours is a lot neater than the four node bare cluster I built a few years ago: https://rwmj.wordpress.com/2014/04/28/caseless-virtualization-cluster-part-5/ https://rwmj.wordpress.com/2014/04/28/caseless-virtualizatio... One issue with caseless machines is the amount of RF they emit. Makes it hard to listen to any broadcast radio near one and probably disturbs the neighbours similarly. I'm now using a cluster of NUCs which is considerably easier to deal with although quite a lot more expensive: https://rwmj.wordpress.com/2018/02/06/nuc-cluster-bring-up/ https://rwmj.wordpress.com/2018/02/06/nuc-cluster-bring-up/
kiseleon 7y ago
Just an aside, your parts list for the pi dramble has 4 Pi 4B's but micro usb power cables
- geerlingguy 7y ago
  Oops! I forgot to update that when I switched everything out for the 4 Bs. I'll update that in a little bit.
  - geerlingguy 7y ago
    It has been updated.
pstadler 7y ago
Great article! Never stop tinkering. Here‘s how I got to know Kubernetes: By end of 2016, the Kubernetes hype was just about to pick up real steam. As somebody who always liked the idea of running something like an own cluster in the cloud, I attended KubeCon Europe in early 2017. The event was sold out, and took place in a venue almost too small for the number of attendees. It was great. During the event I was just about to finish the Hobby Kube[1] project. Back then weren’t any guides that addressed all the problems encountered when running Kubernetes on a budget, using low cost VPS on random cloud providers. So I dived into the subject in the second half of 2016 and started writing a guide, including automated provisioning using Terraform. I discovered WireGuard in the process of looking for a solution to efficiently securing network traffic between hosts. This still makes me feel like I was an early adopter to something that’s becoming hugely popular. If somebody would like to add a Terraform module for deploying to Raspberry Pi, please ping me or open a PR here[2]. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning
Methusalah 7y ago
This seems like a fun project and a great way to learn Kubernetes, but if I'm dropping this much money on it, I'd like it to have some productive purpose afterwards. I'm a full-stack web dev primarily using node/react/postgres. I've also got some projects currently hosted on a Linode instance. Ideas on fun/productive uses for this cluster after I've built it and messed around with Kubernetes?
- birdyrooster 7y ago
  The Kubernetes ecosystem is evolving rapidly, you might want to keep it around to play with different CNIs, CSIs, service meshes, operators for clustered software lifecycle, and more.
- striking 7y ago
  Move the projects to your cluster and see what happens! Consider the fact that, if you make improvements to the cluster, all of your apps will see that same lift. So if you were to set up backups on the cluster's persistent volumes and its databases, you'd get free backups for all of the projects you've moved. Same with monitoring, autoscale, and so on.
- robgibbons 7y ago
  RPI's will handle Node surprisingly well, from my experience. A small cluster of them would be well suited to any number of web projects. A few years back, I played around with haproxy and a few Pi's running Node servers. You may be surprised how well they work as servers, as long as you're not expecting Xeon-level speeds.
acd 7y ago
Thanks Jeff for your great Ansible roles!
madrox 7y ago
This is also how my alma mater (Cal Poly SLO) teaches Hadoop. Building real world clusters are expensive, and giving each student their own is difficult. However, small clusters of Raspberry Pis are cheap, and it's also very easy to demonstrate how unplugging one affects the cluster.
segmondy 7y ago
You can also build a cluster on your computer if it's beefy enough by running multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few years ago and that's my k8s experiment land.
- MuffinFlavored 7y ago
  I don’t fully understand where the line goes from “all I need is a Docker container for nginx + Postgres + Redis + my services” to “I need Kubernetes” When does one need to go from just Docker containers to container orchestration like k8s?
  - Nullabillity 7y ago
    Do you care about availability? Do you care about rolling upgrades? Do you want to practice any of those two? Then it might be time to make the switch. Yes, Docker has Swarm Mode. And yes, it is easier to get started with than K8s. The problem is that it locks you into a lot of choices when you first spin up each (Docker) Service, that can't be changed without taking down the Service and then recreating it with the new option. Want to change how upgrades are rolled out? Too bad! Kubernetes still has a few settings like that, but since (K8s) Services are just responsible for routing traffic to separately-managed Pods, you can always fall back to rolling the upgrade manually, or doing a blue-green deployment.
    - MuffinFlavored 7y ago
      I do blue-green deploys with Docker and I deploy the services as “app-1”, “app-2”, “app-3”, etc.
      - Nullabillity 7y ago
        How do you route dependencies to the correct service? If you're using a stack then you end up needing to blue-green the whole thing, if you're not then you still need to blue-green the whole dependent subset of the dependency graph. And for anything exposed outside of the swarm you'd need to use some reverse proxy, since only one service (in the whole swarm) can bind a port at any one time.
ljm 7y ago
I suppose I was lucky dabbling in one of my side-projects because I had money to burn. I put it all in Google Cloud and then learned just how much you have to complicate the stack (beyond the complication of K8S) to lower infra costs. Suddenly I wasn't using basic Kubernetes any more, I was setting up a new ingress controller so Google wouldn't launch a new load balancer instance for every public service I exposed. Those things aren't cheap. It was an amazing way to experience just how much you can suffer in the cloud, and just how far you can go down the rabbit hole with this kind of tech.
tjpd 7y ago
I can also recommend MagicSandbox (https://www.msb.com/ https://www.msb.com/) which provides a lot of learning content alongside a real k8s remote environment.
crb002 7y ago
For me it was a Blue Gene/L.

470 points by alexellisuk 7y ago

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ https://cilium.io/ (Networking) https://rook.io/ https://rook.io/ (Persistent Storage)

salamander014 7y ago
I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router (https://kube-router.io https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (currently running Canal). Not sure what we will decide on when we go to production.
- fyfy18 7y ago
  I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device. Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It hasn't lost data though, even though one of the SSDs died and was unrecoverable, so props for that. To be fair I haven't tried upgrading since I installed it so maybe my issues would be resolved by that. Overall Rook feels somewhat overkill for a homelab user, but I haven't seen any recommended alternatives.
  - Nullabillity 7y ago
    You need to set a memory target for your OSDs, or they will gobble up as much as possible for their cache. In Rook you do that by specifying the resource limits on your CephCluster[0]. Sadly, Rook enforces some pretty high minimum values (2GiB per OSD, 1 GiB per mon), which is fine for production but can make homelab experimentation annoying. [0]: https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster-wide-resources-configuration-settings https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster...
bogomipz 7y ago
Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is that correct?
- moondev 7y ago
  Not directly, rook provides a storage backend for in-cluster workload persistent volumes. In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.
  - ownagefool 7y ago
    I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume. You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.
    - moondev 7y ago
      The parent comment was asking about root volumes for kubernetes nodes > Does Rook give you the equivalent of EBS root volumes for your nodes then? I think we are saying the same thing if i'm not mistaken?
      - ownagefool 7y ago
        I doubt the parent really cares about whether or not it's a root volume, but for the record you can mount it to root. It'll just essentially be empty, which means you have to add a step of populating root with something useful. You'd have to either try and use an initContainer to copy in a filesystem at runtime, have ceph give you a pre-populated directory, probably via thin provisioning, or do something out of band (I've did this in k8s). This is usually more effort than it's worth though, as container runtimes populate the root for you with whatever you want anyway. Plus, if you start treating containers as blobs like VMs, you'll end up in the situation where you don't know where your important variable data is, which leads to situations where people forget to back it up and test it.

pojntfx 7y ago

@alexellisuk almost all of my interest in Kubernetes is due to your work. Thank you for everything you do!

ricardbejarano 7y ago

Everything I know about Kubernetes I learned from a single-node (pseudo)clustered refurbished ThinkPad X201. It cost me 90€ at an IBM refurbished sell. It is downstairs by the router, and it has been hosting everything for me except my email and my blog (which I want to host but I'm not sure about the reliability of my ISP's service, this morning it went down for 5h12m, without prior notice or anything). It is amazing how much you learn from doing stuff. I'm currently on my 3rd year of univerity for CS, so I've tried the academia-style learning process, reading books on my own and doing stuff on my own. The latter is the best method by far.

mpfundstein 7y ago
I only really learn by doing stuff. I read books usually to get started but mostly quickly have to turn to building something. Then revisiting the book on the go. If I only read the book/paper/whatever, I usually ‘think’ I get it but nearly always that isn’t the case :-)

tyingq 7y ago

Thankfully these days, places like Digital Ocean and Linode have managed K8s where you only pay for the compute nodes, for $5/month each. So it's fairly cheap and easy to learn on a "real" cluster without having to build one.

gatherhunterer 7y ago
Building a cluster is a fun and relatively easy project. You learn much more by having the hardware at your fingertips. You can simulate network failures and power failures or you can crash an important daemon. By causing problems you can see how K8s responds and manages itself when it does not have the nodes it expects. It is important to know these things because, for example, if you create a pod instead of a replica set then the loss of a node will mean the inability to use the pod assigned to that node. You need to know how a pod and a replica set differ in order to create a self-healing stack. You can learn all of that with the cloud solutions but the ability to answer your own questions will always be the superior means of learning. The Agile movement has convinced many that the solution that gets you up and running today is the best. A good engineer is not afraid of working and learning instead of just buying a pre-baked implementation and calling themself an experienced user. The cluster that I know like the back of my hand is my “real” cluster. The production cluster is someone else’s that I just rent.
- geerlingguy 7y ago
  Through my work on this project, I've learned a ton about kubectl, kubeadm, the control plane, etc. that I never learned when managing EKS clusters at a previous job. It's good to know more of the 'guts' of an insanely complex ecosystem like Kubernetes, which is why I often recommend people follow Kelsey Hightower's "Kubernetes the Hard Way" guide to install Kubernetes from primitives at least once.
- tyingq 7y ago
  Sure. It's not necessarily skipping all those things you mention that's the motivation. Just the desired order and upfront effort to learn things about K8S.
- 7y ago

raesene9 7y ago

A good method for learning how clusters work and playing with them without having to spend a lot of time re-building when you break things is kind (https://kind.sigs.k8s.io/ https://kind.sigs.k8s.io/). Each node is a Docker container, but the version of Kubernetes running inside it is vanilla Kubeadm, so it's quite representative of what "real" clusters would look like. The great thing about it is you can spin up a cluster in < 2 mins locally to try things out, then it's a single command to delete.

geerlingguy 7y ago
I use Kind (and Minikube, and a number of other solutions too), but this is kind of my "Kubernetes-the-really-hard-way" fun project. Note that I maintain a parallel configuration that runs a multi-node cluster on Vagrant for local development [1] as well as a docker environment built for CI purposes[2] using almost all the same images, with the same Ansible playbooks to configure them all across platforms. [1] https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/vagrant https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas... [2] https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/docker https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas...
- raesene9 7y ago
  yeah I think this approach will have fewer limitations than kind, and also it's fun to try different approaches! BTW I think your ansible book is great, really helped me get to grips with it.
- vlahmot 7y ago
  Thank you for all of your great Ansible work! I leverage a lot of them at work and they have been great! (I’ll be playing around with these!)
- fizixer 7y ago
  Sorry, late to the party. Thanks for sharing. I fully agree with the fun, and the really-hard-way part. However, if we ignore that part for the moment, would you think having an R-Pi cluster handy is worth it for training/sandboxing/testing with the aim of applying the skills in your day job? Are there any advantages to having access to your own R-Pi kubernetes cluster, despite having stuff like Kind and Minikube available online?

jptoto 7y ago

I LOVE Jeff's work and I own his Ansible book. The setup is pretty awesome. FWIW I think you could do this with local VMs a little more easily and provision with Vagrant. Having said that, a cluster of Pis is super fun!

geerlingguy 7y ago
What do you know, that's exactly how I do local development work for the cluster ;) https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/testing/vagrant https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas...
- jptoto 7y ago
  Aha! Makes sense.
JeremyNT 7y ago
Yeah, using local vms is going to greatly expand access to this kind of learning. A current developer laptop is more than beefy enough, and I'd suggest that there aren't many useful lessons to learn from the pi hardware that can't be learned fully virtualized. But if you're having fun, of course, more power to you :)

Mountain_Skies 7y ago

Wonder if some old netbooks could be used for this purpose. Doubt I am the only one with a small pile of them laying around.

ricardbejarano 7y ago
From experience, anything with 2GB or more RAM can be a master node. Workers can even have 1GB and work just fine. Be warned, though, in my experience etcd requires a somewhat decent read/write latency, or else it's going to fail, and when etcd fails everything fails. Your changes don't apply, etc.
- geerlingguy 7y ago
  _Technically_ 1GB is the limit, but yeah, at 1GB I was hitting OOM errors every day or so (this was on the Pi 3 B+). I was very happy when I found out the Pi 4 had 2 or 4GB models! No matter what, you should also make sure to not schedule Pods on the master node(s) when you have limited memory (or in most cases, really...).
  - ricardbejarano 7y ago
    Or set tight resource limits. Setting resource requests also enables the use of HorizontalPodAutoscalers.
    - geerlingguy 7y ago
      Very true. The Pi Dramble's web pods are set to scale using the HPA.

sogubsys 7y ago

Thanks Jeff!

vyshane 7y ago

3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-tower/ https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.

geerlingguy 7y ago
Very nice; I've considered doing something similar and running some production sites on it from my home, but the limitation has always been my terrible Internet bandwidth through Spectrum. We almost got Verizon gigabit fiber a few years ago... then AT&T ran fiber to the front of my neighborhood last year, and then never ran it to the houses. As it is, I'm stuck with 10 mbps uplink, which is not enough to be able to do most of what I would want to do with a more powerful local cluster.
- deleted 7y ago
  [deleted]
yumraj 7y ago
This is very cool. Curious, roughly, how much did this setup cost?
rwmj 7y ago
Yours is a lot neater than the four node bare cluster I built a few years ago: https://rwmj.wordpress.com/2014/04/28/caseless-virtualization-cluster-part-5/ https://rwmj.wordpress.com/2014/04/28/caseless-virtualizatio... One issue with caseless machines is the amount of RF they emit. Makes it hard to listen to any broadcast radio near one and probably disturbs the neighbours similarly. I'm now using a cluster of NUCs which is considerably easier to deal with although quite a lot more expensive: https://rwmj.wordpress.com/2018/02/06/nuc-cluster-bring-up/ https://rwmj.wordpress.com/2018/02/06/nuc-cluster-bring-up/

kiseleon 7y ago

Just an aside, your parts list for the pi dramble has 4 Pi 4B's but micro usb power cables

geerlingguy 7y ago
Oops! I forgot to update that when I switched everything out for the 4 Bs. I'll update that in a little bit.
- geerlingguy 7y ago
  It has been updated.

pstadler 7y ago

Great article! Never stop tinkering. Here‘s how I got to know Kubernetes: By end of 2016, the Kubernetes hype was just about to pick up real steam. As somebody who always liked the idea of running something like an own cluster in the cloud, I attended KubeCon Europe in early 2017. The event was sold out, and took place in a venue almost too small for the number of attendees. It was great. During the event I was just about to finish the Hobby Kube[1] project. Back then weren’t any guides that addressed all the problems encountered when running Kubernetes on a budget, using low cost VPS on random cloud providers. So I dived into the subject in the second half of 2016 and started writing a guide, including automated provisioning using Terraform. I discovered WireGuard in the process of looking for a solution to efficiently securing network traffic between hosts. This still makes me feel like I was an early adopter to something that’s becoming hugely popular. If somebody would like to add a Terraform module for deploying to Raspberry Pi, please ping me or open a PR here[2]. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning

Methusalah 7y ago

This seems like a fun project and a great way to learn Kubernetes, but if I'm dropping this much money on it, I'd like it to have some productive purpose afterwards. I'm a full-stack web dev primarily using node/react/postgres. I've also got some projects currently hosted on a Linode instance. Ideas on fun/productive uses for this cluster after I've built it and messed around with Kubernetes?

birdyrooster 7y ago
The Kubernetes ecosystem is evolving rapidly, you might want to keep it around to play with different CNIs, CSIs, service meshes, operators for clustered software lifecycle, and more.
striking 7y ago
Move the projects to your cluster and see what happens! Consider the fact that, if you make improvements to the cluster, all of your apps will see that same lift. So if you were to set up backups on the cluster's persistent volumes and its databases, you'd get free backups for all of the projects you've moved. Same with monitoring, autoscale, and so on.
robgibbons 7y ago
RPI's will handle Node surprisingly well, from my experience. A small cluster of them would be well suited to any number of web projects. A few years back, I played around with haproxy and a few Pi's running Node servers. You may be surprised how well they work as servers, as long as you're not expecting Xeon-level speeds.

acd 7y ago

Thanks Jeff for your great Ansible roles!

madrox 7y ago

This is also how my alma mater (Cal Poly SLO) teaches Hadoop. Building real world clusters are expensive, and giving each student their own is difficult. However, small clusters of Raspberry Pis are cheap, and it's also very easy to demonstrate how unplugging one affects the cluster.

segmondy 7y ago

You can also build a cluster on your computer if it's beefy enough by running multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few years ago and that's my k8s experiment land.

MuffinFlavored 7y ago
I don’t fully understand where the line goes from “all I need is a Docker container for nginx + Postgres + Redis + my services” to “I need Kubernetes” When does one need to go from just Docker containers to container orchestration like k8s?
- Nullabillity 7y ago
  Do you care about availability? Do you care about rolling upgrades? Do you want to practice any of those two? Then it might be time to make the switch. Yes, Docker has Swarm Mode. And yes, it is easier to get started with than K8s. The problem is that it locks you into a lot of choices when you first spin up each (Docker) Service, that can't be changed without taking down the Service and then recreating it with the new option. Want to change how upgrades are rolled out? Too bad! Kubernetes still has a few settings like that, but since (K8s) Services are just responsible for routing traffic to separately-managed Pods, you can always fall back to rolling the upgrade manually, or doing a blue-green deployment.
  - MuffinFlavored 7y ago
    I do blue-green deploys with Docker and I deploy the services as “app-1”, “app-2”, “app-3”, etc.
    - Nullabillity 7y ago
      How do you route dependencies to the correct service? If you're using a stack then you end up needing to blue-green the whole thing, if you're not then you still need to blue-green the whole dependent subset of the dependency graph. And for anything exposed outside of the swarm you'd need to use some reverse proxy, since only one service (in the whole swarm) can bind a port at any one time.

ljm 7y ago

I suppose I was lucky dabbling in one of my side-projects because I had money to burn. I put it all in Google Cloud and then learned just how much you have to complicate the stack (beyond the complication of K8S) to lower infra costs. Suddenly I wasn't using basic Kubernetes any more, I was setting up a new ingress controller so Google wouldn't launch a new load balancer instance for every public service I exposed. Those things aren't cheap. It was an amazing way to experience just how much you can suffer in the cloud, and just how far you can go down the rabbit hole with this kind of tech.

tjpd 7y ago

I can also recommend MagicSandbox (https://www.msb.com/ https://www.msb.com/) which provides a lot of learning content alongside a real k8s remote environment.

crb002 7y ago

For me it was a Blue Gene/L.