3 ms·

I’ve been working on a Kubernetes cluster for my home network. I know a lot of people hate on Kubernetes, but it’s pretty cool and not too bad to even get start

I’ve been working on a Kubernetes cluster for my home network. I know a lot of people hate on Kubernetes, but it’s pretty cool and not too bad to even get started from scratch with Kubeadm. I think the constant thought in the back of my mind is… Docker containers and Nginx is definitely enough for my needs. At work, it’s a different story, but more most things, it can be over kill.

vbezhenar 4y ago
My biggest issue with Kubernetes is huge costs of running it. It's 3 servers with 4 GB RAM. It's almost $100/month just for Kubernetes alone. And you need one load balancer for control nodes which is another $25 for my hoster. And you need second load balancer for production workloads which is another $25. So you have to pay $150/month for the privilege of using Kubernetes. And they double your costs for your application, because you want it to be high-available, why bother with Kubernetes otherwise. So something that could be run with $5 droplet, suddenly requires $200. They you want to install grafana, ELK, service mesh and whatnot.
- tux3 4y ago
  k3s will run on a $5 droplet happily. If you want k8s for the yaml and not for the heavy-duty hyperscale redundancy, k3s works great
  - vbezhenar 4y ago
    It's hard to stop when HA is one config line (and few additional dozens of bucks) away.
- yjftsjthsd-h 4y ago
  You don't actually need all that. If the competition is a single $5 machine with no redundancy, you can just run a single node cluster.
- jeroenhd 4y ago
  Three machines with 4GB of RAM for $100? I don't know where you rent your servers but that's ridiculous. You can easily get that kind of compute for a third or lower. You won't get a private LAN (so you'll need some firewall rules on the hoster side) and you'll have to do some setup to get everything served up on the right public IP address(es), but you don't need a dedicated load balancer product. Unless you expect your virtual servers to go down randomly, I suppose. I've set up k0s on Oracle's free service to mess with. It doesn't need nearly as much as 12GiB of RAM. Give it 500MiB per machine to be happy and it _should_ run fine with 10.5GiB to spare unless you're adding tons of overlays and additional services on top of your applications. As far as I can tell, the biggest cost with Kubernetes is maintaining the system and keeping up to date with the advancements and deprecations in the Kubernetes field. You can't just upgrade the Kubernetes version and expect everything to work, and every new concept that replaces anything old comes with Kubernetes layers of complexity. I honestly don't know what so many companies are using all those resources Kubernetes provides for. I have a feeling people are so caught up in the "modern server management" world that they've forgotten how powerful and reliable a $25 VPS is these days.
  - mitjam 4y ago
    The simplest and cheapest self hosted K8s I found is kube-Hetzner [1]. It puts k3s on MicroOS on Hetzner Cloud; for a three node Cluster I pay around 50 EUR per month (3shared AMD Cores, 4GB RAM per Node, 200 GB Storage, 1 TB Backup, small loadbalancer). A single server with the same spec and the same storage and backup and a floating IP would go for about 25 EUR/month. For me, k8s is worth the difference. [1] https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud/latest https://registry.terraform.io/modules/kube-hetzner/kube-hetz...
shepherdjerred 4y ago
Do you think running k8s at home is worthwhile, past the learning experience and cool factor? I have a nice server that I used for 10-20 applications like Time Machine backup, media streaming, file syncing, home automation, etc. I was looking into k8s, but it seemed way too complicated when I only have one physical server, so I went with Docker + Docker Compose instead.
- rschachte 4y ago
  If I was in your situation, probably not. We use it a lot at work, so it’s a playground to grow my skills and play around while doubling as a painless way to host anything. All in all, I still think docker, docker compose and nginx is the way to go for hobbyist stuff. I doubt my ISP would let me host anything at a huge scale off my home router anyways.
  - shepherdjerred 4y ago
    nginx is great, but I've also been hearing good things about caddy[0]. I want to check it out soon. [0]: https://caddyserver.com/v2 https://caddyserver.com/v2
- SOLAR_FIELDS 4y ago
  If you were to bring on two more servers it would probably be worth it. 3 is the magic number for when k3s starts to bring more interesting stuff to the table in terms of reliability and dynamic behaviors.
- cmehdy 4y ago
  All the maintenance you put towards getting things running right now will be maintenance you put towards making sure your cluster is doing what you need and want. If you start having more servers, a k8s cluster can do something like automatically transfer workloads when one machine has an issue for example, or dynamically allocating storage (even cloud) when needed, or allowing you to issue a single command to spin up more instance of whatever task you'd want to run more in parallel, or updating a version in a config file and letting the cluster do the job of stopping/updating/restarting things. But what you also have to do is ensure the underlying networking between machines works, all versions of the binaries on all the machines of the cluster are in close agreement with each others, encryption of data via certificates and CAs is satisfactory, and your master(s) might require different maintenance from the workers - but you'll end up reading the docs to know how to roll updates step by step. In your use case I would stick with a smart use of docker-composes, perhaps just centralized in a private repo, with cron jobs on top. If you buy/rent a couple machines of any kind (from raspberry pis to cloud instances) to put on your network and run things for you, then a cluster becomes increasingly interesting as the amount of tasks needed/wanted and amount of computing available grows. My context: I have some limited experience with work (including some scary "abstractions" on top of K8s & Helm) and a recent CKA (certified kubernetes administrator hands-on certification) from LinuxFoundation.

vbezhenar 4y ago

My biggest issue with Kubernetes is huge costs of running it. It's 3 servers with 4 GB RAM. It's almost $100/month just for Kubernetes alone. And you need one load balancer for control nodes which is another $25 for my hoster. And you need second load balancer for production workloads which is another $25. So you have to pay $150/month for the privilege of using Kubernetes. And they double your costs for your application, because you want it to be high-available, why bother with Kubernetes otherwise. So something that could be run with $5 droplet, suddenly requires $200. They you want to install grafana, ELK, service mesh and whatnot.

tux3 4y ago
k3s will run on a $5 droplet happily. If you want k8s for the yaml and not for the heavy-duty hyperscale redundancy, k3s works great
- vbezhenar 4y ago
  It's hard to stop when HA is one config line (and few additional dozens of bucks) away.
yjftsjthsd-h 4y ago
You don't actually need all that. If the competition is a single $5 machine with no redundancy, you can just run a single node cluster.
jeroenhd 4y ago
Three machines with 4GB of RAM for $100? I don't know where you rent your servers but that's ridiculous. You can easily get that kind of compute for a third or lower. You won't get a private LAN (so you'll need some firewall rules on the hoster side) and you'll have to do some setup to get everything served up on the right public IP address(es), but you don't need a dedicated load balancer product. Unless you expect your virtual servers to go down randomly, I suppose. I've set up k0s on Oracle's free service to mess with. It doesn't need nearly as much as 12GiB of RAM. Give it 500MiB per machine to be happy and it _should_ run fine with 10.5GiB to spare unless you're adding tons of overlays and additional services on top of your applications. As far as I can tell, the biggest cost with Kubernetes is maintaining the system and keeping up to date with the advancements and deprecations in the Kubernetes field. You can't just upgrade the Kubernetes version and expect everything to work, and every new concept that replaces anything old comes with Kubernetes layers of complexity. I honestly don't know what so many companies are using all those resources Kubernetes provides for. I have a feeling people are so caught up in the "modern server management" world that they've forgotten how powerful and reliable a $25 VPS is these days.
- mitjam 4y ago
  The simplest and cheapest self hosted K8s I found is kube-Hetzner [1]. It puts k3s on MicroOS on Hetzner Cloud; for a three node Cluster I pay around 50 EUR per month (3shared AMD Cores, 4GB RAM per Node, 200 GB Storage, 1 TB Backup, small loadbalancer). A single server with the same spec and the same storage and backup and a floating IP would go for about 25 EUR/month. For me, k8s is worth the difference. [1] https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud/latest https://registry.terraform.io/modules/kube-hetzner/kube-hetz...

shepherdjerred 4y ago

Do you think running k8s at home is worthwhile, past the learning experience and cool factor? I have a nice server that I used for 10-20 applications like Time Machine backup, media streaming, file syncing, home automation, etc. I was looking into k8s, but it seemed way too complicated when I only have one physical server, so I went with Docker + Docker Compose instead.

rschachte 4y ago
If I was in your situation, probably not. We use it a lot at work, so it’s a playground to grow my skills and play around while doubling as a painless way to host anything. All in all, I still think docker, docker compose and nginx is the way to go for hobbyist stuff. I doubt my ISP would let me host anything at a huge scale off my home router anyways.
- shepherdjerred 4y ago
  nginx is great, but I've also been hearing good things about caddy[0]. I want to check it out soon. [0]: https://caddyserver.com/v2 https://caddyserver.com/v2
SOLAR_FIELDS 4y ago
If you were to bring on two more servers it would probably be worth it. 3 is the magic number for when k3s starts to bring more interesting stuff to the table in terms of reliability and dynamic behaviors.
cmehdy 4y ago
All the maintenance you put towards getting things running right now will be maintenance you put towards making sure your cluster is doing what you need and want. If you start having more servers, a k8s cluster can do something like automatically transfer workloads when one machine has an issue for example, or dynamically allocating storage (even cloud) when needed, or allowing you to issue a single command to spin up more instance of whatever task you'd want to run more in parallel, or updating a version in a config file and letting the cluster do the job of stopping/updating/restarting things. But what you also have to do is ensure the underlying networking between machines works, all versions of the binaries on all the machines of the cluster are in close agreement with each others, encryption of data via certificates and CAs is satisfactory, and your master(s) might require different maintenance from the workers - but you'll end up reading the docs to know how to roll updates step by step. In your use case I would stick with a smart use of docker-composes, perhaps just centralized in a private repo, with cron jobs on top. If you buy/rent a couple machines of any kind (from raspberry pis to cloud instances) to put on your network and run things for you, then a cluster becomes increasingly interesting as the amount of tasks needed/wanted and amount of computing available grows. My context: I have some limited experience with work (including some scary "abstractions" on top of K8s & Helm) and a recent CKA (certified kubernetes administrator hands-on certification) from LinuxFoundation.