4 ms·

I tried k3s several times in the past few years but I still can't use it in my homelab: 1. the project claims to be production ready and support HA control pla

by zzyzxd 4y ago

I tried k3s several times in the past few years but I still can't use it in my homelab: 1. the project claims to be production ready and support HA control plane setup, but there's no solution for API load balancing out of box. How do you bring up a new node(either control plane or worker node)? You write down the join token produced by the first control plane node, and hardcode the token and the existing control plane's IP in the new node's systemd unit file. Btw, if you use the official installation script, that file is going to have permission 755 and everyone on the server can just read that token. 2. And how do you bring up the first control plane anyways? The official instruction is to `curl` a bash script and pipe into a shell. You can probably translate that script into some ansible playbook, but the whole running-a-bootstrap-script-and-passing-along-secrets approach make the whole process difficult to be converted into some something that's supposed to be idempotent. All the problems can be worked around, in fact I was half way there, but then I suddenly started thinking: "didn't I choose k3s because I thought it was easy?"

jhoelzel 4y ago
Truth be told yes that tricky but can be managed with ansible easily for instance. 1) Your main problem though would probably be the need for a haproxy or bgp which does load balancing for you. There are other solutions like kube-vip but they are more a "failover" solution that HA. Which would be fine for a homelab and is for instance how Rancher Harvester (kubernetes for virtual machines) does it. 2) you have to pass a parameter called --cluster-init for the first node and then join the other nodes. once the cluster is running you dont need any node with that parameter anymore and its common practise to create the first node wiuth --cluster-init then join 3 other ones and take down the first node And on a personal node, you sound like you would be happy with rancher harvester. check it out its bascially turnkey
- zzyzxd 4y ago
  Yup, when I said that I was half way there, I was trying to use ansible to deploy a nginx systemd unit on every node to load-balancing requests to API servers. Many other distributions can do this with a static pod managed by local kubelet, but that's not possible in k3s. I know there are solutions to my problem, but I can either implement them by myself, or like you mentioned, I can just use a different distribution and not having these problems.
cwayne 4y ago
Could check out https://github.com/k3s-io/k3s-ansible https://github.com/k3s-io/k3s-ansible althought admittedly it's a bit in need of some lovin'
- zzyzxd 4y ago
  Yes, I did. Unfortunately, that playbook only supports single master.
ph4te 4y ago
I've gone through this a few times recently and have it in my homelab and at the office. What works good for me using kube-vip to a VIP on the control plane, and then MetalLB to dish out private addresses in the respective networks, or even statically assigned addresses. I have been turning them all up with k3sup which works like a charm. turn up the first node, install kube-vip, switch config to point to the vip, turn up all my other master nodes, then turn up my workers, install metallb, setup my subnet, install rancher, expose it with a LB, install longhorn. then start deploying things. here is an example of what i use to turn up the first one with k3sup. all of the servers are turned up and configured with ansible doing minimal updates, users, sudo access, etc.. k3sup install \ --ip=192.168.1.11 \ --user=k3s-user \ --sudo \ --tls-san=192.168.1.10 \ --cluster \ --k3s-channel=stable \ --k3s-version=v1.24.12+k3s1 \ --no-extras \ --k3s-extra-args "--flannel-iface=ens160 --node-ip=192.168.1.11" \ --merge \ --local-path $HOME/.kube/config \ --context=k3s-lab
fellowniusmonk 4y ago
Thank you for sharing this and saving me time in exploring k3, it's shocking how common it is that an evangelized tool is impractical to setup and use in even a simple homelab configuration. Knowing what NOT to investigate because it isn't "ready" can be one of the biggest time sucks.
MuffinFlavored 4y ago
> but there's no solution for API load balancing out of box Why do you need this in a homelab?
- zzyzxd 4y ago
  Because 1. My homelab has a lot of crappy hardware with zero enterprise support. And when a server fails, I need to be able to easily swap it out. And I don't want to make it a full time job for myself to maintain it. 2. I did knock some control plane nodes down a few times. It was rare, but still not fun. 3. Also, because why not? HA control plane is not particularly challenging these days. I know exactly how to do it. I just don't have the bandwidth to learn how to do it in k3s, especially when it comes for free in some other distributions.

by zzyzxd 4y ago

jhoelzel 4y ago

Truth be told yes that tricky but can be managed with ansible easily for instance. 1) Your main problem though would probably be the need for a haproxy or bgp which does load balancing for you. There are other solutions like kube-vip but they are more a "failover" solution that HA. Which would be fine for a homelab and is for instance how Rancher Harvester (kubernetes for virtual machines) does it. 2) you have to pass a parameter called --cluster-init for the first node and then join the other nodes. once the cluster is running you dont need any node with that parameter anymore and its common practise to create the first node wiuth --cluster-init then join 3 other ones and take down the first node And on a personal node, you sound like you would be happy with rancher harvester. check it out its bascially turnkey

zzyzxd 4y ago
Yup, when I said that I was half way there, I was trying to use ansible to deploy a nginx systemd unit on every node to load-balancing requests to API servers. Many other distributions can do this with a static pod managed by local kubelet, but that's not possible in k3s. I know there are solutions to my problem, but I can either implement them by myself, or like you mentioned, I can just use a different distribution and not having these problems.

cwayne 4y ago

Could check out https://github.com/k3s-io/k3s-ansible https://github.com/k3s-io/k3s-ansible althought admittedly it's a bit in need of some lovin'

zzyzxd 4y ago
Yes, I did. Unfortunately, that playbook only supports single master.

ph4te 4y ago

I've gone through this a few times recently and have it in my homelab and at the office. What works good for me using kube-vip to a VIP on the control plane, and then MetalLB to dish out private addresses in the respective networks, or even statically assigned addresses. I have been turning them all up with k3sup which works like a charm. turn up the first node, install kube-vip, switch config to point to the vip, turn up all my other master nodes, then turn up my workers, install metallb, setup my subnet, install rancher, expose it with a LB, install longhorn. then start deploying things. here is an example of what i use to turn up the first one with k3sup. all of the servers are turned up and configured with ansible doing minimal updates, users, sudo access, etc.. k3sup install \ --ip=192.168.1.11 \ --user=k3s-user \ --sudo \ --tls-san=192.168.1.10 \ --cluster \ --k3s-channel=stable \ --k3s-version=v1.24.12+k3s1 \ --no-extras \ --k3s-extra-args "--flannel-iface=ens160 --node-ip=192.168.1.11" \ --merge \ --local-path $HOME/.kube/config \ --context=k3s-lab

fellowniusmonk 4y ago

Thank you for sharing this and saving me time in exploring k3, it's shocking how common it is that an evangelized tool is impractical to setup and use in even a simple homelab configuration. Knowing what NOT to investigate because it isn't "ready" can be one of the biggest time sucks.

MuffinFlavored 4y ago

> but there's no solution for API load balancing out of box Why do you need this in a homelab?

zzyzxd 4y ago
Because 1. My homelab has a lot of crappy hardware with zero enterprise support. And when a server fails, I need to be able to easily swap it out. And I don't want to make it a full time job for myself to maintain it. 2. I did knock some control plane nodes down a few times. It was rare, but still not fun. 3. Also, because why not? HA control plane is not particularly challenging these days. I know exactly how to do it. I just don't have the bandwidth to learn how to do it in k3s, especially when it comes for free in some other distributions.