3 ms·

The issues with large clusters you've surfaced are ones I've never once experienced in my homelab. I'm already an experienced devops (?) engineer so the transit

by MrOwen 5y ago

The issues with large clusters you've surfaced are ones I've never once experienced in my homelab. I'm already an experienced devops (?) engineer so the transition to k8s only took a few weeks and a couple sleepness nights which will not be everyone's experience, especially beginners. But it's a cohesive system and once you learn it, you understand the various patterns and you make it work for you vs you sucking time into it. For myself, learning k8s was way easier than keeping track of various little scripts to do this or that in a swarm cluster. Folks here are making sound arguments for swarm and alternatives and they are totally valid. But for an experienced engineer who needed to start their homelab back up after a long hiatus, I will never look back once for choosing k8s. Honestly some of the most fun I've had with computers in a long time and great experience for when I move into a k8s related $job.

WastingMyTime89 5y ago
> the transition to k8s only took a few weeks and a couple sleepness nights Only? I have had a home server for fifteen years and used to have a homemade router. That's significantly more time than I have ever spent on them. I don't even bother maintaining scripts for my home installation. I have reinstalled it once in the past decade and it took me significantly less time than writing said scripts would have. There was enough things I wanted to do differently that any scripts would have been useless by that point anyway. At that point, it sounds more like a hobby than something saving you time.
- MrOwen 5y ago
  I guess I never considered self-hosting as not a hobby? Web apps with modern stacks are usually pretty complex, so hard for me to imagine someone just casually jumping in as a non-hobby and not running into big problems at some point. Or getting hacked because you didn't bother or forgot to secure something.
  - WastingMyTime89 5y ago
    > Web apps with modern stacks are usually pretty complex, so hard for me to imagine someone just casually jumping in as a non-hobby and not running into big problems at some point. Or getting hacked because you didn't bother or forgot to secure something. That's not my experience at all. Plenty of web apps for self hosting are pretty simple. You just install the packet, read the config files, turn on and off what you need and you are good to go. Since systemd made that easy, most apps now come preconfigured to use a dedicated users and drop the capabilities they don't need. If they don't, you are two lines aways from doing it in their service file. Then I just throw Caddy in front of everything I want to access from outside my local network and call it a day.
KronisLV 5y ago
That's great! Did you use a Kubernetes distribution that's provided by the OS, like MicroK8s ( https://microk8s.io/ https://microk8s.io/ )? Or perhaps one of the lightweight ones, like K3s ( https://k3s.io/ https://k3s.io/ ) or k0s ( https://k0sproject.io/ https://k0sproject.io/ )? Maybe a turnkey one, like RKE ( https://rancher.com/products/rke/ https://rancher.com/products/rke/ )? Or did you use something like kubespray ( https://kubespray.io/#/ https://kubespray.io/#/ ) to create a cluster in a semi automated fashion? Alternatively, perhaps you built your cluster from scratch or maybe used kubeadm ( https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ https://kubernetes.io/docs/setup/production-environment/tool... ) and simply got lucky? Out of all of those, my experiences have only been positive with K3s (though i haven't tried k0s) and RKE, especially when using the latter with Rancher ( https://rancher.com/ https://rancher.com/ ) or when using the former with Portainer ( https://www.portainer.io/solutions https://www.portainer.io/solutions ). In most of the other cases, i've run into a variety of problems: - different networking implementations that work inconsistently ( https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy ) - general problems with networking, communication between nodes, and certain node components refusing to talk to others and thus nodes not allowing anything to be scheduled on them - large CPU/RAM resource usage for even small clusters, which is unfortunate, given that i'm too poor to afford lots of either - no overprovisioning support (when i last tried), leading to pods not being scheduled on nodes when there's plenty of resources available, due to unnecessary reservations - problems with PVCs and storing data in host directories, like how Docker has bind mounts with few to no issues - also, they recently stopped supporting the Docker shim and now you have to use containerd, which is unfortunate from a debugging perspective (since a lot of people out there actually like Docker CLI) If i could afford proper server hardware instead of 200GEs with value RAM then i bet the situation would be different, but in my experience this is a representation of how things would run in typical small VPSes (should you self host the control plane), which doesn't inspire confidence. That said, when i compared the RAM usage of Docker Swarm with that of K3s, the picture was far better! My current approach is to introduce clients and companies to containers in the following fashion, if they don't use the cloud vendors: Docker --> Docker Compose --> Docker Swarm --> (check if there's need for more functionality) --> K3s. The beautiful think is that because of the OCI standard you can choose where and how to run your app, as well as use tools like Kompose to make migration very easy: https://kompose.io/ https://kompose.io/

by MrOwen 5y ago

WastingMyTime89 5y ago

> the transition to k8s only took a few weeks and a couple sleepness nights Only? I have had a home server for fifteen years and used to have a homemade router. That's significantly more time than I have ever spent on them. I don't even bother maintaining scripts for my home installation. I have reinstalled it once in the past decade and it took me significantly less time than writing said scripts would have. There was enough things I wanted to do differently that any scripts would have been useless by that point anyway. At that point, it sounds more like a hobby than something saving you time.

MrOwen 5y ago
I guess I never considered self-hosting as not a hobby? Web apps with modern stacks are usually pretty complex, so hard for me to imagine someone just casually jumping in as a non-hobby and not running into big problems at some point. Or getting hacked because you didn't bother or forgot to secure something.
- WastingMyTime89 5y ago
  > Web apps with modern stacks are usually pretty complex, so hard for me to imagine someone just casually jumping in as a non-hobby and not running into big problems at some point. Or getting hacked because you didn't bother or forgot to secure something. That's not my experience at all. Plenty of web apps for self hosting are pretty simple. You just install the packet, read the config files, turn on and off what you need and you are good to go. Since systemd made that easy, most apps now come preconfigured to use a dedicated users and drop the capabilities they don't need. If they don't, you are two lines aways from doing it in their service file. Then I just throw Caddy in front of everything I want to access from outside my local network and call it a day.

KronisLV 5y ago

That's great! Did you use a Kubernetes distribution that's provided by the OS, like MicroK8s ( https://microk8s.io/ https://microk8s.io/ )? Or perhaps one of the lightweight ones, like K3s ( https://k3s.io/ https://k3s.io/ ) or k0s ( https://k0sproject.io/ https://k0sproject.io/ )? Maybe a turnkey one, like RKE ( https://rancher.com/products/rke/ https://rancher.com/products/rke/ )? Or did you use something like kubespray ( https://kubespray.io/#/ https://kubespray.io/#/ ) to create a cluster in a semi automated fashion? Alternatively, perhaps you built your cluster from scratch or maybe used kubeadm ( https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ https://kubernetes.io/docs/setup/production-environment/tool... ) and simply got lucky? Out of all of those, my experiences have only been positive with K3s (though i haven't tried k0s) and RKE, especially when using the latter with Rancher ( https://rancher.com/ https://rancher.com/ ) or when using the former with Portainer ( https://www.portainer.io/solutions https://www.portainer.io/solutions ). In most of the other cases, i've run into a variety of problems: - different networking implementations that work inconsistently ( https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy ) - general problems with networking, communication between nodes, and certain node components refusing to talk to others and thus nodes not allowing anything to be scheduled on them - large CPU/RAM resource usage for even small clusters, which is unfortunate, given that i'm too poor to afford lots of either - no overprovisioning support (when i last tried), leading to pods not being scheduled on nodes when there's plenty of resources available, due to unnecessary reservations - problems with PVCs and storing data in host directories, like how Docker has bind mounts with few to no issues - also, they recently stopped supporting the Docker shim and now you have to use containerd, which is unfortunate from a debugging perspective (since a lot of people out there actually like Docker CLI) If i could afford proper server hardware instead of 200GEs with value RAM then i bet the situation would be different, but in my experience this is a representation of how things would run in typical small VPSes (should you self host the control plane), which doesn't inspire confidence. That said, when i compared the RAM usage of Docker Swarm with that of K3s, the picture was far better! My current approach is to introduce clients and companies to containers in the following fashion, if they don't use the cloud vendors: Docker --> Docker Compose --> Docker Swarm --> (check if there's need for more functionality) --> K3s. The beautiful think is that because of the OCI standard you can choose where and how to run your app, as well as use tools like Kompose to make migration very easy: https://kompose.io/ https://kompose.io/