3 ms·

I implemented self-hosted k8s in my current company and had to learn it from the ground. I'm not a professional, but I learned a thing or two. My opinion is as

I implemented self-hosted k8s in my current company and had to learn it from the ground. I'm not a professional, but I learned a thing or two. My opinion is as follows: Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux. Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard. Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before. Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy. Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works. Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well. Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.

__turbobrew__ 4y ago
Agree on storage being the hard part. You either need a robust SAN to store persistent volumes or you need to store persistent volumes on the local storage of kubelets which is a major pain. Once a pod binds a local pvc to a kubelet that pod must now run on that kubelet for the rest of eternity. If you want to get rid of a kubelet with a local pvc you either need the application be durable to data loss (which many are not) or you need to implement some sort of stateful pod draining logic. When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.

__turbobrew__ 4y ago

Agree on storage being the hard part. You either need a robust SAN to store persistent volumes or you need to store persistent volumes on the local storage of kubelets which is a major pain. Once a pod binds a local pvc to a kubelet that pod must now run on that kubelet for the rest of eternity. If you want to get rid of a kubelet with a local pvc you either need the application be durable to data loss (which many are not) or you need to implement some sort of stateful pod draining logic. When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.