4 ms·

k8s is not as hard as its advertised

by fallat 3y ago

ghaff 3y ago
One of my recent conclusions is that it's not so much that Kubernetes on its own is so very complicated; it's that if someone tries to glue their own container platform together from Kubernetes plus another dozen or so cloud-native projects and integrate the whole thing, it can get wickedly complex very quickly.
- derefr 3y ago
  Right — Kubernetes was evolved in the context of being a Google Cloud service, and despite now having other deployments, still is mostly architected around the assumption that your Kubernetes cluster is surrounded by a featureful IaaS platform providing things like managed load balancers, VM auto-scaling pools, iSCSI volumes on a dedicated SAN network, non-throughput-bottlenecked object storage (and a container image repo on top of that!), a centralized logs and metrics sink, etc. A k8s cluster expects these like a computer expects a bootable storage device. If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects. (And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
  - ghaff 3y ago
    There are other container platforms like Red Hat OpenShift as well (disclaimer: I work there) either operated for you or on-prem (doesn't need to be on top of OpenStack and typically isn't these days). I'm biased but Kubernetes complexity tends to be conflated with the complexity of a DIY container platform which can be significant, especially given that Kubernetes itself has generally made the decision to keep its scope relatively narrow.
vbezhenar 3y ago
I implemented self-hosted k8s in my current company and had to learn it from the ground. I'm not a professional, but I learned a thing or two. My opinion is as follows: Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux. Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard. Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before. Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy. Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works. Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well. Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.
- __turbobrew__ 3y ago
  Agree on storage being the hard part. You either need a robust SAN to store persistent volumes or you need to store persistent volumes on the local storage of kubelets which is a major pain. Once a pod binds a local pvc to a kubelet that pod must now run on that kubelet for the rest of eternity. If you want to get rid of a kubelet with a local pvc you either need the application be durable to data loss (which many are not) or you need to implement some sort of stateful pod draining logic. When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.

k8s is not as hard as its advertised

by fallat 3y ago

k8s is not as hard as its advertised

One of my recent conclusions is that it's not so much that Kubernetes on its own is so very complicated; it's that if someone tries to glue their own container platform together from Kubernetes plus another dozen or so cloud-native projects and integrate the whole thing, it can get wickedly complex very quickly.

derefr 3y ago
Right — Kubernetes was evolved in the context of being a Google Cloud service, and despite now having other deployments, still is mostly architected around the assumption that your Kubernetes cluster is surrounded by a featureful IaaS platform providing things like managed load balancers, VM auto-scaling pools, iSCSI volumes on a dedicated SAN network, non-throughput-bottlenecked object storage (and a container image repo on top of that!), a centralized logs and metrics sink, etc. A k8s cluster expects these like a computer expects a bootable storage device. If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects. (And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
- ghaff 3y ago
  There are other container platforms like Red Hat OpenShift as well (disclaimer: I work there) either operated for you or on-prem (doesn't need to be on top of OpenStack and typically isn't these days). I'm biased but Kubernetes complexity tends to be conflated with the complexity of a DIY container platform which can be significant, especially given that Kubernetes itself has generally made the decision to keep its scope relatively narrow.

vbezhenar 3y ago

I implemented self-hosted k8s in my current company and had to learn it from the ground. I'm not a professional, but I learned a thing or two. My opinion is as follows: Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux. Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard. Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before. Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy. Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works. Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well. Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.

__turbobrew__ 3y ago
Agree on storage being the hard part. You either need a robust SAN to store persistent volumes or you need to store persistent volumes on the local storage of kubelets which is a major pain. Once a pod binds a local pvc to a kubelet that pod must now run on that kubelet for the rest of eternity. If you want to get rid of a kubelet with a local pvc you either need the application be durable to data loss (which many are not) or you need to implement some sort of stateful pod draining logic. When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.