4 ms·
> Keep it managed since that legitimately becomes a reliability headache. This is the thing that I think will always give me pause. If I have to pay a third p
by kelnos 4y ago
> Keep it managed since that legitimately becomes a reliability headache.
This is the thing that I think will always give me pause. If I have to pay a third party to manage my cluster orchestration backplane, that seems like a pretty big piece of overhead.
Sure, I can do it myself, but then I have to deal with said reliability headache. It seems ironic that a cluster management framework -- that touts its ability to reliably keep your applications running -- has its own reliability issues.
- yunwal 4y agoI don't disagree with you, but if you can reliably trade your dataplane outages for control plane outages, that's still usually a good tradeoff.
- kelnos 4y agoThat's a really good point. Certainly you don't want either to go down, but outages are more tolerable if customer requests are still getting serviced regardless.
- lmarcos 4y agoExactly. We are building these incredible open source tools... but they grow so complex that we need to pay others in order to use them effectively? What would you say if you had to pay Google if you want to use Golang in an effective way (because the language has become so complex that it's difficult to handle it on your own?). Crazy. I wanted to take a look at how to use K8s on my own cluster, and damn it, to install the whole thing is not that straightforward. So, now to keep my sanity I need to pay a cloud provider to use k8s! I guess that's the trick: build some open source monster that's very hard to install/maintain/use but has cool features. People will love it and they'll pay for it.
- vbezhenar 4y agoI think people are overestimating the difficulty of setting up k8s. It's not that hard. Grab Debian, install containerd.io, set one sysctl, load one kernel module and you're all set. Install few more packages, run kubeadm init and that's all. The only thing that I've found truly hard is autoscaling VMs. Managed clouds got it easy, I was not able to make it so far. But it doesn't seem that hard either, I just need to write one particularly careful bash script, I just don't have time for that yet. Google does have some secret sauce. For example there's horizontal scaling which spins up more pods as load grows. There's vertical scaling that adjusts resources for individual pods. Those are generally incompatible with each other and people adjust resources manually. GKE autopilot has multi-dimensional scaling which they didn't open source, it does both. May be I wasn't hit with something nasty yet. But so far I think that those complains about managing k8s are somewhat strange. Yes, if you manage thousands of VMs, it might get another full-time job, but that's a scale. I manage few VMs and that's not an issue.
- CommanderData 4y agoI don't fully understand, there's benefits for using a managed service in instances the control plane is something you only interact with but don't manage. Not every Ops team will have a CKA administrator at hand to delve into etcd or controller manager. Open a ticket and it's generally fixed. Then there's situations where you want full control over the control plane itself, I've worked with companies that had clusters installed on bare metal in their stores in the UK. A CKA engineer is an essential in this case but brings it's own reliability headaches.
- lumost 4y agoSomething that isn't appreciated enough is how reliability issues demolish your teams throughput. Got a random cluster restart heisenbug taking customers offline? good luck debugging it for the next 6 weeks. Alternately, ignore the problem while your engineers get woken up at night until they quit... The progress of software has been towards managed offerings for a reason. A company can make it their entire business to own fixing these reliability issues for you, and keeping them at bay. Do you really want to be int the business of managing a cloud on your cloud?
- NightMKoder 4y agoThis may not be a surprise to some, but when folks talk about reliability of the control plane, they usually think failure means their web service goes down. That’s not true. If you shot the kubernetes control plane, the individual servers can’t talk to it anymore - so they do nothing. Docker images that were running stay running. They even get restarted if they crash (via restartPolicy). Services that had specific other pods they were referencing continue referencing those pods. In net: everything except for kubectl and other kubenetes internals keeps working. That said, one piece that isn’t talked about frequently is the network overlay. Kubernetes virtualizes IPs (so each pod gets an IP), which is awesome to work with when it works. But if your overlay network goes down - god help you. DNS failures are the first to show up, but it’s all downhill from there. Most overlays take great care to degrade well, so they’re not tied to the control plane, but I have yet to find one that’s perfect. The overlay is the part of kube that truly isn’t failure tolerant in my experience.
- 411111111111111 4y ago> Kubernetes virtualizes IPs (so each pod gets an IP), which is awesome to work with when it works Kubernetes does no such thing. Weave Net, which is likely the most used CNI, does. There are other options however, and some of them use baremetal routers via bridging or even VLANs for example. https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model https://kubernetes.io/docs/concepts/cluster-administration/n...
- simiones 4y agoThe fact that each pod has an IP is a core assumption of Kubernetes. Sure, the CNIs are responsible for actually implementing this, but it is a required part of their contract to provide 1 unique IP per pod (or, more precisely, either 1 IPv4 or 1 IPv6 or both per virtual NIC per pod - to cover dual-stack support in 1.24+ and Multus).
- 411111111111111 4y agoThat's probably true, but also irrelevant to the question wherever kubernetes virtualizes IPs. But now that I'm rereading my comment: it does look as if I'm also talking about each pod having one IP. That was bad quoting / phrasing on my part, as I wasn't contesting that at all. With flannel you could provision the IP through DHCP by bridging the network adapter of the pod to the physical interface to get an IP from a router appliance for example. It's probably also possible to dedicate actual network adapters to the pod, but I've never attempted that... And that obviously wouldn't scale as it's hardware