4 ms·
Running a separate cluster for every service assures high overhead and poor utilization. Fine if you can afford it, but be aware that you are paying it.
by jeffbee 5y ago
Running a separate cluster for every service assures high overhead and poor utilization. Fine if you can afford it, but be aware that you are paying it.
- q3k 5y agoYeah, especially in production bare metal clusters. If you want N+2 redundancy that's at least 5 physical machines for just the control plane (etcd & apiservers), more if you don't want to colocate worker nodes with that... Even if you have full bare metal automation and thousands of machines that seems like unnecessary waste.
- dharmab 5y agoAdd the cost of administrative services: DNS resolvers, ingress controllers, log forwarders, monitoring (e.g. Prometheus, some exporters), autoscalers, tracing infrastructure, admission controllers, backups/disaster recovery tools (e.g. Velero)... It can add up to millions per year if you aren't auditing your costs.
- dijit 5y agoYeah, that's insane as a concept. One of the larger selling points of Kubernetes was bin packing. Removing that selling point leaves you with... * Orchestration of jobs (restart, start); this can be achieved easily without the complexity of k8s * Sidecar loading; Literally the easiest thing to do with normal VMs. and...?
- dolni 5y agoPacking everything together is a "selling point" until you find that a service can fill up ephemeral storage and take down other services, or consume bandwidth without limit. Let's not forget the potential security implications of not keeping things properly isolated. People who were around when provisioning on bare-metal was still a thing already learned all these lessons. Somehow it seems they have been forgotten by all the people driving hype around Kubernetes.
- dijit 5y agoKubernetes has the concept of limits, especially on ephemeral storage; additionally: if your node becomes unhealthy then the workloads would be rescheduled on another node. I’m super not hypey about kubernetes, mostly because the complexity surrounding networking is opaque and built on a foundation of sand... But let’s not argue things that aren’t true.
- dolni 5y ago> Kubernetes has the concept of limits, especially on ephemeral storage So... why are these issues open? https://github.com/kubernetes/enhancements/issues/1029 https://github.com/kubernetes/enhancements/issues/1029 https://github.com/kubernetes/enhancements/issues/361 https://github.com/kubernetes/enhancements/issues/361 https://github.com/kubernetes/kubernetes/issues/54384 https://github.com/kubernetes/kubernetes/issues/54384 > additionally: if your node becomes unhealthy then the workloads would be rescheduled on another node. Well of course, but you're going to run into that issue (likely) on all of the nodes where the offending service lives. > But let’s not argue things that aren’t true. If what I've said is untrue, looking at open GitHub issues and the Kubernetes documentation is certainly no indication. That's a massive problem all by itself.
- q3k 5y agoThe first issue you've linked concerns quota support for ephemeral storage requests/limits - which is not about the limits themselves, but the ability to set a limit quotas per tenant/namespace. Eg., team A cannot use a total of more than 100G ephemeral storage in total in the cluster. EDIT: No, sorry, it's about using underlying filesystem quotas for limiting ephemeral storage, vs. the current implementation, see the third point below. Also see KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1029-ephemeral-storage-quotas https://github.com/kubernetes/enhancements/tree/master/keps/... The second is a tracking issue for a KEP that has been implemented but is still in alpha/beta. This will be closed when all the related features are stable. There's also some discussion about related functionality that might be added as part of this KEP/design. The third issue is about integrating Docker storage quotas with Kubernetes ephemeral quotas - ie., translating ephemeral storage limits into disk quotas (which would result in -ENOSPC to workloads), vs. the standard kubelet implementation which just kills/evicts workloads that run past their limit. I agree these are difficult to understand if you're not familiar with the k8s development/design process. I also had to spend a few minutes on each one of them to understand what the actual state of the issues is. However, they're in a development issue tracker, and the end-user k8s documentation clearly states that Ephemeral Storage requests/limits works, how it works, and what its limitations are: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#local-ephemeral-storage https://kubernetes.io/docs/concepts/configuration/manage-res...
- Spooky23 5y agoRemember in big companies the internal politics rule the day. It’s cheaper to buy more computers than to become the overlord of computing.
- ffo 5y agoYou don’t exactly need to run a cluster per service ;-) Instead you can choose to collocate services who belong together and form a „domain“. But don‘t go the route and build the almighty one Kubernetes cluster where all your domains run in one single place.
- eliodorro 5y agoDecreases utilization but also decreases coordination between teams (no man-bear-pigs). Also weight the long-term costs of poorly maintained platflorms and infrastructure in desaster cases, security issues or when migrating to other providers. High overhead can be automated away, google ORBOS.