3 ms·

It's not an ignorant question! Running 47 different clusters is insane.

szszrk 2y ago
While I'm pretty sure the article is clickbait (can't tell, paywalled too soon), having many clusters ain't dumb nowadays. The automation of Kubernetes maintenance is great nowadays. For any case, bare metal, onprem, public cloud managed. Leveraging that makes it easier to manage multiple clusters and give each team/project it's own cluster, than implement proper mechanisms on a single cluster. Like proper rbac between projects, network boundaries etc.. Nowadays you can easily move that complexity one layer up and treat whole clusters as some volatile component that is defined in code and is also not a snowflake. That way you can let each team/project have core things different and managed by themselves, like implementing their contrasting opinions on network meshes, or CRDs that would otherwise be in conflict etc. The overhead is not that huge, or at least doesn't have to be. My test clusters with multiple environments of my apps consume around 4GB of mem in total (aside of my apps themselves), that includes any k8s stuff, logging aggregation, metrics aggregation and so on. You don't even have to manage your own control plane - cloud can give you a shared one (like Azure has in two lower tiers), or use services that provide you just the control plane, while nodes are on whatever hardware (like scaleway kosmos). So yeah, it's not for everyone, but it surely can grow to those numbers of clusters, especially if you multiply by dev/staging/qa/prod for each team and add some Infra to actual tests of infra/IaC. Although, why they had such an overhead is a mystery to me, would be cool to see that part described.
benterix 2y ago
Speaking in absolute numbers without any reference point makes no sense.
- reissbaker 2y ago
  The article states they moved all of their stateless stuff to ECS, stateful stuff to Docker containers on EC2, batch jobs to AWS Batch, and event-driven stuff to AWS Lambda. Previously they ran 47 different K8s clusters on 3 different clouds. Given that all of their needs could be satisfied with 4 AWS services, and for nearly half a million dollars a year cheaper than their previous setup, I think running 47 different clusters in 3 clouds was insane.
  - benterix 2y ago
    Yes, in this context I don't see any reason for that number of separate clusters.

While I'm pretty sure the article is clickbait (can't tell, paywalled too soon), having many clusters ain't dumb nowadays. The automation of Kubernetes maintenance is great nowadays. For any case, bare metal, onprem, public cloud managed. Leveraging that makes it easier to manage multiple clusters and give each team/project it's own cluster, than implement proper mechanisms on a single cluster. Like proper rbac between projects, network boundaries etc.. Nowadays you can easily move that complexity one layer up and treat whole clusters as some volatile component that is defined in code and is also not a snowflake. That way you can let each team/project have core things different and managed by themselves, like implementing their contrasting opinions on network meshes, or CRDs that would otherwise be in conflict etc. The overhead is not that huge, or at least doesn't have to be. My test clusters with multiple environments of my apps consume around 4GB of mem in total (aside of my apps themselves), that includes any k8s stuff, logging aggregation, metrics aggregation and so on. You don't even have to manage your own control plane - cloud can give you a shared one (like Azure has in two lower tiers), or use services that provide you just the control plane, while nodes are on whatever hardware (like scaleway kosmos). So yeah, it's not for everyone, but it surely can grow to those numbers of clusters, especially if you multiply by dev/staging/qa/prod for each team and add some Infra to actual tests of infra/IaC. Although, why they had such an overhead is a mystery to me, would be cool to see that part described.

benterix 2y ago

Speaking in absolute numbers without any reference point makes no sense.

reissbaker 2y ago
The article states they moved all of their stateless stuff to ECS, stateful stuff to Docker containers on EC2, batch jobs to AWS Batch, and event-driven stuff to AWS Lambda. Previously they ran 47 different K8s clusters on 3 different clouds. Given that all of their needs could be satisfied with 4 AWS services, and for nearly half a million dollars a year cheaper than their previous setup, I think running 47 different clusters in 3 clouds was insane.
- benterix 2y ago
  Yes, in this context I don't see any reason for that number of separate clusters.