7 ms·

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’

Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crosstalk your control nodes can take, but that’s your architecture team’s job?

tonfreed 2y ago
I work for a large corp and we have three for apps (dev, integrated testing, prod) plus I think two or three more for the platform team that I don't interact with. 47 seems horrendously excessive
m00x 2y ago
Yeah, it's not very smart. I'm at a company with a $50B+ MC and we run prod on a cluster, staging on another one, then it's tooling clusters like dev spaces, ML, etc. I think in total we have 6 of 7 for ~1000 devs and thousands of nodes. It makes sense that getting off k8s helped them if they were using it incorrectly.
pm90 2y ago
There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).
- oofbey 2y ago
  Honest question - why would you want stateful workloads in a separate cluster from stateless? Why not just use a namespace?
  - pm90 2y ago
    because namespaces aren’t a failure boundary. If your api gets hosed, you can create a new cluster and tear down the old one and call it a day. With a stateful cluster, you can’t do that. As such you put in a lot more care with eg k8s upgrades, or introducing new controllers or admissions/mutating webhooks.
- yjftsjthsd-h 2y ago
  > It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation Are these not covered by taint/toleration? I guess maybe isolation depending on what exactly they're demanding but even then I'd think it could work.
  - pm90 2y ago
    Yes only at the node (data plane) level but not the api (control plane).
- rvense 2y ago
  > It helps Can you elaborate on that? What does it do?
- raverbashing 2y ago
  This doesn't make sense to me, and I feel like this is "holding it wrong" But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it
- wasmitnetzen 2y ago
  Yes, a few, maybe even 10, 12, but 47? It's also a prime number, so it's not something like running each thing three times for dev, stage and prod.
  - pm90 2y ago
    yeahhhh 47 seems insane
- lenkite 2y ago
  Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.
  - mdaniel 2y ago
    I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limit https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi... I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something
    - lenkite 2y ago
      It isn't a mere warning. It is strongly recommended as the upper limit. https://www.perfectscale.io/blog/etcd-8gb https://www.perfectscale.io/blog/etcd-8gb https://github.com/etcd-io/etcd/issues/9771 https://github.com/etcd-io/etcd/issues/9771 And yes, I agree not allowing a pluggable replacement is really stupid.
      - mdaniel 2y ago
        > https://github.com/etcd-io/etcd/issues/9771 https://github.com/etcd-io/etcd/issues/9771 > stale bot marked this as completed (by fucking closing it) Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly
jan_g 2y ago
I'm wondering the same. Either they are quite a big company, so such infrastructure comes naturally from many products/teams or their use case is to be in the clusters business (provisioning and managing k8s clusters for other companies). In both cases I'd say there should be a dedicated devops team that knows their way around k8s. Other than that, the experience I have is that using a managed solution like EKS and one cluster per env (dev, staging, prod) with namespaces to isolate projects takes you a long way. Having used k8s for years now, I'm probably biased, but in general I disagree with many of the k8s-related posts that are frequently upvoted on the front page. I find it gives me freedom, I can iterate fast on services, change structure easily without worrying too much about isolation, networking and resources. In general I feel more nimble than I used to before k8s.
karmarepellent 2y ago
We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster. After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare production and test clusters either unless you have a good reason.
- oblio 2y ago
  Kubernetes upgrades? Don't those risk bricking everything with just 1 environment?
  - karmarepellent 2y ago
    In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good. Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when you design your infrastructure. But when experience tells you that accounting for this potential risk may not be worth it in practice, you might get away with discarding it and keeping your infra lean. (Yes, I am starting to sweat just writing this).
    - mst 2y ago
      "I am cutting this corner because I absolutely cannot make a business case I believe in for doing it the hard (but more correct) way but believe me I am still going to be low key paranoid about it indefinitely" is an experience that I think a lot of us can relate to. I've actually asked for a task to be reassigned to somebody else before now on the grounds that I knew it deserved to be done the simple way but could not for the life of me bring myself to implement that. (the trick is to find a colleague with a task you *can* do that they hate more and arrange a mutually beneficial swap)
motbus3 2y ago
Exactly. You might want to have one cluster per environment so you can test your deployments and rollback plans
torginus 2y ago
Because they have k8s engineers each of whom wants to put it on their resume that they designed and implemented a working cluster in prod.
- astura 2y ago
  Resume Driven Development.
  - AstroJetson 2y ago
    Wouldn't that give you a prod mix of Kubernetes, Docker, Fargate, Rancher, etc?
zelphirkalt 2y ago
Maybe they had 47 different Kubernetes consultants coming in sequentially and each one found something to do different from the last one, but none of them got any time to update their predecessor's stuff.
- deleted 2y ago
  [deleted]
kgeist 2y ago
We have 1 cluster per region (3 regions: Asia, US, EU), multiplied by redundancy clusters = 6 clusters. Then we have test environments, around 20 of them: 20 clusters. Then there are 10s of clusters installed on customers' infra. So 47 clusters isn't really a huge/strange number.
- fragmede 2y ago
  Curious how the 20 test clusters differ by.
- gitaarik 2y ago
  Why 20 test clusters? Do you have more developers than users?
thephyber 2y ago
Don’t know about the writer of the article, but there are some legit reasons to use multiple K8s clusters. Single-tenant environments, segregation of resources into different cloud accounts, redundancy (although there are probably lots of ways to do most of this within a single cluster), 46 different developer / QA / CI clusters, etc.
mrweasel 2y ago
> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47. Out in the real world I've frequently seen companies build a cluster per service, or group of services, to better control load and scaling and again to control network access. It could also be as simple as not all staff being allowed to access the same cluster, due to regulatory concerns. Also you might not want internal tooling to run on the public facing production cluster. You also don't want one service, either do to misconfiguration or design flaws, taking down everything, because you placed all your infrastructure in one cluster. I've seen Kubernetes crash because some service spun out of control and then causing the networking pods to crash, taking out the entire cluster. You don't really want that. Kubernetes doesn't really provide the same type of isolation as something like VmWare, or at least it's not trusted to the same extend.
- rootlocus 2y ago
  > Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47. Why couldn't you do that with a dedicated node pool, namespaces, taints and affinities? This is how we run our simulators and analytics within the same k8s cluster.
  - mrweasel 2y ago
    You could do a dedicated node pool and limit the egress to those nodes, but it seems simpler, as in someone is less likely to provision something incorrect, by having a separate cluster. In my experience companies do not trust Kubernetes to the same extend as they'd trust VLANs and VMs. That's probably not entirely fair, but as you can see from many of the other comments, people find managing Kubernetes extremely difficult to get right. For some special cases you also have regulatory requirements that maybe could be fulfilled by some Kubernetes combination of node pools, namespacing and so on, but it's not really worth the risk. From dealing with clients wanting hosted Kubernetes, I can only say that 100% of them have been running multiple clusters. Sometimes for good reason, other times because hosting costs where per project and it's just easier to price out a cluster, compared to buying X% of the capacity on an existing cluster. One customer I've worked with even ran an entire cluster for a single container, but that was done because no one told the developers to not use that service as an excuse to play with Kubernetes. That was its own kind of stupid.
Foobar8568 2y ago
I had a client who had a single K8 cluster, too much ops for the team, so their idea was to transfer that to each product dev team and thus was born the one K8 per product. They had at least a few 100s of products.
this_user 2y ago
It's just a matter of time before someone releases an orchestration layer for k8s clusters so the absurd Rube Goldberg machine that is modern devops stacks can grow even more complex.
- englishspot 2y ago
  karmada? https://karmada.io/ https://karmada.io/
  - UltraSane 2y ago
    Is it ever a good idea to actually use this?
- teeray 2y ago
  We can call it k8tamari
- jppittma 2y ago
  https://istio.io/latest/ https://istio.io/latest/
  - processunknown 2y ago
    Istio is not a k8s orchestration layer though
- mad_vill 2y ago
  cluster-api in a nutshell
- jhklkjj 2y ago
  This has already happened. There are orgs managing hundreds to thousands of k8s clusters.
andix 2y ago
Isn't one of the strategies also to run one or two backup clusters for any production cluster? Which can take over the workloads if the primary cluster fails for some reason? In a cloud environment the backup cluster can be scaled up quickly if it has to take over, so while it's idling it only requires a few smaller nodes.
- fragmede 2y ago
  You might run a cluster per region, but the whole point of Kubernetes is that it's highly available. What specific piece are you worried about will go down in one cluster that you need two production clusters all the time? Upgrades are a special case where I could see spinning up a backup cluster for.
  - andix 2y ago
    A lot of things can break (hardware, networking, ...). Spanning the workload over multiple clusters in different regions is already satisfying the "backup cluster" recommendation. Many workloads don't need to be multi-region as a requirement. So they might run just on one cluster with the option to fail over to another region in case of an emergency. Running a workload on one cluster at a time (even with some downtime for a manual failover) makes a lot of things much easier. Many workloads don't need 99,99% availability, and nothing awful happens if they are down for a few hours.
levifig 2y ago
To answer your question directly: yes, that's the point. You may have different clusters for different logical purposes but, yes: less clusters, more node groups is a better practice.
mmcnl 2y ago
In my company (large corporate) we only have 3 clusters: dev/acc/prod. Everything runs on it. I love it.

tonfreed 2y ago

I work for a large corp and we have three for apps (dev, integrated testing, prod) plus I think two or three more for the platform team that I don't interact with. 47 seems horrendously excessive

m00x 2y ago

Yeah, it's not very smart. I'm at a company with a $50B+ MC and we run prod on a cluster, staging on another one, then it's tooling clusters like dev spaces, ML, etc. I think in total we have 6 of 7 for ~1000 devs and thousands of nodes. It makes sense that getting off k8s helped them if they were using it incorrectly.

pm90 2y ago

There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).

oofbey 2y ago
Honest question - why would you want stateful workloads in a separate cluster from stateless? Why not just use a namespace?
- pm90 2y ago
  because namespaces aren’t a failure boundary. If your api gets hosed, you can create a new cluster and tear down the old one and call it a day. With a stateful cluster, you can’t do that. As such you put in a lot more care with eg k8s upgrades, or introducing new controllers or admissions/mutating webhooks.
yjftsjthsd-h 2y ago
> It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation Are these not covered by taint/toleration? I guess maybe isolation depending on what exactly they're demanding but even then I'd think it could work.
- pm90 2y ago
  Yes only at the node (data plane) level but not the api (control plane).
rvense 2y ago
> It helps Can you elaborate on that? What does it do?
raverbashing 2y ago
This doesn't make sense to me, and I feel like this is "holding it wrong" But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it
wasmitnetzen 2y ago
Yes, a few, maybe even 10, 12, but 47? It's also a prime number, so it's not something like running each thing three times for dev, stage and prod.
- pm90 2y ago
  yeahhhh 47 seems insane
lenkite 2y ago
Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.
- mdaniel 2y ago
  I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limit https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi... I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something
  - lenkite 2y ago
    It isn't a mere warning. It is strongly recommended as the upper limit. https://www.perfectscale.io/blog/etcd-8gb https://www.perfectscale.io/blog/etcd-8gb https://github.com/etcd-io/etcd/issues/9771 https://github.com/etcd-io/etcd/issues/9771 And yes, I agree not allowing a pluggable replacement is really stupid.
    - mdaniel 2y ago
      > https://github.com/etcd-io/etcd/issues/9771 https://github.com/etcd-io/etcd/issues/9771 > stale bot marked this as completed (by fucking closing it) Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly

jan_g 2y ago

I'm wondering the same. Either they are quite a big company, so such infrastructure comes naturally from many products/teams or their use case is to be in the clusters business (provisioning and managing k8s clusters for other companies). In both cases I'd say there should be a dedicated devops team that knows their way around k8s. Other than that, the experience I have is that using a managed solution like EKS and one cluster per env (dev, staging, prod) with namespaces to isolate projects takes you a long way. Having used k8s for years now, I'm probably biased, but in general I disagree with many of the k8s-related posts that are frequently upvoted on the front page. I find it gives me freedom, I can iterate fast on services, change structure easily without worrying too much about isolation, networking and resources. In general I feel more nimble than I used to before k8s.

karmarepellent 2y ago

We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster. After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare production and test clusters either unless you have a good reason.

oblio 2y ago
Kubernetes upgrades? Don't those risk bricking everything with just 1 environment?
- karmarepellent 2y ago
  In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good. Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when you design your infrastructure. But when experience tells you that accounting for this potential risk may not be worth it in practice, you might get away with discarding it and keeping your infra lean. (Yes, I am starting to sweat just writing this).
  - mst 2y ago
    "I am cutting this corner because I absolutely cannot make a business case I believe in for doing it the hard (but more correct) way but believe me I am still going to be low key paranoid about it indefinitely" is an experience that I think a lot of us can relate to. I've actually asked for a task to be reassigned to somebody else before now on the grounds that I knew it deserved to be done the simple way but could not for the life of me bring myself to implement that. (the trick is to find a colleague with a task you *can* do that they hate more and arrange a mutually beneficial swap)

motbus3 2y ago

Exactly. You might want to have one cluster per environment so you can test your deployments and rollback plans

torginus 2y ago

Because they have k8s engineers each of whom wants to put it on their resume that they designed and implemented a working cluster in prod.

astura 2y ago
Resume Driven Development.
- AstroJetson 2y ago
  Wouldn't that give you a prod mix of Kubernetes, Docker, Fargate, Rancher, etc?

zelphirkalt 2y ago

Maybe they had 47 different Kubernetes consultants coming in sequentially and each one found something to do different from the last one, but none of them got any time to update their predecessor's stuff.

deleted 2y ago
[deleted]

kgeist 2y ago

We have 1 cluster per region (3 regions: Asia, US, EU), multiplied by redundancy clusters = 6 clusters. Then we have test environments, around 20 of them: 20 clusters. Then there are 10s of clusters installed on customers' infra. So 47 clusters isn't really a huge/strange number.

fragmede 2y ago
Curious how the 20 test clusters differ by.
gitaarik 2y ago
Why 20 test clusters? Do you have more developers than users?

thephyber 2y ago

Don’t know about the writer of the article, but there are some legit reasons to use multiple K8s clusters. Single-tenant environments, segregation of resources into different cloud accounts, redundancy (although there are probably lots of ways to do most of this within a single cluster), 46 different developer / QA / CI clusters, etc.

mrweasel 2y ago

> Isn’t the point of kubernetes that you can run your entire infra in a single cluster I've never seen that, but yes 47 seems like a lot. Often you'd need production, staging, test, development, something like that. Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47. Out in the real world I've frequently seen companies build a cluster per service, or group of services, to better control load and scaling and again to control network access. It could also be as simple as not all staff being allowed to access the same cluster, due to regulatory concerns. Also you might not want internal tooling to run on the public facing production cluster. You also don't want one service, either do to misconfiguration or design flaws, taking down everything, because you placed all your infrastructure in one cluster. I've seen Kubernetes crash because some service spun out of control and then causing the networking pods to crash, taking out the entire cluster. You don't really want that. Kubernetes doesn't really provide the same type of isolation as something like VmWare, or at least it's not trusted to the same extend.

rootlocus 2y ago
> Then you'd add an additional cluster for running auxiliary service, this is services that has special network access or are not related to you "main product". Maybe a few of these. Still there's a long way to 47. Why couldn't you do that with a dedicated node pool, namespaces, taints and affinities? This is how we run our simulators and analytics within the same k8s cluster.
- mrweasel 2y ago
  You could do a dedicated node pool and limit the egress to those nodes, but it seems simpler, as in someone is less likely to provision something incorrect, by having a separate cluster. In my experience companies do not trust Kubernetes to the same extend as they'd trust VLANs and VMs. That's probably not entirely fair, but as you can see from many of the other comments, people find managing Kubernetes extremely difficult to get right. For some special cases you also have regulatory requirements that maybe could be fulfilled by some Kubernetes combination of node pools, namespacing and so on, but it's not really worth the risk. From dealing with clients wanting hosted Kubernetes, I can only say that 100% of them have been running multiple clusters. Sometimes for good reason, other times because hosting costs where per project and it's just easier to price out a cluster, compared to buying X% of the capacity on an existing cluster. One customer I've worked with even ran an entire cluster for a single container, but that was done because no one told the developers to not use that service as an excuse to play with Kubernetes. That was its own kind of stupid.

Foobar8568 2y ago

I had a client who had a single K8 cluster, too much ops for the team, so their idea was to transfer that to each product dev team and thus was born the one K8 per product. They had at least a few 100s of products.

this_user 2y ago

It's just a matter of time before someone releases an orchestration layer for k8s clusters so the absurd Rube Goldberg machine that is modern devops stacks can grow even more complex.

englishspot 2y ago
karmada? https://karmada.io/ https://karmada.io/
- UltraSane 2y ago
  Is it ever a good idea to actually use this?
teeray 2y ago
We can call it k8tamari
jppittma 2y ago
https://istio.io/latest/ https://istio.io/latest/
- processunknown 2y ago
  Istio is not a k8s orchestration layer though
mad_vill 2y ago
cluster-api in a nutshell
jhklkjj 2y ago
This has already happened. There are orgs managing hundreds to thousands of k8s clusters.

andix 2y ago

Isn't one of the strategies also to run one or two backup clusters for any production cluster? Which can take over the workloads if the primary cluster fails for some reason? In a cloud environment the backup cluster can be scaled up quickly if it has to take over, so while it's idling it only requires a few smaller nodes.

fragmede 2y ago
You might run a cluster per region, but the whole point of Kubernetes is that it's highly available. What specific piece are you worried about will go down in one cluster that you need two production clusters all the time? Upgrades are a special case where I could see spinning up a backup cluster for.
- andix 2y ago
  A lot of things can break (hardware, networking, ...). Spanning the workload over multiple clusters in different regions is already satisfying the "backup cluster" recommendation. Many workloads don't need to be multi-region as a requirement. So they might run just on one cluster with the option to fail over to another region in case of an emergency. Running a workload on one cluster at a time (even with some downtime for a manual failover) makes a lot of things much easier. Many workloads don't need 99,99% availability, and nothing awful happens if they are down for a few hours.

levifig 2y ago

To answer your question directly: yes, that's the point. You may have different clusters for different logical purposes but, yes: less clusters, more node groups is a better practice.

mmcnl 2y ago

In my company (large corporate) we only have 3 clusters: dev/acc/prod. Everything runs on it. I love it.