24 ms·

What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfiguratio

What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations?

tamalsaha001 5y ago
Both for different reasons!
raesene9 5y ago
with the usual caveat that it depends on your threat model, I'd say that having separate clusters is likely to provide better segregation. Of course that reduces the benefits of Kubernetes from a cost perspective, and increases administrative overhead, so it's a trade-off.
joncp 5y ago
Hmm... seems to call for a tool for a cluster-of-clusters. If "kubernetes" is Greek for the ship's pilot, what's the name for the captain or maybe the admiral of the armada?
- adolph 5y ago
  This is how GKE Anthos works on prem
- sascha_sl 5y ago
  It exists, sort of. https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/
  - antonvs 5y ago
    More directly relevant is Cluster Federation: https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
- cmckn 5y ago
  Rancher.
- boilerupnc 5y ago
  Cluster-as-a-Service :-) Hive for OpenShift [0] - Provisioning of the K8s Fleet Advanced Cluster Management [1] - Management of the K8s Fleet [0] https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-service https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-se... [1] https://cloud.redhat.com/products/advanced-cluster-management https://cloud.redhat.com/products/advanced-cluster-managemen... [Disclosure: I'm an IBMer]
- baq 5y ago
  "Nauarch, in ancient Greece, an admiral or supreme commander of the navy, used as an official title primarily in Sparta in the late 5th and early 4th centuries bc." - google cites britannica (!)
- antonvs 5y ago
  One of the several tools in this space is called Admiralty: https://caylent.com/kubernetes-cluster-federation-with-admiralty https://caylent.com/kubernetes-cluster-federation-with-admir... Google says the Greek for that is Ναυαρχείο. Now if you see a product named Navarxeio or Navarcheio, you'll know what it is.
- CorralPeltzer 5y ago
  https://github.com/karmada-io/karmada https://github.com/karmada-io/karmada
bsamuels 5y ago
If I could go back, single cluster. Any benefits you get from going multi-cluster can be achieved by configuring a single cluster correctly.
- agilob 5y ago
  >a single cluster correctly Can you elaborate?
  - antonvs 5y ago
    At a high level, almost anything you would want to use multiple clusters for can be done on a single cluster, using e.g. node pools, affinity, and taints to ensure that workloads only run on the machines you want them to. As a simple example, you can set up a separate node pool for production, and use node affinity and/or taints to ensure that only production workloads can run there. One exception, as other have mentioned, is blast radius - with a single cluster, a problem with Kubernetes itself could take down everything.
    - dharmab 5y ago
      Another issue is scaling limits. We've found a few dozen ways to break a cluster by scaling along a certain axis. (Most are not related to "vanilla" Kubernetes but the backing cloud provider or specific add-on components.)
    - allset_ 5y ago
      Other management tasks are easier when you have separate clusters, such as applying environment-specific OPA policies and not having to filter them based on labels or annotations you hope everyone is using correctly.
- kelseyhightower 5y ago
  The one benefit you get is protection from bugs in Kubernetes itself and a reduced blast radius. Even if you could produce a secure and H/A cluster, you still leave yourself open to Kubernetes bugs and configuration mistakes such as adding a network policy that blocks all communication across all namespaces. Multiple clusters protects you from these types of configuration mistakes by reducing the blast radius and providing an additional landing zone to roll out changes over time.
dharmab 5y ago
At our very large org we do both. At least two clusters per region to isolate platform changes, all hardened to the same standards using automated tooling.
zzyzxd 5y ago
You can't skip "spending a ton of time hardening one cluster" anyways. Having multiple clusters may help reduce the blast radius of _certain_ attacks, to some degree. However, managing multiple clusters is a lot more difficult than managing one, and you will potentially replicate bad practices, vulnerabilities to multiple places and increase maintenance burden.
outworlder 5y ago
> What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations? Not sure this is a valid dichotomy. If you are spinning up multiple clusters, you are presumably doing so in an automated fashion. If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.
- kortilla 5y ago
  > If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. This is why cloud providers don’t stick you on the same k8s cluster as another tenant. > Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk. It’s not really clear what you’re trying to say here. If someone compromises credentials shared between all clusters that’s the same as compromising credentials used by one mega cluster.
  - RandomThrow321 5y ago
    > Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. But if the clusters are configured similarly, a flaw in one is likely present in the others. GPs point is that if you invest in hardening, you can easily apply it to multiple clusters. > It’s not really clear what you’re trying to say here. I assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.).
    - kortilla 5y ago
      > But if the clusters are configured similarly, a flaw in one is likely present in the others. That doesn’t matter. The point is that you isolate applications/tenants into different clusters. So if someone exploits their own, they haven’t gained access to some other application. > assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.). That doesn’t even make sense though. In our strawman scenario these are cookie cutter things. Many is not more vulnerable than one in this case.
kenm47 5y ago
what about tooling that hardens it across clusters?

What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations?

tamalsaha001 5y ago

Both for different reasons!

raesene9 5y ago

with the usual caveat that it depends on your threat model, I'd say that having separate clusters is likely to provide better segregation. Of course that reduces the benefits of Kubernetes from a cost perspective, and increases administrative overhead, so it's a trade-off.

joncp 5y ago

Hmm... seems to call for a tool for a cluster-of-clusters. If "kubernetes" is Greek for the ship's pilot, what's the name for the captain or maybe the admiral of the armada?

adolph 5y ago
This is how GKE Anthos works on prem
sascha_sl 5y ago
It exists, sort of. https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/
- antonvs 5y ago
  More directly relevant is Cluster Federation: https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
cmckn 5y ago
Rancher.
boilerupnc 5y ago
Cluster-as-a-Service :-) Hive for OpenShift [0] - Provisioning of the K8s Fleet Advanced Cluster Management [1] - Management of the K8s Fleet [0] https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-service https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-se... [1] https://cloud.redhat.com/products/advanced-cluster-management https://cloud.redhat.com/products/advanced-cluster-managemen... [Disclosure: I'm an IBMer]
baq 5y ago
"Nauarch, in ancient Greece, an admiral or supreme commander of the navy, used as an official title primarily in Sparta in the late 5th and early 4th centuries bc." - google cites britannica (!)
antonvs 5y ago
One of the several tools in this space is called Admiralty: https://caylent.com/kubernetes-cluster-federation-with-admiralty https://caylent.com/kubernetes-cluster-federation-with-admir... Google says the Greek for that is Ναυαρχείο. Now if you see a product named Navarxeio or Navarcheio, you'll know what it is.
CorralPeltzer 5y ago
https://github.com/karmada-io/karmada https://github.com/karmada-io/karmada

bsamuels 5y ago

If I could go back, single cluster. Any benefits you get from going multi-cluster can be achieved by configuring a single cluster correctly.

agilob 5y ago
>a single cluster correctly Can you elaborate?
- antonvs 5y ago
  At a high level, almost anything you would want to use multiple clusters for can be done on a single cluster, using e.g. node pools, affinity, and taints to ensure that workloads only run on the machines you want them to. As a simple example, you can set up a separate node pool for production, and use node affinity and/or taints to ensure that only production workloads can run there. One exception, as other have mentioned, is blast radius - with a single cluster, a problem with Kubernetes itself could take down everything.
  - dharmab 5y ago
    Another issue is scaling limits. We've found a few dozen ways to break a cluster by scaling along a certain axis. (Most are not related to "vanilla" Kubernetes but the backing cloud provider or specific add-on components.)
  - allset_ 5y ago
    Other management tasks are easier when you have separate clusters, such as applying environment-specific OPA policies and not having to filter them based on labels or annotations you hope everyone is using correctly.
kelseyhightower 5y ago
The one benefit you get is protection from bugs in Kubernetes itself and a reduced blast radius. Even if you could produce a secure and H/A cluster, you still leave yourself open to Kubernetes bugs and configuration mistakes such as adding a network policy that blocks all communication across all namespaces. Multiple clusters protects you from these types of configuration mistakes by reducing the blast radius and providing an additional landing zone to roll out changes over time.

dharmab 5y ago

At our very large org we do both. At least two clusters per region to isolate platform changes, all hardened to the same standards using automated tooling.

zzyzxd 5y ago

You can't skip "spending a ton of time hardening one cluster" anyways. Having multiple clusters may help reduce the blast radius of _certain_ attacks, to some degree. However, managing multiple clusters is a lot more difficult than managing one, and you will potentially replicate bad practices, vulnerabilities to multiple places and increase maintenance burden.

outworlder 5y ago

> What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations? Not sure this is a valid dichotomy. If you are spinning up multiple clusters, you are presumably doing so in an automated fashion. If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.

kortilla 5y ago
> If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. This is why cloud providers don’t stick you on the same k8s cluster as another tenant. > Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk. It’s not really clear what you’re trying to say here. If someone compromises credentials shared between all clusters that’s the same as compromising credentials used by one mega cluster.
- RandomThrow321 5y ago
  > Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. But if the clusters are configured similarly, a flaw in one is likely present in the others. GPs point is that if you invest in hardening, you can easily apply it to multiple clusters. > It’s not really clear what you’re trying to say here. I assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.).
  - kortilla 5y ago
    > But if the clusters are configured similarly, a flaw in one is likely present in the others. That doesn’t matter. The point is that you isolate applications/tenants into different clusters. So if someone exploits their own, they haven’t gained access to some other application. > assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.). That doesn’t even make sense though. In our strawman scenario these are cookie cutter things. Many is not more vulnerable than one in this case.

kenm47 5y ago

what about tooling that hardens it across clusters?