4 ms·

A single k8s cluster spanning multiple datacenters feels mind boggling to me. I know it's not exactly uncommon for HA even if you just have a little one in your

A single k8s cluster spanning multiple datacenters feels mind boggling to me. I know it's not exactly uncommon for HA even if you just have a little one in your cloud provider of choice but I'm sure it's a totally different beast than the toy ones I've created.

Havoc 3y ago
I guess if you have really fast fibre connections between them then the separation isn’t all that separate even with distance
ekimekim 3y ago
At its core Kubernetes is mostly a HTTP API being used to sync state between nodes. I see no reason that part shouldn't work even across the world, albeit at the cost of slower syncing of state (eg. pods taking longer to be created after a scale-up). That HTTP API is backed by etcd which uses Raft and that is where running over a large area is more likely to cause problems. One approach would be to keep the etcd instances in one region (and probably also co-locate the scheduler, controller manager there), while having far-flung worker nodes. This creates a risk of losing the control plane but in most cases services would keep running (but would be unable to react to any further issues until the control plane recovers). You would also want to carefully design your workloads with topology constraints and region-specific services to avoid high application layer latencies though. Overall it's a fun thought experiment. In practical terms I think cross-datacenter in a small geographic area would work fine but I probably wouldn't want to run a single worldwide cluster, both for the reasons above and for other scaling reasons.
- oceanplexian 3y ago
  You could implement geographic taints and tolerations and constrain certain workloads to certain regions. Lots of places span clusters across several AZs and have the same problem in theory. However I don’t personally like it, because you’re engineering a cross-datacenter failure domains which is usually a bad design decision unless done for specific reasons. And as for the the control plane, for it to reliably span multiple locations while also tolerating failures, you now have to deal with all kinds of weird split brain scenarios, so most just run multiple clusters instead of rewriting k8s for this kind of design.
iampims 3y ago
It’s no different than running EKS in Multi-AZ configuration, except for the cross-az egress fees…

Havoc 3y ago

I guess if you have really fast fibre connections between them then the separation isn’t all that separate even with distance

ekimekim 3y ago

At its core Kubernetes is mostly a HTTP API being used to sync state between nodes. I see no reason that part shouldn't work even across the world, albeit at the cost of slower syncing of state (eg. pods taking longer to be created after a scale-up). That HTTP API is backed by etcd which uses Raft and that is where running over a large area is more likely to cause problems. One approach would be to keep the etcd instances in one region (and probably also co-locate the scheduler, controller manager there), while having far-flung worker nodes. This creates a risk of losing the control plane but in most cases services would keep running (but would be unable to react to any further issues until the control plane recovers). You would also want to carefully design your workloads with topology constraints and region-specific services to avoid high application layer latencies though. Overall it's a fun thought experiment. In practical terms I think cross-datacenter in a small geographic area would work fine but I probably wouldn't want to run a single worldwide cluster, both for the reasons above and for other scaling reasons.

oceanplexian 3y ago
You could implement geographic taints and tolerations and constrain certain workloads to certain regions. Lots of places span clusters across several AZs and have the same problem in theory. However I don’t personally like it, because you’re engineering a cross-datacenter failure domains which is usually a bad design decision unless done for specific reasons. And as for the the control plane, for it to reliably span multiple locations while also tolerating failures, you now have to deal with all kinds of weird split brain scenarios, so most just run multiple clusters instead of rewriting k8s for this kind of design.

iampims 3y ago

It’s no different than running EKS in Multi-AZ configuration, except for the cross-az egress fees…