3 ms·

We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that spac

• sleepybrett 4y ago
  A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many volumes per node and I used to see problems with ebs volumes 'unmounting' from nodes getting stuck. The later was always problematic and required an admin with a hammer. However i've never seens a 'kube' problem per se, they've all been aws problems.
  • nijave 4y ago
    Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck) Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)