3 ms·

I don't have a lot of experience with production k8s so I haven't seen too many issues. I have seen PVCs get stuck though. Recently, I was unable to do a rollin

I don't have a lot of experience with production k8s so I haven't seen too many issues. I have seen PVCs get stuck though. Recently, I was unable to do a rolling restart because I messed up and used zonal disks on a regional cluster. The scheduler would assign pods to nodes in the "wrong" zone and then the disks could never be attached properly. The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).

dilyevsky 4y ago
9/10 times stuckness is a bug in particular csi or node image that you use. Those are fixable. The zonal scheduling thing I believe was fixed a while ago
- powerhour 4y ago
  Ah, I've been on 1.19 for a while so I'm definitely behind. I'm using GKE regional clusters and as such cannot do zero downtime upgrades to the control plane (they break DNS) so it's risky upgrading.
  - dilyevsky 4y ago
    Hm the topology-aware provisioning should be fixed since at least 1.12/13 and is working well on our self-managed regional clusters. I'd bug GKE support for this... Just curious - how does GKE control plane upgrade break DNS?
    - powerhour 4y ago
      I don't have access to GKE support, unfortunately, so the best I can do is submit product feedback. The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.