4 ms·
This is something that plagued me when I started using Kubernetes because out of the box you will get 503s with a load balanced service. If you're on EKS with
by nickjj 5y ago
This is something that plagued me when I started using Kubernetes because out of the box you will get 503s with a load balanced service.
If you're on EKS with the AWS Load Balancer Controller, after some research I ended up stumbling on https://github.com/foriequal0/pod-graceful-drain https://github.com/foriequal0/pod-graceful-drain and never looked back. This has been working great so far. You install it once into your cluster and you're done. No dropped connections during a rollout and no need to set up preStop lifecycle hooks. The only downside I've seen so far is it takes slightly longer (about a minute or so) to terminate pods but I'd much rather have that than have to worry about 503s in production during a rollout.
- pm90 5y agoThis is a really neat idea, to intercept the pod deletion requests using an admission hooks controller and just delay it for a bit. Thanks for sharing!
- sascha_sl 5y ago"load balanced service" means very little here typically a service of type LoadBalancer is the same as a NodePort, except the CM will not write the status field, expecting some other piece of software to handle it. the stock AWS controller does this so badly, you will reach one random pod per node, and probably not even on the same node, and sometimes the same pod twice, and this connection usually persists until the pod dies so scaling up is near useless. that's not a kubernetes problem kube-proxy and ingress controllers usually rewrite their routing within 1-2 seconds - but I've seen some other designs that take up to 10 minutes to properly set up their load balancers (GKE). once again, not a kubernetes problem