4 ms·

From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminat

by krab 5y ago

From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminating pod from the service endpoints, so it won't receive new connections. Therefore, if the application can serve all remaining queries during the grace period (30s by default), it should be fine. Probes are useful for ensuring long term health and also for determining when to send traffic to a starting pod. They aren't strictly necessary in the termination process. > I suspect your early attempts at setting this up safely also failed. IMHO Kubernetes is too complicated for many use cases. The zero downtime deployment may require some testing, but it's definitely supported. It's not a rare requirement. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...

otterley 5y ago
First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.
- bogomipz 5y ago
  >"Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism" Could you or someone else say which LBs are currently bypassing node ports? Also how does a LB actually bypass a nodeport since a service type of Loadbalancer in Kubernetes always uses a nodeport? How exactly does an external Loadbalancer connect to a pod without first connecting to a port on the physical host?
  - otterley 5y ago
    The AWS load balancer controller for Elastic Load Balancers is capable of configuring routing directly to pods in an EKS cluster because those pods get a routable VPC subnet IP address by default. They’re not stuck behind bridges connected to non-routable subnets, so no NAT needed.
- krab 5y ago
  I can agree with that. :-)

by krab 5y ago

otterley 5y ago

First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.

bogomipz 5y ago
>"Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism" Could you or someone else say which LBs are currently bypassing node ports? Also how does a LB actually bypass a nodeport since a service type of Loadbalancer in Kubernetes always uses a nodeport? How exactly does an external Loadbalancer connect to a pod without first connecting to a port on the physical host?
- otterley 5y ago
  The AWS load balancer controller for Elastic Load Balancers is capable of configuring routing directly to pods in an EKS cluster because those pods get a routable VPC subnet IP address by default. They’re not stuck behind bridges connected to non-routable subnets, so no NAT needed.
krab 5y ago
I can agree with that. :-)