4 ms·

I've been managing a high traffic website running as 100s of Kubernetes microservices. We have 0 issues with 500s due to deployments. As long as you have well

I've been managing a high traffic website running as 100s of Kubernetes microservices. We have 0 issues with 500s due to deployments. As long as you have well set up probes and old pods have enough time to fulfil lingering requests before going away, there's no reason why you should have 500s at all.

otterley 5y ago
There’s no question that it can be worked around by adding arbitrary delays in places, but finding the right delay then becomes a guessing game and a trade-off between safety and paying to run pods that will do no more work. Different applications and L4/L7 transports also require different behaviors. It also requires that the author of the podspec know to implement this workaround, instead of having this knowledge baked into the orchestrator where it rightly belongs. I suspect your early attempts at setting this up safely also failed.
- krab 5y ago
  From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminating pod from the service endpoints, so it won't receive new connections. Therefore, if the application can serve all remaining queries during the grace period (30s by default), it should be fine. Probes are useful for ensuring long term health and also for determining when to send traffic to a starting pod. They aren't strictly necessary in the termination process. > I suspect your early attempts at setting this up safely also failed. IMHO Kubernetes is too complicated for many use cases. The zero downtime deployment may require some testing, but it's definitely supported. It's not a rare requirement. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...
  - otterley 5y ago
    First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.
    - bogomipz 5y ago
      >"Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism" Could you or someone else say which LBs are currently bypassing node ports? Also how does a LB actually bypass a nodeport since a service type of Loadbalancer in Kubernetes always uses a nodeport? How exactly does an external Loadbalancer connect to a pod without first connecting to a port on the physical host?
      - otterley 5y ago
        The AWS load balancer controller for Elastic Load Balancers is capable of configuring routing directly to pods in an EKS cluster because those pods get a routable VPC subnet IP address by default. They’re not stuck behind bridges connected to non-routable subnets, so no NAT needed.
    - krab 5y ago
      I can agree with that. :-)

otterley 5y ago

There’s no question that it can be worked around by adding arbitrary delays in places, but finding the right delay then becomes a guessing game and a trade-off between safety and paying to run pods that will do no more work. Different applications and L4/L7 transports also require different behaviors. It also requires that the author of the podspec know to implement this workaround, instead of having this knowledge baked into the orchestrator where it rightly belongs. I suspect your early attempts at setting this up safely also failed.

krab 5y ago
From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminating pod from the service endpoints, so it won't receive new connections. Therefore, if the application can serve all remaining queries during the grace period (30s by default), it should be fine. Probes are useful for ensuring long term health and also for determining when to send traffic to a starting pod. They aren't strictly necessary in the termination process. > I suspect your early attempts at setting this up safely also failed. IMHO Kubernetes is too complicated for many use cases. The zero downtime deployment may require some testing, but it's definitely supported. It's not a rare requirement. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...
- otterley 5y ago
  First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.
  - bogomipz 5y ago
    >"Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism" Could you or someone else say which LBs are currently bypassing node ports? Also how does a LB actually bypass a nodeport since a service type of Loadbalancer in Kubernetes always uses a nodeport? How exactly does an external Loadbalancer connect to a pod without first connecting to a port on the physical host?
    - otterley 5y ago
      The AWS load balancer controller for Elastic Load Balancers is capable of configuring routing directly to pods in an EKS cluster because those pods get a routable VPC subnet IP address by default. They’re not stuck behind bridges connected to non-routable subnets, so no NAT needed.
  - krab 5y ago
    I can agree with that. :-)