8 ms·

Problems with “graceful shutdown” in Kubernetes (2019)

255 points by richardfey 5y ago

spullara 5y ago
Tomcat had similar behavior when I was using it except it would bind the listener before it was ready to serve traffic with similar results.
motoboi 5y ago
There is a bug in GKE causing 502 when using CNI network. The big is triggered by port names in ingresses. Use port numbers and you should be good to go.
TurningCanadian 5y ago
Check out https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream https://kubernetes.github.io/ingress-nginx/user-guide/nginx-... if you're running nginx. Consider setting it to true instead of the default (false) --- By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments .
- nhoughto 5y ago
  ah good tip, I don't care about session affinity or custom balancing algos so that works. I'd imagine running in GKE or AWS you would also avoid the DNAT / conntrack overhead as pods by default use a routable VPC IP instead of a magic CNI IP. Would have to test that though. Quote from related issue: The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.
  - sascha_sl 5y ago
    How does this remove conntrack overhead? It should add more, because the node with the ingress controller now has to hold an extra <Cluster IP Cluster Port, Pod IP, Pod Port> mapping, regardless of what CNI is used (flannel in gateway mode also eliminates this overhead, by the way - you just need to make sure there is nothing like the default EC2 source destination check in place).
- nopurpose 5y ago
  You just moved problem from ingress controller to kube-proxy (or it's replacement) on the same node. Race condition is still present as before.
- sascha_sl 5y ago
  This is not a good idea, you're doubling the load on conntrack. Maybe ingress-nginx should fix their config generation instead. Had a cluster with at least one change per second to the ingresses, and nginx would regularly just die (and orphan existing requests after 20 seconds).
nhoughto 5y ago
Is this true of the OOTB GKE nginx ingress? Hard to tell by 'load balancer' do they mean nginx ingress reverse proxy? I can imagine the delay between updating the GCP global load balancer service from GKE would be much higher than nginx-ingress reacting to changes in pod health/endpoints. Either way I guess the takeaway is there is a race there between endpoints being updated and those updates propagating, and seems like that isn't handled as perfectly as people assume, and this likely gets worse with node contention and Kube API performance problems.
- cyberpunk 5y ago
  By load balancer they mean internal kubernetes “service” object that a given ingress uses as it’s backing service.
mad_vill 5y ago
the issues I see with kubernetes ingress are more related to an ingress pod going down than the upstream.
- cyberpunk 5y ago
  What controller are you using? I’ve absolutely smashed nginx and the aws elb controllers and never seen them flinch…
blaisio 5y ago
Yes! I think this is a really under-reported issue. It's basically caused by kubernetes doing things without confirming everyone responded to prior status updates. It affects every ingress controller, and it also affects services of type "Load Balancer" and there isn't a real fix. Even if you add a timeout in the pre stop hook, that still might not handle it 100% of the time. IMO it is a design flaw in Kubernetes.
- deleted 5y ago
  [deleted]
- LimaBearz 5y ago
  Not defending the situation of a preStop hook at least in the case of API's k8s can handle it 100%, its just messy. We have a preStop hook of 62s. 60s timeouts are set in our apps, 61s is set on the ALBs (ensuring the ALB is never the cause of the hangup), and 62s on the preStop to make sure nothing has come into the container in the last 62s. Then we set a terminationGracePeriodSeconds of 60 just to make sure it doesn't pop off too fast. This gives us 120s where nothing happens and anything in flight can get to where its going.
  - thecosmicfrog 5y ago
    Yep, same configuration here other than we use 60/65/70 for (admittedly) completely unscientific reasons.
  - chippiewill 5y ago
    > Then we set a terminationGracePeriodSeconds of 60 just to make sure it doesn't pop off too fast. I think the grace period includes the prestop duration doesn't it?
    - bogomipz 5y ago
      The grace period does include the time spent to execute the preStop hook yes.
AaronBBrown 5y ago
This is a design flaw in Kubernetes. The article doesn't really explain what's happening though. The real problem is that there is no synchronization between the ingress controller (which manages the ingress software configuration, e.g. nginx from the Endpoints resources), kube-proxy (which manages iptables rules from the Endpoints resource), and kubelet (which sends the signals to the container). A presStop hook w/ a sleep equivalent to an acceptable timeout will handle the 99%+ cases (and the cases it doesn't will have exceeded your timeout anyhow). Things become more complicated when there are sidecar containers (say an envoy or nginx routing to another container in the same pod) and that often requires shenanigans such as shared emptyDir{} volumes that waits (with fsnotify or similar) for socket files to be closed to ensure requests are fully completed.
- kodah 5y ago
  I mean, technically, you can recreate this scenario on a single host as well. Send a sigterm to an application and try to swap in another instance of it. System fundamentals are at the heart of that problem: SIGTERM is just what it is, it's a signal and an application can choose to acknowledge it and do something or catch it and ignore it. The system also has no way of knowing what the application chose to do. All that to say, I'm not sure it's as much of a flaw in Kubernetes as much as it's the way systems work and Kubernetes is reflecting that.
  - lolc 5y ago
    In my view it is a clear flaw that the signal to terminate can arrive while the server is still getting new requests. Being able to steer traffic based on your knowledge of the state of the system is one of the reasons why you'd want to set up an integrated environment where the load-balancer and servers are controlled from the same process. The time to send the signal is entirely under control of the managing process. It could synchronize with the load-balancer before sending pods the term signal, and I'm unclear why this isn't done.
    - deleted 5y ago
      [deleted]
zeckalpha 5y ago
(2019)
- richardfey 5y ago
  It is still 100% applicable (AFAIK) and informative, with the (2019) in title readers will think it's not relevant anymore?
  - zeckalpha 5y ago
    No, but if they have read it before they may realize it sooner. I have seen (2021) in titles here already.
gscho 5y ago
Wait until you find out that kubernetes secrets aren’t actually secrets but base64 encoded strings.
- twalla 5y ago
  Encryption at rest for secrets can be enabled, the base64 thing is more of an artifact of how JSON serialization works with byte arrays.
- notwedtm 5y ago
  I think K8S secrets get a bad wrap. They are not intended to be secret in the sense that they are "kept from prying eyes by default". The secret object is simply a first-class citizen that differentiates it from a ConfigMap in a way that allows distinct ACL's. Most organizations I know will still use something like ExternalSecret for source control and then populate the Secret with the values once in cluster and to an object with very few access points.
  - gscho 5y ago
    I think calling it a secret when it isn’t gave it a bad wrap. The last time I looked at the documentation it didn’t even clearly describe that it is not a secure object (that may have changed recently). Why call it a secret when it is not even close to one? I guess thing-to-store-secrets-if-you-use-rbac was too long.
    - threeseed 5y ago
      But it can be a secret. You can store Base64-encoded, encrypted data. And you can encode it for example using an external KMS.
      - gscho 5y ago
        Yes I understand that. My point is until you configure it in that way it is not “secret” and the name of the object is a bit misleading, especially when first learning k8s.
rifelpet 5y ago
There's a KubeCon North America talk (also from 2019) that goes into more detail on this very issue including some additional recommendations https://youtu.be/0o5C12kzEDI?t=57 https://youtu.be/0o5C12kzEDI?t=57
nickjj 5y ago
This is something that plagued me when I started using Kubernetes because out of the box you will get 503s with a load balanced service. If you're on EKS with the AWS Load Balancer Controller, after some research I ended up stumbling on https://github.com/foriequal0/pod-graceful-drain https://github.com/foriequal0/pod-graceful-drain and never looked back. This has been working great so far. You install it once into your cluster and you're done. No dropped connections during a rollout and no need to set up preStop lifecycle hooks. The only downside I've seen so far is it takes slightly longer (about a minute or so) to terminate pods but I'd much rather have that than have to worry about 503s in production during a rollout.
- pm90 5y ago
  This is a really neat idea, to intercept the pod deletion requests using an admission hooks controller and just delay it for a bit. Thanks for sharing!
- sascha_sl 5y ago
  "load balanced service" means very little here typically a service of type LoadBalancer is the same as a NodePort, except the CM will not write the status field, expecting some other piece of software to handle it. the stock AWS controller does this so badly, you will reach one random pod per node, and probably not even on the same node, and sometimes the same pod twice, and this connection usually persists until the pod dies so scaling up is near useless. that's not a kubernetes problem kube-proxy and ingress controllers usually rewrite their routing within 1-2 seconds - but I've seen some other designs that take up to 10 minutes to properly set up their load balancers (GKE). once again, not a kubernetes problem
jugg1es 5y ago
Everyone that uses kubernetes must run into this at some point but we all end up doing exactly what is in this article. But we do it quietly because we all think we must be idiots since there is virtually no one else writing anything about it.
merb 5y ago
btw. this can be fixed with readiness probes. and applications should implement them. i.e. application still gets traffic, readiness probe fails because of SIGTERM, application will respond with an error on the probe, the loadbalancer/service sees that and removes the endpoint/service the application can still respond to the inflight requests since it should not stop directly on sigterm, if everything inflight was done it should stop. (this also works with non http, if SIGTERM is handled)
- esprehn 5y ago
  Yeah this is resolved with lameduck mode: https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ Ex. https://coredns.io/plugins/health/ https://coredns.io/plugins/health/
- blaisio 5y ago
  No, sorry but I don't think you understood the article. Readiness probes don't fix this. In fact, they could exacerbate the problem (although you obviously need them anyway). The problem is Kubernetes and the Load Balancer react to pod status changes at different times. (In fact, it's worse - different Kubernetes components also react to status changes at different times.) The load balancer is almost always too slow to react, so, it sends traffic to an instance that has already started shutting down.
  - merb 5y ago
    Most loadbalancers do use the readiness probes tough
- agilob 5y ago
  >btw. this can be fixed with readiness probes. and applications should implement them. What rate would you set on readiness probes for high throughput services (100k requests per second) to keep +99% uptime? :)
yonran 5y ago
This peculiar behavior (where the Service health check is unaware of the Deployment’s known instances) mirrors Google Compute Engine (where the httpLoadBalancer’s healthCheck is independent of the instanceGroupManager’s known instances). If you want your program to exit as soon as possible rather than waiting for a SIGKILL from the instance group manager, you have to hard-code the health check timeouts like so: 1. After a SIGTERM, the shutdownHook should keep the HTTP server running. Future /@status requests must return an error, but user requests must still succeed! 2. The shutdownHook sleeps for a minimum of the load balancer health check’s checkIntervalSec * unhealthyThreshold + timeoutSec (which by the way must be less than the instanceGroupManager’s health check’s checkIntervalSec * unhealthyThreshold if it uses the same endpoint) 3. Now the load balancer should not be sending new requests. The shutdownHook then waits for any existing requests to drain. 4. After requests drain, the shutdownHook can finally exit gracefully. It is annoying to have to wait for the health check’s delay (rather than simply draining existing requests as in AWS), but it seems to be necessary for Google-designed load balancers and instance group managers.
- paulfurtado 5y ago
  With kubernetes you can at least add a preStop hook that sleeps for 60-120sec if the app is not designed to handle SIGTERM. The pod enters the terminating state just prior to executing the preStop hook.
dboreham 5y ago
Was hoping this was an article about a k8s cluster that become sentient then took control of its own power supply, like in that movie...
cphoover 5y ago
Wait... sigterm can be handled to gracefully handle shutdown of your pod... i.e. finish handling all existing requests before closing. I'm not sure the issue here, that sigterm is sent before removal from the load balancer? If you handle the sigterm gracefully doesnt this solve your issue... I'm no k8s expert so maybe I am misunderstanding the problem
- notreallyserio 5y ago
  > I'm not sure the issue here, that sigterm is sent before removal from the load balancer? That's exactly the issue. It'd be better for the orchestration system to handle removing the service from the LB in response to a configuration change than to wait for e.g. health checks to fail. The SIGTERM method means you have to modify every service to accept SIGTERM (no big deal) for some amount of time (fiddly configuration detail) and in the mean time wait for the LB to see the terminating pod's health check fail so it can stop sending it traffic. If we flipped it around a bit, having the LB remove the traffic before sending a term signal, we could be more assured that poorly behaving pods (pods that die too quickly or too slowly) don't result in user-facing issues.
joshuamorton 5y ago
Isn't this fundamentally k8s treating the pods as cattle? Your service could fail ungracefully at any time for all kinds of reasons, so the client needs to be somewhat prepared (with retries or whatever). If your client can retry then this doesn't matter unless you have no replication? Am I missing something?
- yashap 5y ago
  Clients can only retry idempotent requests safely. Standard POSTs, for example, cannot be safely retried - you can end up with duplicates.
aequitas720 5y ago
Shameless plug... This is the go package I wrote to handle this and use in all of my servers: https://github.com/abursavich/graceful https://github.com/abursavich/graceful
paxys 5y ago
Their proposed solution isn't really a solution either, and will probably make things worse. Current behavior - Kubernetes sends SIGTERM signal to your container, it starts the shutdown process, stops the web server, and finishes up the requests in flight. Ingress controller continues to send some requests to the pod regardless, and they fail to connect. The client can handle these failures in whatever way it deems fit. Their solution - Kubernetes sends SIGTERM signal, container ignores it and keeps taking new requests. After a while the process is forcefully killed, all requests in flight fail in unpredictable ways, and other resources (like DB connections) may be left dangling.
potamic 5y ago
Would this be mitigated if the ingress load balancer also did health checks with the pods? So when a pod goes into shutdown state, the ingress can detect and stop routing traffic.
- Too 5y ago
  Health checks typically only run at like 30sec frequency. Even if you set it to 1sec that’s still a big window for requests to slip in before ingress detects the pod is gone.
benlivengood 5y ago
Non-graceful shutdowns must be handled anyway (generally with client-side retries), so this shouldn't be an issue in practice. Kubernetes is intended for distributed/clustered services, not traditional HA services.
rdsubhas 5y ago
This is why we inject `preStop: ["/bin/sleep", "30"]` in every kubernetes PodSpec via an admission controller. Kubernetes service/ingress are designed to be event-driven and eventually consistent. For example, the nginx ingress controller has a direct list of Pod IPs as backends, and will remove them asynchronously based on receiving pod lifecycle events. In real life, it means the pod will be shutdown long before the endpoint is actually removed, leading to error rates. The preStop hook is necessary to avoid request failures, at the expense of longer pod shutdown.
deepsun 5y ago
Is it still the issue with GKE NEGs? NEG is "network endpoint groups" which basically means that Load Balancers deal not with nodes, but with pods directly. So LBs can know what state the pod is. You can even avoid using Ingress (instead, annotate your k8s Service with neg name, and use backend service for that neg in LB), so it'd be possible to combine k8s and non-k8s services under the same Load Balancer, e.g. for migration. But I'm afraid NEG or not-NEG, LB still relies on health checks anyway, so it doesn't matter.

255 points by richardfey 5y ago

Tomcat had similar behavior when I was using it except it would bind the listener before it was ready to serve traffic with similar results.

motoboi 5y ago

There is a bug in GKE causing 502 when using CNI network. The big is triggered by port names in ingresses. Use port numbers and you should be good to go.

TurningCanadian 5y ago

Check out https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream https://kubernetes.github.io/ingress-nginx/user-guide/nginx-... if you're running nginx. Consider setting it to true instead of the default (false) --- By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments .

nhoughto 5y ago
ah good tip, I don't care about session affinity or custom balancing algos so that works. I'd imagine running in GKE or AWS you would also avoid the DNAT / conntrack overhead as pods by default use a routable VPC IP instead of a magic CNI IP. Would have to test that though. Quote from related issue: The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.
- sascha_sl 5y ago
  How does this remove conntrack overhead? It should add more, because the node with the ingress controller now has to hold an extra <Cluster IP Cluster Port, Pod IP, Pod Port> mapping, regardless of what CNI is used (flannel in gateway mode also eliminates this overhead, by the way - you just need to make sure there is nothing like the default EC2 source destination check in place).
nopurpose 5y ago
You just moved problem from ingress controller to kube-proxy (or it's replacement) on the same node. Race condition is still present as before.
sascha_sl 5y ago
This is not a good idea, you're doubling the load on conntrack. Maybe ingress-nginx should fix their config generation instead. Had a cluster with at least one change per second to the ingresses, and nginx would regularly just die (and orphan existing requests after 20 seconds).

nhoughto 5y ago

Is this true of the OOTB GKE nginx ingress? Hard to tell by 'load balancer' do they mean nginx ingress reverse proxy? I can imagine the delay between updating the GCP global load balancer service from GKE would be much higher than nginx-ingress reacting to changes in pod health/endpoints. Either way I guess the takeaway is there is a race there between endpoints being updated and those updates propagating, and seems like that isn't handled as perfectly as people assume, and this likely gets worse with node contention and Kube API performance problems.

cyberpunk 5y ago
By load balancer they mean internal kubernetes “service” object that a given ingress uses as it’s backing service.

mad_vill 5y ago

the issues I see with kubernetes ingress are more related to an ingress pod going down than the upstream.

cyberpunk 5y ago
What controller are you using? I’ve absolutely smashed nginx and the aws elb controllers and never seen them flinch…

blaisio 5y ago

Yes! I think this is a really under-reported issue. It's basically caused by kubernetes doing things without confirming everyone responded to prior status updates. It affects every ingress controller, and it also affects services of type "Load Balancer" and there isn't a real fix. Even if you add a timeout in the pre stop hook, that still might not handle it 100% of the time. IMO it is a design flaw in Kubernetes.

deleted 5y ago
[deleted]
LimaBearz 5y ago
Not defending the situation of a preStop hook at least in the case of API's k8s can handle it 100%, its just messy. We have a preStop hook of 62s. 60s timeouts are set in our apps, 61s is set on the ALBs (ensuring the ALB is never the cause of the hangup), and 62s on the preStop to make sure nothing has come into the container in the last 62s. Then we set a terminationGracePeriodSeconds of 60 just to make sure it doesn't pop off too fast. This gives us 120s where nothing happens and anything in flight can get to where its going.
- thecosmicfrog 5y ago
  Yep, same configuration here other than we use 60/65/70 for (admittedly) completely unscientific reasons.
- chippiewill 5y ago
  > Then we set a terminationGracePeriodSeconds of 60 just to make sure it doesn't pop off too fast. I think the grace period includes the prestop duration doesn't it?
  - bogomipz 5y ago
    The grace period does include the time spent to execute the preStop hook yes.

AaronBBrown 5y ago

This is a design flaw in Kubernetes. The article doesn't really explain what's happening though. The real problem is that there is no synchronization between the ingress controller (which manages the ingress software configuration, e.g. nginx from the Endpoints resources), kube-proxy (which manages iptables rules from the Endpoints resource), and kubelet (which sends the signals to the container). A presStop hook w/ a sleep equivalent to an acceptable timeout will handle the 99%+ cases (and the cases it doesn't will have exceeded your timeout anyhow). Things become more complicated when there are sidecar containers (say an envoy or nginx routing to another container in the same pod) and that often requires shenanigans such as shared emptyDir{} volumes that waits (with fsnotify or similar) for socket files to be closed to ensure requests are fully completed.

kodah 5y ago
I mean, technically, you can recreate this scenario on a single host as well. Send a sigterm to an application and try to swap in another instance of it. System fundamentals are at the heart of that problem: SIGTERM is just what it is, it's a signal and an application can choose to acknowledge it and do something or catch it and ignore it. The system also has no way of knowing what the application chose to do. All that to say, I'm not sure it's as much of a flaw in Kubernetes as much as it's the way systems work and Kubernetes is reflecting that.
- lolc 5y ago
  In my view it is a clear flaw that the signal to terminate can arrive while the server is still getting new requests. Being able to steer traffic based on your knowledge of the state of the system is one of the reasons why you'd want to set up an integrated environment where the load-balancer and servers are controlled from the same process. The time to send the signal is entirely under control of the managing process. It could synchronize with the load-balancer before sending pods the term signal, and I'm unclear why this isn't done.
  - deleted 5y ago
    [deleted]

zeckalpha 5y ago

(2019)

richardfey 5y ago
It is still 100% applicable (AFAIK) and informative, with the (2019) in title readers will think it's not relevant anymore?
- zeckalpha 5y ago
  No, but if they have read it before they may realize it sooner. I have seen (2021) in titles here already.

gscho 5y ago

Wait until you find out that kubernetes secrets aren’t actually secrets but base64 encoded strings.

twalla 5y ago
Encryption at rest for secrets can be enabled, the base64 thing is more of an artifact of how JSON serialization works with byte arrays.
notwedtm 5y ago
I think K8S secrets get a bad wrap. They are not intended to be secret in the sense that they are "kept from prying eyes by default". The secret object is simply a first-class citizen that differentiates it from a ConfigMap in a way that allows distinct ACL's. Most organizations I know will still use something like ExternalSecret for source control and then populate the Secret with the values once in cluster and to an object with very few access points.
- gscho 5y ago
  I think calling it a secret when it isn’t gave it a bad wrap. The last time I looked at the documentation it didn’t even clearly describe that it is not a secure object (that may have changed recently). Why call it a secret when it is not even close to one? I guess thing-to-store-secrets-if-you-use-rbac was too long.
  - threeseed 5y ago
    But it can be a secret. You can store Base64-encoded, encrypted data. And you can encode it for example using an external KMS.
    - gscho 5y ago
      Yes I understand that. My point is until you configure it in that way it is not “secret” and the name of the object is a bit misleading, especially when first learning k8s.

rifelpet 5y ago

There's a KubeCon North America talk (also from 2019) that goes into more detail on this very issue including some additional recommendations https://youtu.be/0o5C12kzEDI?t=57 https://youtu.be/0o5C12kzEDI?t=57

nickjj 5y ago

This is something that plagued me when I started using Kubernetes because out of the box you will get 503s with a load balanced service. If you're on EKS with the AWS Load Balancer Controller, after some research I ended up stumbling on https://github.com/foriequal0/pod-graceful-drain https://github.com/foriequal0/pod-graceful-drain and never looked back. This has been working great so far. You install it once into your cluster and you're done. No dropped connections during a rollout and no need to set up preStop lifecycle hooks. The only downside I've seen so far is it takes slightly longer (about a minute or so) to terminate pods but I'd much rather have that than have to worry about 503s in production during a rollout.

pm90 5y ago
This is a really neat idea, to intercept the pod deletion requests using an admission hooks controller and just delay it for a bit. Thanks for sharing!
sascha_sl 5y ago
"load balanced service" means very little here typically a service of type LoadBalancer is the same as a NodePort, except the CM will not write the status field, expecting some other piece of software to handle it. the stock AWS controller does this so badly, you will reach one random pod per node, and probably not even on the same node, and sometimes the same pod twice, and this connection usually persists until the pod dies so scaling up is near useless. that's not a kubernetes problem kube-proxy and ingress controllers usually rewrite their routing within 1-2 seconds - but I've seen some other designs that take up to 10 minutes to properly set up their load balancers (GKE). once again, not a kubernetes problem

jugg1es 5y ago

Everyone that uses kubernetes must run into this at some point but we all end up doing exactly what is in this article. But we do it quietly because we all think we must be idiots since there is virtually no one else writing anything about it.

merb 5y ago

btw. this can be fixed with readiness probes. and applications should implement them. i.e. application still gets traffic, readiness probe fails because of SIGTERM, application will respond with an error on the probe, the loadbalancer/service sees that and removes the endpoint/service the application can still respond to the inflight requests since it should not stop directly on sigterm, if everything inflight was done it should stop. (this also works with non http, if SIGTERM is handled)

esprehn 5y ago
Yeah this is resolved with lameduck mode: https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ Ex. https://coredns.io/plugins/health/ https://coredns.io/plugins/health/
blaisio 5y ago
No, sorry but I don't think you understood the article. Readiness probes don't fix this. In fact, they could exacerbate the problem (although you obviously need them anyway). The problem is Kubernetes and the Load Balancer react to pod status changes at different times. (In fact, it's worse - different Kubernetes components also react to status changes at different times.) The load balancer is almost always too slow to react, so, it sends traffic to an instance that has already started shutting down.
- merb 5y ago
  Most loadbalancers do use the readiness probes tough
agilob 5y ago
>btw. this can be fixed with readiness probes. and applications should implement them. What rate would you set on readiness probes for high throughput services (100k requests per second) to keep +99% uptime? :)

yonran 5y ago

This peculiar behavior (where the Service health check is unaware of the Deployment’s known instances) mirrors Google Compute Engine (where the httpLoadBalancer’s healthCheck is independent of the instanceGroupManager’s known instances). If you want your program to exit as soon as possible rather than waiting for a SIGKILL from the instance group manager, you have to hard-code the health check timeouts like so: 1. After a SIGTERM, the shutdownHook should keep the HTTP server running. Future /@status requests must return an error, but user requests must still succeed! 2. The shutdownHook sleeps for a minimum of the load balancer health check’s checkIntervalSec * unhealthyThreshold + timeoutSec (which by the way must be less than the instanceGroupManager’s health check’s checkIntervalSec * unhealthyThreshold if it uses the same endpoint) 3. Now the load balancer should not be sending new requests. The shutdownHook then waits for any existing requests to drain. 4. After requests drain, the shutdownHook can finally exit gracefully. It is annoying to have to wait for the health check’s delay (rather than simply draining existing requests as in AWS), but it seems to be necessary for Google-designed load balancers and instance group managers.

paulfurtado 5y ago
With kubernetes you can at least add a preStop hook that sleeps for 60-120sec if the app is not designed to handle SIGTERM. The pod enters the terminating state just prior to executing the preStop hook.

dboreham 5y ago

Was hoping this was an article about a k8s cluster that become sentient then took control of its own power supply, like in that movie...

cphoover 5y ago

Wait... sigterm can be handled to gracefully handle shutdown of your pod... i.e. finish handling all existing requests before closing. I'm not sure the issue here, that sigterm is sent before removal from the load balancer? If you handle the sigterm gracefully doesnt this solve your issue... I'm no k8s expert so maybe I am misunderstanding the problem

notreallyserio 5y ago
> I'm not sure the issue here, that sigterm is sent before removal from the load balancer? That's exactly the issue. It'd be better for the orchestration system to handle removing the service from the LB in response to a configuration change than to wait for e.g. health checks to fail. The SIGTERM method means you have to modify every service to accept SIGTERM (no big deal) for some amount of time (fiddly configuration detail) and in the mean time wait for the LB to see the terminating pod's health check fail so it can stop sending it traffic. If we flipped it around a bit, having the LB remove the traffic before sending a term signal, we could be more assured that poorly behaving pods (pods that die too quickly or too slowly) don't result in user-facing issues.

joshuamorton 5y ago

Isn't this fundamentally k8s treating the pods as cattle? Your service could fail ungracefully at any time for all kinds of reasons, so the client needs to be somewhat prepared (with retries or whatever). If your client can retry then this doesn't matter unless you have no replication? Am I missing something?

yashap 5y ago
Clients can only retry idempotent requests safely. Standard POSTs, for example, cannot be safely retried - you can end up with duplicates.

aequitas720 5y ago

Shameless plug... This is the go package I wrote to handle this and use in all of my servers: https://github.com/abursavich/graceful https://github.com/abursavich/graceful

paxys 5y ago

Their proposed solution isn't really a solution either, and will probably make things worse. Current behavior - Kubernetes sends SIGTERM signal to your container, it starts the shutdown process, stops the web server, and finishes up the requests in flight. Ingress controller continues to send some requests to the pod regardless, and they fail to connect. The client can handle these failures in whatever way it deems fit. Their solution - Kubernetes sends SIGTERM signal, container ignores it and keeps taking new requests. After a while the process is forcefully killed, all requests in flight fail in unpredictable ways, and other resources (like DB connections) may be left dangling.

potamic 5y ago

Would this be mitigated if the ingress load balancer also did health checks with the pods? So when a pod goes into shutdown state, the ingress can detect and stop routing traffic.

Too 5y ago
Health checks typically only run at like 30sec frequency. Even if you set it to 1sec that’s still a big window for requests to slip in before ingress detects the pod is gone.

benlivengood 5y ago

Non-graceful shutdowns must be handled anyway (generally with client-side retries), so this shouldn't be an issue in practice. Kubernetes is intended for distributed/clustered services, not traditional HA services.

rdsubhas 5y ago

This is why we inject `preStop: ["/bin/sleep", "30"]` in every kubernetes PodSpec via an admission controller. Kubernetes service/ingress are designed to be event-driven and eventually consistent. For example, the nginx ingress controller has a direct list of Pod IPs as backends, and will remove them asynchronously based on receiving pod lifecycle events. In real life, it means the pod will be shutdown long before the endpoint is actually removed, leading to error rates. The preStop hook is necessary to avoid request failures, at the expense of longer pod shutdown.

deepsun 5y ago

Is it still the issue with GKE NEGs? NEG is "network endpoint groups" which basically means that Load Balancers deal not with nodes, but with pods directly. So LBs can know what state the pod is. You can even avoid using Ingress (instead, annotate your k8s Service with neg name, and use backend service for that neg in LB), so it'd be possible to combine k8s and non-k8s services under the same Load Balancer, e.g. for migration. But I'm afraid NEG or not-NEG, LB still relies on health checks anyway, so it doesn't matter.