4 ms·
btw. this can be fixed with readiness probes. and applications should implement them. i.e. application still gets traffic, readiness probe fails because of SIG
by merb 5y ago
btw. this can be fixed with readiness probes. and applications should implement them.
i.e. application still gets traffic, readiness probe fails because of SIGTERM, application will respond with an error on the probe, the loadbalancer/service sees that and removes the endpoint/service the application can still respond to the inflight requests since it should not stop directly on sigterm, if everything inflight was done it should stop.
(this also works with non http, if SIGTERM is handled)
- esprehn 5y agoYeah this is resolved with lameduck mode: https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ Ex. https://coredns.io/plugins/health/ https://coredns.io/plugins/health/
- blaisio 5y agoNo, sorry but I don't think you understood the article. Readiness probes don't fix this. In fact, they could exacerbate the problem (although you obviously need them anyway). The problem is Kubernetes and the Load Balancer react to pod status changes at different times. (In fact, it's worse - different Kubernetes components also react to status changes at different times.) The load balancer is almost always too slow to react, so, it sends traffic to an instance that has already started shutting down.
- merb 5y agoMost loadbalancers do use the readiness probes tough
- agilob 5y ago>btw. this can be fixed with readiness probes. and applications should implement them. What rate would you set on readiness probes for high throughput services (100k requests per second) to keep +99% uptime? :)