See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending t

See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending that job traffic. The concept of lameduck is you report yourself as unhealthy but are still available to serve traffic, you wait before all of your clients that were loaded balanced to you before you stop a sending traffic. https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html https://landing.google.com/sre/book/chapters/load-balancing-... Health checks are part of GCPs load balancer. https://cloud.google.com/compute/docs/load-balancing/health-checks https://cloud.google.com/compute/docs/load-balancing/health-...