Someone linked the Google SRE book. It explains that client services watch the global distribution (for that client) of number or retries. It will normally retr

Someone linked the Google SRE book. It explains that client services watch the global distribution (for that client) of number or retries. It will normally retry immediately on another node, because as you say, some failures are expected. But if it notices that more than about 1% of requests are having to retry twice, that indicates the server service needs reduced load and it starts refusing to retry. If it gets really bad it even starts preemptively failing first attempts so that the service doesn't get contacted at all.