> I kinda wonder if there needs to be some kind of "retry cost" for requests, so high level things retry less when previous attempts consumed a lot of resources

> I kinda wonder if there needs to be some kind of "retry cost" for requests, so high level things retry less when previous attempts consumed a lot of resources further down... but I'm not sure how that could be tracked without either reliable communication of failures (ha) or a global storage of some kind (a potentially massive point of failure) You could pretty easily get rough awareness of “things down the stack being expensive” without any sort of reliable communication or centralized storage if you tracked it per request. If you’re already passing something like a request id through to support tracing across your services, more or less the same idea except pass some sort of budget. All your entrypoints assign a budget based on some reasonable expectation of cost for the request plus whatever margin you want for retries and failures. Every caller, when making a request, passes the current budget value through to the callee. If the callee returns, it can include a header with the cost so the caller can deduct it from the budget. If you wanted to use the latency idea, you could replace the cost with a scaling factor to multiply by the latency so any service that was seeing latency spikes would automatically get more expensive. If it doesn’t return, the caller has some relatively high fixed cost it deducts instead. (Optimally, a value that’s related to the general magnitude of cost for the request normally.) If any service tries to initiate a request and the budget is <= 0, it instead throws an exception. It returns the costs and now the caller has reached zero and if it retries it will throw an exception back to its caller and so on and so forth until it gets back to the original consumer. You’ve basically set an upper limit on how much effort you’ll expend servicing any request… whatever the reason it’s taking more work or longer than expected. Main issue I’d see though is… that’s a lot of numbers to tune and keep in alignment. While it would probably be good to know that some call you’re making to some other team’s service just got 10x more expensive because they integrated some other thing… probably not by your API suddenly blowing up. Or the reverse, some team optimizes some stuff and reduces the cost of some requests and now you actually have enough budget to retry too many times and you’re almost back where you started. I generally just lean towards retries being something that happens in one place, at one layer. Or even not at all—the user can see a “Something went wrong. Try again?” and we can retry when they ask us to.