Here’s a simple one from my career. A calls B calls C. C makes a bad deploy. B passes through the errors from C and A retries against them. The retries push B’

Here’s a simple one from my career. A calls B calls C. C makes a bad deploy. B passes through the errors from C and A retries against them. The retries push B’s utilization past the limit and instances start getting OOM killed. The C deploy is reverted. But the retries from A to B aren’t about C anymore, they’re about the fact that B is overloaded. A and B are in a neat little death spiral. We have to stop A, let B recover, and then slowly turn A back on. After this incident we installed a per-instance rate limit on all services.