If you have multiple workers each managing their own throttling, there is a particular failure mode you may run into: everything is operating normally, then req

If you have multiple workers each managing their own throttling, there is a particular failure mode you may run into: everything is operating normally, then requests increase to the point that one of your servers starts throttling requests. Your load balancer redistributes traffic to the remaining machines, which see even greater load, and each in turn start throttling requests. At some point you have no working servers, and all requests are failing. You start adding servers, but as soon as a server enters the load balancer pool it is bombarded with all requests and shuts off. At this point you have more than enough servers to handle requests, but all of them are disabled. Most solutions to the thundering herd problem involve rate limiting at the load balancer, or a finer-grained heuristic to ensure that each server does as much work as it can, even under load.