The problem is queueing at the worker end and not the proxy side. Using a fair queueing system (such as Nginx HttpUpstreamFairModule) can limit the number of c

The problem is queueing at the worker end and not the proxy side. Using a fair queueing system (such as Nginx HttpUpstreamFairModule) can limit the number of concurrent requests sent to workers so they don't get overloaded. It will make requests queue up at the proxy and then get distributed evenly. See: recent discussion on problems caused by Heroku's random order load balancing.