The problem is the request arrival rate vs the distribution of service times in your app. New Relic may be giving you an average number you feel happy about, b

The problem is the request arrival rate vs the distribution of service times in your app. New Relic may be giving you an average number you feel happy about, but the 99th percentile numbers are extremely important. If you have a small fraction of requests that take much longer to process, you'll end up with queuing, even with a predictive least loaded balancing policy. This is a very common performance problem in rails apps, because developers often use active record's associations without any sort of limit on row count, not considering that in the future individual users might have 10000 posts/friends/whatever associated object. Fix this and you'll see your end user latency come back in line.