> However, that may very well change with fair queueing. It doesn't matter what kind of queueing system you put in place. No matter how fancy it is, and even

> However, that may very well change with fair queueing. It doesn't matter what kind of queueing system you put in place. No matter how fancy it is, and even if the queue adds negligible overhead, if your average request arrival rate is higher than the average handling rate, your queue will grow unbounded. Then, for some requests, the system will appear to grind to a halt. There's no way around it. A priority queue just means that high-priority requests get processed faster. Low-priority requests might end up never getting handled. Here's a concrete example. I've got a web service that requires a login. I'm using bcrypt to hash passwords with a sufficiently large work factor that I can only hash 5 passwords per second and have a queue for the password hashing. If 6 people are trying to login every second, then after 1 minute, I will have received 360 login requests. 300 of them will have completed, but the queue has grown to 60. Anybody trying to log in at that point will have to wait a minimum of 12 seconds to get their login processed. But what if I also have users trying to sign up? I have to hash their password, so those get added to the queue as well. If I prioritize logins over signups, then under my scenario, a signup will never complete. There's really nothing that can be done. An unbounded queue means unbounded latency. Eventually, a login request would have to get rejected just because the queue is full. Of course, in the real world, this creates frustration for the user. They'll either try to login again, or give up.