The Little’s law[1] offers considerable guidance here. In the shortish term it’s safe to model the system as being stationary and then you need to provision for

The Little’s law[1] offers considerable guidance here. In the shortish term it’s safe to model the system as being stationary and then you need to provision for seasonality[2]. Upward or downward trending demand after correcting for seasonality indicates that the system isn’t stationary and your choices are either to scale up or reject requests. Nevertheless the optimal queue size is not necessarily zero except in cases where extremely low latency is necessary. The optimal queue capacity for most web service type applications is one that trends toward almost but not quite full when seasonal requests are higher. [1] https://en.m.wikipedia.org/wiki/Little%27s_law https://en.m.wikipedia.org/wiki/Little%27s_law [2] broadly construed. For example a service that gets a mass of requests at the top of every hour displays seasonality.