The easiest way would be to quantize the model, and serve different quants based on the current demand. Higher volumes == worse quant == more customers served p

The easiest way would be to quantize the model, and serve different quants based on the current demand. Higher volumes == worse quant == more customers served per GPU