4 ms·
There's still a throughput/latency tradeoff curve, at least for any sort of interactive models. One of the reasons why inference providers sell batch discounts
by calaphos 1y ago
There's still a throughput/latency tradeoff curve, at least for any sort of interactive models.
One of the reasons why inference providers sell batch discounts.