2 ms·
That's not the issue, the issue is GPU utilization. Batching enables higher utilization and higher images per second throughput, but doesn't improve latency.
by modeless 3y ago
That's not the issue, the issue is GPU utilization. Batching enables higher utilization and higher images per second throughput, but doesn't improve latency.
- Kubuxu 3y agoIn essence batching allows for more efficient usage of memory bandwidth. Without batching for every generation you need to transfer the whole model from GPU memory to GPU core once for every image which sets an upper bound on speed. With batching the bottlenecks start showing up elsewhere. For SD1.5 4090 is able to do ~17it/s without batching and ~90-100it/s with batching. Although these numbers might be old at this point, I looked at it ~3mo ago.