4 ms·
Yeah I call BS on this. This does nothing to address the main issues with autoregressive transformer models (memory bandwidth). GPU compute units are mostly si
by qeternity 3y ago
Yeah I call BS on this. This does nothing to address the main issues with autoregressive transformer models (memory bandwidth).
GPU compute units are mostly sitting idle these days waiting for chip cache to receive data fr VRAM.
This does nothing to solve that.
- pavelstoev 3y agoNot exactly idle but only at around 30% utilization on average (measured on a ~900 GPU cluster over ~25 days)
- sp332 3y agoIf it's at 30% utilization then it's "mostly idle".
- pavelstoev 3y agoI agree. I am surprised that many folks, not you of course, think that is okay.
- dramlord 3y agoYou can amortize memory loading with large continuous batching. I imagine more compute would help the problem for certain workloads like speculative decoding
- qeternity 3y agoBatching helps throughput and anyone running in production will be doing batching. But it's not free, and still comes at a cost of per-stream latency. Speculative decoding seems less effective in practice than in theory.