7 ms·
Are there situations where GPUs are compute bottlenecked at batch size 1? For the time being at least, that's how local models will be used.
by reitzensteinm 3y ago
Are there situations where GPUs are compute bottlenecked at batch size 1? For the time being at least, that's how local models will be used.
- londons_explore 3y agoEveryone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.
- brucethemoose2 3y agoAt long contexts.