3 ms·
Nah. Try vLLM and 405B FP8 on that hardware. And make sure you’re benchmarking with some concurrency for max TPS.
by mmaunder 2y ago
Nah. Try vLLM and 405B FP8 on that hardware. And make sure you’re benchmarking with some concurrency for max TPS.
- zackangelo 2y agoRelated recent discussion on twitter: https://x.com/Teknium1/status/1858987850739728635 https://x.com/Teknium1/status/1858987850739728635 Looks like other folks get 80 tok/s with max batch size, that's surprising to me but vLLM is definitely more optimized than my implementation.