4 ms·
1000+ tokens/sec on H100s, a 5–10x speedup over typical autoregressive models — and without needing exotic hardware like Groq or Cerebras - impressive
by badmonster 1y ago
1000+ tokens/sec on H100s, a 5–10x speedup over typical autoregressive models — and without needing exotic hardware like Groq or Cerebras - impressive
- lostmsu 1y agoWould batch inference increase throughput further? Or does it already peak the FLOPS?