3 ms·
If it's 150 t/s, that's barely faster than Nvidia GPUs who are batching a lot more and are a lot more cost effective. Add in the Groq piece and Nvidia claims it
by aurareturn 4mo ago
If it's 150 t/s, that's barely faster than Nvidia GPUs who are batching a lot more and are a lot more cost effective. Add in the Groq piece and Nvidia claims it can do 400 tokens/s.
- eli 4mo agoI assume it’s just oversubscribed. I’m sure it “can” go faster. But yeah that was my point.