4 ms·
> we were the clear leader running on NVIDIA GPUs for both latency and throughput per public data from real-world use on OpenRouter. Baseten: 592.6 tps Groq: 7
by nektro 1y ago
> we were the clear leader running on NVIDIA GPUs for both latency and throughput per public data from real-world use on OpenRouter.
Baseten: 592.6 tps
Groq: 784.6 tps
Cerebras: 4,245 tps
still impressive work
- philipkiely 1y agoYeah the custom hardware providers are super good at TPS. Kudos to their teams for sure, and the demos of instant reasoning are incredibly impressive. That said, we are serving the model at its full 131K context window, and they are serving 33K max, which could matter for some edge case prompts. Additionally, NVIDIA hardware is much more widely available if you are scaling a high-traffic application.