4 ms·
> There's not even a way to compare providers, AFAICT That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think
by lelandbatey 23d ago
> There's not even a way to compare providers, AFAICT
That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For example see the page for Qwen 3.8 27B: https://openrouter.ai/qwen/qwen3.8-27b https://openrouter.ai/qwen/qwen3.8-27b
Some of the killer stats they show per provider:
- Pricing: Effective price accounting for cache hit rate, by provider
- Performance: Throughput in tok/s, latency, E2E latency, tool call error rate, structured output error rate, and more; all per provider.
- Uptime: You have to click on the provider to see their specific uptime, but doing so does show the last-7-days uptime, and you can click to see more.
- Gracana 23d agoThe thing they don't show is the one we really need, especially because model providers can skimp on quality (run lower quantization, lower kv cache precision, etc) to improve their pricing and performance. I agree that it's probably too expensive to keep running the benchmark, but we need some way to hold the providers to a certain standard, otherwise every user has to discover the problems on their own.
- numlocked 23d agoWe are doing continuous benchmarking of each endpoint, for each provider, and it is very expensive :)
- bbor 22d agoNo I know, I was in the GUI as I wrote that lol. As the other person said: if they vary this much in quality, not including that way above updtime and performance is absurd. What would you use a fast, always-up, broken endpoint for?