3 ms·

> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even

> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this. > I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly. This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them. I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.

bbor 23d ago
if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models. This is quite literally the point of OpenRouter. Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.
- embedding-shape 23d ago
  > I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it. Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices. For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
  - bbor 23d ago
    If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter Despite your confidence, that is indeed the basis of this massive corporations entire business plan. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRoute If OpenRouter is only for bad accuracy, they should say as much and fade into deserved obscurity.
    - embedding-shape 23d ago
      [flagged]
Spacemolte 23d ago
1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price. You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.
- embedding-shape 23d ago
  > 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable. > 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models. > 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price. Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter. > You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
  - ShalevYoni 23d ago
    [flagged]
kelvinjps10 23d ago
I mean that should be their job to verify the quality of the providers they have in their platform. Then what is the 5% for?
dofm 23d ago
> I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you This helps forecast one possible future for OpenRouter: they begin to offer in-house provision, and people begin to migrate their uses off the "marketplace" providers and onto the "fulfilled-by-us" provision.

bbor 23d ago

if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models. This is quite literally the point of OpenRouter. Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.

embedding-shape 23d ago
> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it. Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices. For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
- bbor 23d ago
  If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter Despite your confidence, that is indeed the basis of this massive corporations entire business plan. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRoute If OpenRouter is only for bad accuracy, they should say as much and fade into deserved obscurity.
  - embedding-shape 23d ago
    [flagged]

Spacemolte 23d ago

1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price. You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.

embedding-shape 23d ago
> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable. > 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models. > 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price. Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter. > You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
- ShalevYoni 23d ago
  [flagged]

kelvinjps10 23d ago

I mean that should be their job to verify the quality of the providers they have in their platform. Then what is the 5% for?

dofm 23d ago

> I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you This helps forecast one possible future for OpenRouter: they begin to offer in-house provision, and people begin to migrate their uses off the "marketplace" providers and onto the "fulfilled-by-us" provision.