3 ms·

That looks absolutely horrifying. What are the alternatives ??

by bsaul 20d ago

john01dav 20d ago
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
james-bcn 20d ago
Vercel AI Gateway is one I've used: https://vercel.com/ai-gateway/models https://vercel.com/ai-gateway/models
- maxcoding 20d ago
  Vercel AI Gateway also route to other providers, so the same issues can occur there as well.
bakugo 20d ago
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general. If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you. Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
TZubiri 20d ago
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer. Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
- bbor 20d ago
  > vendor-neutral slop Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time. RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term. I feel like I must be missing something?
  - mrngld 20d ago
    How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in. The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor. As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please. Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
    - TZubiri 20d ago
      >Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming. This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter. They promise to save some hypothetical work in the future if your boss asks to change vendors, and it turns out to be very trivial work that is just a regular part of our programming job, changing a couple of lines in order to change vendor. It's worth noting that there exists a similar set of technologies with a reasonable tradeoff, using a framework that targets different user-platforms makes sense, write-once and deploy at iOS and Android is a reasonable tradeoff, but because you are deploying to those providers simultaneously and it's a user-choice so you don't get to pick one or the other (without losing clients), there's still arguments to chosing just one and losing market share, or doubling the workload and building native for both, but this is a true engineering choice. I feel like stuff like OpenRouter and TerraForm take elements of these frontend abstraction technologies and wastefully apply them to backend tech. A particularly egregious case is when there's an aggregation layer for aggregation layers, say, a tool that generates TerraForm or Chef configs, or a tool that generates Docker and Podman containers, or a tool that generates LiteLLM/OpenRouter configs. Sounds dumb, but it happens when there's a market share for it. Can even get to 3 layers deep. At the foundation might be an aversion to making an irreversible choice, which is an innate emergent psychological phenomenon, but is supported by the Bezos Amazon policy of reversible and irreversible doors. But again, even if you want to be light, using some of these aggregating tools isn't necessary, you can just build on top of a tech, and switch later. The only thing you get with an aggregating layer is that the API ends up being the common denominator so you lose out on the competitive advantages of each choice, or are forced to use even more complex API logic like LLM(commonParam1, commonParam2, vendorParams= {"vendor1"=:{"vendorParam1":"blabla"}} or worse, use hard-coded aggregator provided mappings between the aggregator API and the vendor API that may be incomplete and relies on updates from the aggregator dev. Less is more.
lukasbm 20d ago
If you only care about open source models, cline and opencode provide usage based access and subscriptions for general API access
maeln 20d ago
Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.
danvdb 20d ago
I've used Requesty (https://www.requesty.ai https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.
- nacs 20d ago
  Openrouter lets you pin or blacklist providers or specify provider per-request as well.
vinhnx 20d ago
I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/ https://gateway.merge.dev/
- matt_merge 17d ago
  Glad you’ve been enjoying Gateway! We also have some huge discounts on open models like GLM 5.3 Flash.
  - vinhnx 17d ago
    Hi Matt, awesome you're here. I've been using GLM-5.3 Flash since it came out and really enjoy using with via Merge AI Gateway.
JaceComix 20d ago
Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with. It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.
- dannyw 20d ago
  Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
  - JaceComix 20d ago
    Good info, thanks!
dools 20d ago
I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.

That looks absolutely horrifying. What are the alternatives ??

by bsaul 20d ago

That looks absolutely horrifying. What are the alternatives ??

I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.

james-bcn 20d ago

Vercel AI Gateway is one I've used: https://vercel.com/ai-gateway/models https://vercel.com/ai-gateway/models

maxcoding 20d ago
Vercel AI Gateway also route to other providers, so the same issues can occur there as well.

bakugo 20d ago

There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general. If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you. Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.

TZubiri 20d ago

Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer. Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.

bbor 20d ago
> vendor-neutral slop Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time. RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term. I feel like I must be missing something?
- mrngld 20d ago
  How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in. The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor. As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please. Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
  - TZubiri 20d ago
    >Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming. This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter. They promise to save some hypothetical work in the future if your boss asks to change vendors, and it turns out to be very trivial work that is just a regular part of our programming job, changing a couple of lines in order to change vendor. It's worth noting that there exists a similar set of technologies with a reasonable tradeoff, using a framework that targets different user-platforms makes sense, write-once and deploy at iOS and Android is a reasonable tradeoff, but because you are deploying to those providers simultaneously and it's a user-choice so you don't get to pick one or the other (without losing clients), there's still arguments to chosing just one and losing market share, or doubling the workload and building native for both, but this is a true engineering choice. I feel like stuff like OpenRouter and TerraForm take elements of these frontend abstraction technologies and wastefully apply them to backend tech. A particularly egregious case is when there's an aggregation layer for aggregation layers, say, a tool that generates TerraForm or Chef configs, or a tool that generates Docker and Podman containers, or a tool that generates LiteLLM/OpenRouter configs. Sounds dumb, but it happens when there's a market share for it. Can even get to 3 layers deep. At the foundation might be an aversion to making an irreversible choice, which is an innate emergent psychological phenomenon, but is supported by the Bezos Amazon policy of reversible and irreversible doors. But again, even if you want to be light, using some of these aggregating tools isn't necessary, you can just build on top of a tech, and switch later. The only thing you get with an aggregating layer is that the API ends up being the common denominator so you lose out on the competitive advantages of each choice, or are forced to use even more complex API logic like LLM(commonParam1, commonParam2, vendorParams= {"vendor1"=:{"vendorParam1":"blabla"}} or worse, use hard-coded aggregator provided mappings between the aggregator API and the vendor API that may be incomplete and relies on updates from the aggregator dev. Less is more.

lukasbm 20d ago

If you only care about open source models, cline and opencode provide usage based access and subscriptions for general API access

maeln 20d ago

Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.

danvdb 20d ago

I've used Requesty (https://www.requesty.ai https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.

nacs 20d ago
Openrouter lets you pin or blacklist providers or specify provider per-request as well.

vinhnx 20d ago

I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/ https://gateway.merge.dev/

matt_merge 17d ago
Glad you’ve been enjoying Gateway! We also have some huge discounts on open models like GLM 5.3 Flash.
- vinhnx 17d ago
  Hi Matt, awesome you're here. I've been using GLM-5.3 Flash since it came out and really enjoy using with via Merge AI Gateway.

JaceComix 20d ago

Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with. It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.

dannyw 20d ago
Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
- JaceComix 20d ago
  Good info, thanks!

dools 20d ago

I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.