3 ms·

You can use something like OpenRouter, which lets you access essentially all commercially available models. Including open-source models. There are no rate limi

You can use something like OpenRouter, which lets you access essentially all commercially available models. Including open-source models. There are no rate limits. You pay a different rate per model (OpenRouter shows the pricing transparently). You load your account with credits. I use it daily (undoubtedly far more than the average user) and loaded 50$ with credits five months ago, but I still have over 1/2 of it left. I think it is hard to believe that Kagi would be any cheaper and have no rate limits.

cstuder 2y ago
Thanks for the hint, a usage based model looks way more attractive to me right now.
- exe34 2y ago
  deepinfra is pretty easy to use too, for llamas and other "open" ones.
stavros 2y ago
Keep in mind that, while OpenRouter gives you the upstream price for OpenAI/Anthropic models (so you pay the same per token), there's a loading charge, so if you want to load $10 in credits you pay $12 or so. This means that it's more expensive than calling OpenAI directly, even though they have the same price per token.
- bangaladore 2y ago
  Where the loading charge is amortized over all the calls made. If you want to use precisely one API, paying directly for that API is cheaper. However, that's only true with closed-source providers. Anyone can host a server running llama 3.1 that OpenRouter could (in theory) use, bringing price competition to model cost. Closed-source models have a monopoly and can set their price wherever they want. I'm okay with spending an extra 2$ every six months to access the APIs of any model I want.
  - stavros 2y ago
    Sure, but I only use the hosted APIs, so for me it doesn't make much sense to pay the extra premium. Maybe it doesn't for others either.
hn_user2 2y ago
Chatblade cli is also worth checking out. No loading fees and you can pipe code results to files.
SirYandi 2y ago
Just plug your API keys into a front end like https://github.com/enricoros/big-AGI https://github.com/enricoros/big-AGI and pay as you go for all commercially available models
- freedomben 2y ago
  Have you tried open-webui?[1] I've been using that and really loving it, but wondering if I should try out big-AGI [1]: https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
  - xur17 2y ago
    Not to add more to the mix, but I've been very happy with Librechat: https://github.com/danny-avila/LibreChat https://github.com/danny-avila/LibreChat
    - vunderba 2y ago
      LibreChat is also one of the few LLM chat interfaces that works with both external APIs (OpenAI, Anthropic, etc) and local ones (hosted via Ollama, etc) out of the box. If it has a downside, I'd say its a little more involved to get setup, lots of docker containers, etc then a more batteries-included approach like Jan. https://github.com/janhq/jan https://github.com/janhq/jan
sergiotapia 2y ago
There's also BigAgi (really a weird ass name - probably hurting them) that is good for the same use case. Just paste in your API key and you get a really nice UI to chat through at-cost. https://get.big-agi.com/ https://get.big-agi.com/
- bangaladore 2y ago
  A nice thing about this as I'm reading it is you can hook up OpenRouter to it. OpenRouter's interface leaves a lot to be desired.
Havoc 2y ago
OpenRouter absolutely does have rate limits: https://openrouter.ai/docs/limits https://openrouter.ai/docs/limits ...haven't had issues with them, but they are there
- bangaladore 2y ago
  Good catch. It's not relevant if used as an "assistant," though. And if you cared and loaded your credits, you can do 200 req/s (maybe also a surge limit? docs are unclear)

cstuder 2y ago

Thanks for the hint, a usage based model looks way more attractive to me right now.

exe34 2y ago
deepinfra is pretty easy to use too, for llamas and other "open" ones.

stavros 2y ago

Keep in mind that, while OpenRouter gives you the upstream price for OpenAI/Anthropic models (so you pay the same per token), there's a loading charge, so if you want to load $10 in credits you pay $12 or so. This means that it's more expensive than calling OpenAI directly, even though they have the same price per token.

bangaladore 2y ago
Where the loading charge is amortized over all the calls made. If you want to use precisely one API, paying directly for that API is cheaper. However, that's only true with closed-source providers. Anyone can host a server running llama 3.1 that OpenRouter could (in theory) use, bringing price competition to model cost. Closed-source models have a monopoly and can set their price wherever they want. I'm okay with spending an extra 2$ every six months to access the APIs of any model I want.
- stavros 2y ago
  Sure, but I only use the hosted APIs, so for me it doesn't make much sense to pay the extra premium. Maybe it doesn't for others either.

hn_user2 2y ago

Chatblade cli is also worth checking out. No loading fees and you can pipe code results to files.

SirYandi 2y ago

Just plug your API keys into a front end like https://github.com/enricoros/big-AGI https://github.com/enricoros/big-AGI and pay as you go for all commercially available models

freedomben 2y ago
Have you tried open-webui?[1] I've been using that and really loving it, but wondering if I should try out big-AGI [1]: https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
- xur17 2y ago
  Not to add more to the mix, but I've been very happy with Librechat: https://github.com/danny-avila/LibreChat https://github.com/danny-avila/LibreChat
  - vunderba 2y ago
    LibreChat is also one of the few LLM chat interfaces that works with both external APIs (OpenAI, Anthropic, etc) and local ones (hosted via Ollama, etc) out of the box. If it has a downside, I'd say its a little more involved to get setup, lots of docker containers, etc then a more batteries-included approach like Jan. https://github.com/janhq/jan https://github.com/janhq/jan

sergiotapia 2y ago

There's also BigAgi (really a weird ass name - probably hurting them) that is good for the same use case. Just paste in your API key and you get a really nice UI to chat through at-cost. https://get.big-agi.com/ https://get.big-agi.com/

bangaladore 2y ago
A nice thing about this as I'm reading it is you can hook up OpenRouter to it. OpenRouter's interface leaves a lot to be desired.

Havoc 2y ago

OpenRouter absolutely does have rate limits: https://openrouter.ai/docs/limits https://openrouter.ai/docs/limits ...haven't had issues with them, but they are there

bangaladore 2y ago
Good catch. It's not relevant if used as an "assistant," though. And if you cared and loaded your credits, you can do 200 req/s (maybe also a surge limit? docs are unclear)