3 ms·
This service (llm proxy to all providers) are a dime-a-dozen This one has very little on monitoring and no reference to OTEL in the docs
by verdverm 1y ago
This service (llm proxy to all providers) are a dime-a-dozen
This one has very little on monitoring and no reference to OTEL in the docs
- vultour 1y agoWhich self-hosted one would you recommend?
- SOLAR_FIELDS 1y agoLiteLLM is one of the most popular solutions. You would self host the gateway
- sothatsit 1y agoWe use LiteLLM and it is a bit of a dumpster fire of enterprise features and bugs. I can't even update the budget on keys in the UI (enterprise feature, although it may be a bug that it is marked as such). I can still update budgets through the API, but the API is a bit of a mess as well. Then we've ran into a lot of bugs like the UI DDOSing itself when the retry mechanism broke and it just started spamming API requests. And then basic features like the cleanup of old logs is an enterprise feature. We are actively looking to switch away from it, so it was nice to stumble on a post like this. Something so simple as a proxy with budgeting for keys should not be such a tangled mess.
- jetbalsa 1y agoI'm currently using apisix its ai rate limits are fine and the webui is a little json heavy but got me going on load balancing a bunch of models across ollama installs
- NeutralCrane 1y agoAre there other alternatives you have been looking at? I’m just getting started looking at these LLM gateways. I was under the impression that LiteLLM was pretty popular but you are not the only one here with negative things to say about it.
- sothatsit 1y agoI am planning to try any-llm-gateway that this post is about. We don't need anything fancy, so it seems that this might cover our needs.
- Manouchehri 1y agoLiteLLM was good in the early days. I ran into more features than bugs. Sadly in the past year or so, I run into more bugs than features.