6 ms·

Show HN: Any-LLM – Lightweight router to access any LLM Provider

125 points by AMeckes 1y ago

We built any-llm because we needed a lightweight router for LLM providers with minimal overhead. Switching between models is just a string change : update "openai/gpt-4" to "anthropic/claude-3" and you're done. It uses official provider SDKs when available, which helps since providers handle their own compatibility updates. No proxy or gateway service needed either, so getting started is pretty straightforward - just pip install and import. Currently supports 20+ providers including OpenAI, Anthropic, Google, Mistral, and AWS Bedrock. Would love to hear what you think!

sparacha 1y ago
There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve
- CuriouslyC 1y ago
  LiteLLM is kind of a mess TBH, I guess it's ok if you just want a docker container to proxy to for personal projects, but actually using it in production isn't great.
  - dlojudice 1y ago
    > but actually using it in production isn't great. I only use it in development. Could you elaborate on why you don't recommend using it in production?
  - honorable_coder 1y ago
    the people behind envoy proxy built: https://github.com/katanemo/archgw https://github.com/katanemo/archgw - has the learnings of Envoy but natively designed to process/route prompts to agents and LLMs. Would be curious about your thoughts
  - tom_usher 1y ago
    I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic (https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L3160 https://github.com/BerriAI/litellm/blob/main/litellm/utils.p...)
    - swyx 1y ago
      can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
      - otabdeveloper4 1y ago
        LiteLLM is the worst code I have ever read in my life. Quite an accomplishment, lol.
        swyx 1y ago
        ok still not helpful in giving substantial criticism
        honorable_coder 1y ago
        and you say you aren't "vested" in liteLLM?
        swyx 1y ago
        yes, green text hn account, i am not. i just want help in properly identifying flaws in litellm. clearly nobody here is offering actual analysis.
        otabdeveloper4 1y ago
        Sorry if this sounds harsh, but I'm not really interested in spending time to code review the worst code I've ever seen in 30 years of programming. Is LiteLLM's code written by an LLM?
- ieuanking 1y ago
  we are trying to apply model-routing to academic work and pdf chat with ubik.studio -- def lmk what you think
- swyx 1y ago
  portkey as well which is both js and open source https://www.latent.space/p/gateway https://www.latent.space/p/gateway
  - pzo 1y ago
    why provide link if there is not a single portkey keyword there?
    - swyx 1y ago
      its my interview w portkey folks which has more thoughts on the category
- wongarsu 1y ago
  And all of them despite 80% of model providers offering an OpenAI compatible endpoint
  - troyvit 1y ago
    I think Mozilla of all people would understand why standardizing on one private organization's way of doing things might not be best for the overall ecosystem. Building a tool that meets LLM providers where they are instead of relying on them to homogenize on OpenAI's choices seems like a great reason for this project.
dlojudice 1y ago
I use Litellm Proxy, even in a dev environment via Docker, because the Usage and Logs feature greatly helps in providing visibility into LLM usage. The Caching functionality greatly helps in reducing costs for repetitive testing.
weinzierl 1y ago
Not to be confused with AnythingLLM.
honorable_coder 1y ago
a proxy means you offload observability, filtering, caching rules, global rate limiters to a specialized piece of software - pushing this in application code means you _cannot_ do things centrally and it doesn't scale as more copies of your application code get deployed. You can bounce a single proxy server neatly vs. updating a fleet of your application server just to monkey patch some proxy functionality.
- RussianCow 1y ago
  You can do all of that without a proxy. Just store the current state in your database or a Redis instance.
  - honorable_coder 1y ago
    and managed from among the application servers that are greedily trying to store/retrieve this state? Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.
    - RussianCow 1y ago
      > Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. I have to do this already with practically all software I write, so the comexity is already baked in. Sure, if you don't already have a database or cache, maybe a proxy is simpler, but otherwise it's just extra infrastructure you need to manage. > The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale. I've been doing stuff like the above (not for LLMs but similar use cases) for years "at scale" without issues. But in any case, you need to store state the moment you scale beyond a single proxy server anyway. Plus, most products never achieve a scale where this discussion matters.
- 1y ago
swyx 1y ago
> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.
- chuckhend 1y ago
  LiteLLM is quite battle tested at this point as well. > it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
  - amanda99 1y ago
    Being battle tested is the only good thing I can say about LiteLLM.
    - scosman 1y ago
      You can add in it's still 10x better than LangChain
  - AMeckes 1y ago
    That's true. We traded API compatibility work for SDK compatibility work. Our bet is that providers are better at maintaining their own SDKs than we are at reimplementing their APIs. SDKs break less often and more predictably than APIs, plus we get provider-implemented features (retries, auth refresh, etc) "for free." Not zero maintenance, but definitely less. We use this in production at Mozilla.ai, so it'll stay actively maintained.
- 1y ago
renewiltord 1y ago
In truth it wasn’t that hard for me to ask Claude Code to just implement the text completion API so routing wasn’t that much of a problem.
piker 1y ago
This looks awesome. Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.
- pzo 1y ago
  for js/ts you have vercel aisdk [0], for c++ you have [1], for flutter/reactnative/kotlin there is [2] [0] https://github.com/vercel/ai https://github.com/vercel/ai [1] https://github.com/ClickHouse/ai-sdk-cpp https://github.com/ClickHouse/ai-sdk-cpp [2] https://github.com/cactus-compute/cactus https://github.com/cactus-compute/cactus
- retrovrv 1y ago
  we essentially built the gateway as a service rather than an SDK: https://github.com/portkey-AI/gateway https://github.com/portkey-AI/gateway
- Shark1n4Suit 1y ago
  That's the key question. It feels like many of these tools are trying to solve a systems-level problem (cross-language model execution) at the application layer (with a Python library). A truly universal solution would likely need to exist at a lower level of abstraction, completely decoupling the application's language from the model's runtime. It's a much harder problem to solve there, but it would be a huge step forward.
mkw5053 1y ago
Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this. [1] https://github.com/Airbolt-AI/airbolt https://github.com/Airbolt-AI/airbolt
- swyx 1y ago
  (retracted after GP edited their comment)
  - qntmfred 1y ago
    don't you post links to your own stuff all the time? i don't think their comment was out of line.
  - mkw5053 1y ago
    I didn’t intend my original comment to be overly-promotional without relevance. I'm genuinely curious about the tradeoffs between different LLM API routing solutions, most acutely as a consumer.
amanda99 1y ago
I'm excited to see this. Have been using LiteLLM but it's honestly a huge mess once you peek under the hood, and it's being developed very iteratively and not very carefully. For example. for several months recently (haven't checked in ~a month though), their Ollama structured outputs were completely botched and just straight up broken. Docs are a hot mess, etc.
nexarithm 1y ago
I have been also working on very similar open source project for python llm abstraction layer. I needed one for my research job. I inspired from that and created one for more generic usage. Github: https://github.com/proxai/proxai https://github.com/proxai/proxai Website: https://proxai.co/ https://proxai.co/
nodesocket 1y ago
This is awesome, will give it a try tonight. I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?
t_minus_100 1y ago
https://xkcd.com/927/ https://xkcd.com/927/ . LiteLLM rocks !
- deleted 1y ago
  [deleted]
- AMeckes 1y ago
  I didn't even need to click the link to know what this comic was. LiteLLM is great, we just needed something slightly different for our use case.
klntsky 1y ago
Anything like this, but in TypeScript?
- AMeckes 1y ago
  Python only for now. Most providers have official TypeScript SDKs though, so the same approach (wrapping official SDKs) would work well in TS too.
- funerr 1y ago
  ai-sdk by vercel?
- retrovrv 1y ago
  there's portkey that we've been working on: https://github.com/portkey-AI/gateway https://github.com/portkey-AI/gateway
pglevy 1y ago
How does this differ from this project? https://github.com/simonw/llm https://github.com/simonw/llm
- peskypotato 1y ago
  From my understanding of Simon's project it only supports OpenAI and OpenAI-compatible models in addition to local model support. For example, if I wanted to use a model on Amazon Bedrock I'd have to first deploy (and manage) a gateway/proxy layer[1] to make it OpenAI-compatible. Mozzila's project boosts of a lot of existing interfaces already, much like LiteLLM, which has the benefit of directly being able to use a wider range or supported models. > No Proxy or Gateway server required so you don't need to deal with setting up any other service to talk to whichever LLM provider you need. Now how it compares to LiteLLM, I don't have enough experience in either to tell. [1] https://github.com/aws-samples/bedrock-access-gateway https://github.com/aws-samples/bedrock-access-gateway
  - delijati 1y ago
    Not true is use it with gemini https://llm.datasette.io/en/stable/plugins/directory.html#remote-apis https://llm.datasette.io/en/stable/plugins/directory.html#re...
    - peskypotato 1y ago
      Plugins! I completely missed that when testing this earlier. Thank you, will have to take another look at it.
omneity 1y ago
Crazy timing! I shipped a similar abstraction for llms a bit over a week ago: https://github.com/omarkamali/borgllm https://github.com/omarkamali/borgllm pip install borgllm I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.
deleted 1y ago
[deleted]
bdhcuidbebe 1y ago
What is mozilla-ai? Seems like reputation parasitism.
- daveguy 1y ago
  It is an official Mozilla Foundation subsidiary. Their website is here: https://www.mozilla.ai/ https://www.mozilla.ai/
  - bdhcuidbebe 1y ago
    Interesting. I made my comment after visiting their repo and website. Didnt see a pixel worth of the mozilla brand there, hence my comment. On a second visit I notice a link to mozilla.org on their footer. Still doesent ring official by me from being a veteran mozilla user (netscape, mdn, firefox) but ok, thanks for the explanation.
    - daveguy 1y ago
      I agree it's not very clear. They would do well to mention it somewhere besides the main site footer because it would probably help adoption / community / testing too. That said, any company with a lawyer wouldn't let that stand as a name-squat for long.
    - JohnPDickerson 1y ago
      Good feedback. Some of this is intentional - as an independent and growing ~20-person company, we're able to operate more quickly than the larger Mozilla organizations, and we're purposefully distancing ourselves from the associated bureaucracy that comes with any large organization. We are very much in line with the Mozilla ethos around personal ownership, privacy, control, and agency. We're figuring out how to best push on those principles in the world of AI, and appreciate feedback and contributions from the community.
- JohnPDickerson 1y ago
  Common question, thanks for asking! We’re a public benefit corporation focused on democratizing access to AI tech, on enabling non-AI experts to benefit from and control their own AI tools, and on empowering the open source AI ecosystem. Our majority shareholder is the Mozilla Foundation - the other shareholders being our employees, soon :). As access to knowledge and people shifts due to AI, we’re working to make sure people retain choice, ownership, privacy, and dignity. We're very small compared to the Mozilla mothership, but moving quickly to support open source AI in any way we can.
spooky_deep 1y ago
Really needs a Docker image (maybe just not mentioned?) so one doesn’t have to wrestle pip and python versions.
gapeleon 1y ago
You guys need to fact check your AI-generated blog posts: https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to-access-any-llm-provider/ https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to... > One popular solution, LiteLLM, is highly valued for its wide support of different providers and modalities, making it a great choice for many developers. However, it re-implements provider interfaces rather than leveraging SDKs that are managed and released by the providers themselves. As a result, the approach can lead to compatibility issues and unexpected modifications in behavior, making it difficult to keep up with the changes happening among all the providers. LiteLLM is rock-solid in practice. The underlying API providers announce breaking changes well in advance, and LiteLLM has never been caught out by this. LLMs will come up with hypothetical cons like this upon request. > Lastly, proxy/gateway solutions like OpenRouter and Portkey require users to set up a hosted proxy server to act as an intermediary between their code and the LLM provider. Although this can effectively abstract away the complicated logic from the developer, it adds an extra layer of complexity and a dependency on external services, which might not be ideal for all use cases. OpenRouter is a hosted service that provides the proxy/gateway infrastructure. Users don't "set up a hosted proxy server" themselves; they just make API calls to OpenRouter's endpoints. But older LLMs don't know what OpenRouter is and will assume it's a self-hosted proxy server. > Another option, AISuite, was created by Andrew NG and offers a clean and modular design. However, it is not actively maintained (its last release was in December of 2024) and lacks consistent Python-typed interfaces. Okay so you clicked the "releases" tab and saw December 2024. Next time check https://github.com/andrewyng/aisuite/commits/main/ https://github.com/andrewyng/aisuite/commits/main/ Small, fast moving community projects like this, exllamav2, etc don't necessarily tag releases. I've got nothing against using AI to write posts like this, but at least take the time to fact check before dumping on other people's work. If not for the Mozilla branding, I'd have assumed this was a scam/malware - especially since it's name is so similar to Anything-LLM.

125 points by AMeckes 1y ago

sparacha 1y ago

There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve

CuriouslyC 1y ago
LiteLLM is kind of a mess TBH, I guess it's ok if you just want a docker container to proxy to for personal projects, but actually using it in production isn't great.
- dlojudice 1y ago
  > but actually using it in production isn't great. I only use it in development. Could you elaborate on why you don't recommend using it in production?
- honorable_coder 1y ago
  the people behind envoy proxy built: https://github.com/katanemo/archgw https://github.com/katanemo/archgw - has the learnings of Envoy but natively designed to process/route prompts to agents and LLMs. Would be curious about your thoughts
- tom_usher 1y ago
  I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic (https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L3160 https://github.com/BerriAI/litellm/blob/main/litellm/utils.p...)
  - swyx 1y ago
    can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
    - otabdeveloper4 1y ago
      LiteLLM is the worst code I have ever read in my life. Quite an accomplishment, lol.
      - swyx 1y ago
        ok still not helpful in giving substantial criticism
        honorable_coder 1y ago
        and you say you aren't "vested" in liteLLM?
        swyx 1y ago
        yes, green text hn account, i am not. i just want help in properly identifying flaws in litellm. clearly nobody here is offering actual analysis.
        otabdeveloper4 1y ago
        Sorry if this sounds harsh, but I'm not really interested in spending time to code review the worst code I've ever seen in 30 years of programming. Is LiteLLM's code written by an LLM?
ieuanking 1y ago
we are trying to apply model-routing to academic work and pdf chat with ubik.studio -- def lmk what you think
swyx 1y ago
portkey as well which is both js and open source https://www.latent.space/p/gateway https://www.latent.space/p/gateway
- pzo 1y ago
  why provide link if there is not a single portkey keyword there?
  - swyx 1y ago
    its my interview w portkey folks which has more thoughts on the category
wongarsu 1y ago
And all of them despite 80% of model providers offering an OpenAI compatible endpoint
- troyvit 1y ago
  I think Mozilla of all people would understand why standardizing on one private organization's way of doing things might not be best for the overall ecosystem. Building a tool that meets LLM providers where they are instead of relying on them to homogenize on OpenAI's choices seems like a great reason for this project.

dlojudice 1y ago

I use Litellm Proxy, even in a dev environment via Docker, because the Usage and Logs feature greatly helps in providing visibility into LLM usage. The Caching functionality greatly helps in reducing costs for repetitive testing.

weinzierl 1y ago

Not to be confused with AnythingLLM.

honorable_coder 1y ago

a proxy means you offload observability, filtering, caching rules, global rate limiters to a specialized piece of software - pushing this in application code means you _cannot_ do things centrally and it doesn't scale as more copies of your application code get deployed. You can bounce a single proxy server neatly vs. updating a fleet of your application server just to monkey patch some proxy functionality.

RussianCow 1y ago
You can do all of that without a proxy. Just store the current state in your database or a Redis instance.
- honorable_coder 1y ago
  and managed from among the application servers that are greedily trying to store/retrieve this state? Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.
  - RussianCow 1y ago
    > Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. I have to do this already with practically all software I write, so the comexity is already baked in. Sure, if you don't already have a database or cache, maybe a proxy is simpler, but otherwise it's just extra infrastructure you need to manage. > The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale. I've been doing stuff like the above (not for LLMs but similar use cases) for years "at scale" without issues. But in any case, you need to store state the moment you scale beyond a single proxy server anyway. Plus, most products never achieve a scale where this discussion matters.
1y ago

swyx 1y ago

> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues) you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.

chuckhend 1y ago
LiteLLM is quite battle tested at this point as well. > it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
- amanda99 1y ago
  Being battle tested is the only good thing I can say about LiteLLM.
  - scosman 1y ago
    You can add in it's still 10x better than LangChain
- AMeckes 1y ago
  That's true. We traded API compatibility work for SDK compatibility work. Our bet is that providers are better at maintaining their own SDKs than we are at reimplementing their APIs. SDKs break less often and more predictably than APIs, plus we get provider-implemented features (retries, auth refresh, etc) "for free." Not zero maintenance, but definitely less. We use this in production at Mozilla.ai, so it'll stay actively maintained.
1y ago

renewiltord 1y ago

In truth it wasn’t that hard for me to ask Claude Code to just implement the text completion API so routing wasn’t that much of a problem.

piker 1y ago

This looks awesome. Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.

pzo 1y ago
for js/ts you have vercel aisdk [0], for c++ you have [1], for flutter/reactnative/kotlin there is [2] [0] https://github.com/vercel/ai https://github.com/vercel/ai [1] https://github.com/ClickHouse/ai-sdk-cpp https://github.com/ClickHouse/ai-sdk-cpp [2] https://github.com/cactus-compute/cactus https://github.com/cactus-compute/cactus
retrovrv 1y ago
we essentially built the gateway as a service rather than an SDK: https://github.com/portkey-AI/gateway https://github.com/portkey-AI/gateway
Shark1n4Suit 1y ago
That's the key question. It feels like many of these tools are trying to solve a systems-level problem (cross-language model execution) at the application layer (with a Python library). A truly universal solution would likely need to exist at a lower level of abstraction, completely decoupling the application's language from the model's runtime. It's a much harder problem to solve there, but it would be a huge step forward.

mkw5053 1y ago

Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this. [1] https://github.com/Airbolt-AI/airbolt https://github.com/Airbolt-AI/airbolt

swyx 1y ago
(retracted after GP edited their comment)
- qntmfred 1y ago
  don't you post links to your own stuff all the time? i don't think their comment was out of line.
- mkw5053 1y ago
  I didn’t intend my original comment to be overly-promotional without relevance. I'm genuinely curious about the tradeoffs between different LLM API routing solutions, most acutely as a consumer.

amanda99 1y ago

I'm excited to see this. Have been using LiteLLM but it's honestly a huge mess once you peek under the hood, and it's being developed very iteratively and not very carefully. For example. for several months recently (haven't checked in ~a month though), their Ollama structured outputs were completely botched and just straight up broken. Docs are a hot mess, etc.

nexarithm 1y ago

I have been also working on very similar open source project for python llm abstraction layer. I needed one for my research job. I inspired from that and created one for more generic usage. Github: https://github.com/proxai/proxai https://github.com/proxai/proxai Website: https://proxai.co/ https://proxai.co/

nodesocket 1y ago

This is awesome, will give it a try tonight. I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?

t_minus_100 1y ago

https://xkcd.com/927/ https://xkcd.com/927/ . LiteLLM rocks !

deleted 1y ago
[deleted]
AMeckes 1y ago
I didn't even need to click the link to know what this comic was. LiteLLM is great, we just needed something slightly different for our use case.

klntsky 1y ago

Anything like this, but in TypeScript?

AMeckes 1y ago
Python only for now. Most providers have official TypeScript SDKs though, so the same approach (wrapping official SDKs) would work well in TS too.
funerr 1y ago
ai-sdk by vercel?
retrovrv 1y ago
there's portkey that we've been working on: https://github.com/portkey-AI/gateway https://github.com/portkey-AI/gateway

pglevy 1y ago

How does this differ from this project? https://github.com/simonw/llm https://github.com/simonw/llm

peskypotato 1y ago
From my understanding of Simon's project it only supports OpenAI and OpenAI-compatible models in addition to local model support. For example, if I wanted to use a model on Amazon Bedrock I'd have to first deploy (and manage) a gateway/proxy layer[1] to make it OpenAI-compatible. Mozzila's project boosts of a lot of existing interfaces already, much like LiteLLM, which has the benefit of directly being able to use a wider range or supported models. > No Proxy or Gateway server required so you don't need to deal with setting up any other service to talk to whichever LLM provider you need. Now how it compares to LiteLLM, I don't have enough experience in either to tell. [1] https://github.com/aws-samples/bedrock-access-gateway https://github.com/aws-samples/bedrock-access-gateway
- delijati 1y ago
  Not true is use it with gemini https://llm.datasette.io/en/stable/plugins/directory.html#remote-apis https://llm.datasette.io/en/stable/plugins/directory.html#re...
  - peskypotato 1y ago
    Plugins! I completely missed that when testing this earlier. Thank you, will have to take another look at it.

omneity 1y ago

Crazy timing! I shipped a similar abstraction for llms a bit over a week ago: https://github.com/omarkamali/borgllm https://github.com/omarkamali/borgllm pip install borgllm I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.

deleted 1y ago

[deleted]

bdhcuidbebe 1y ago

What is mozilla-ai? Seems like reputation parasitism.

daveguy 1y ago
It is an official Mozilla Foundation subsidiary. Their website is here: https://www.mozilla.ai/ https://www.mozilla.ai/
- bdhcuidbebe 1y ago
  Interesting. I made my comment after visiting their repo and website. Didnt see a pixel worth of the mozilla brand there, hence my comment. On a second visit I notice a link to mozilla.org on their footer. Still doesent ring official by me from being a veteran mozilla user (netscape, mdn, firefox) but ok, thanks for the explanation.
  - daveguy 1y ago
    I agree it's not very clear. They would do well to mention it somewhere besides the main site footer because it would probably help adoption / community / testing too. That said, any company with a lawyer wouldn't let that stand as a name-squat for long.
  - JohnPDickerson 1y ago
    Good feedback. Some of this is intentional - as an independent and growing ~20-person company, we're able to operate more quickly than the larger Mozilla organizations, and we're purposefully distancing ourselves from the associated bureaucracy that comes with any large organization. We are very much in line with the Mozilla ethos around personal ownership, privacy, control, and agency. We're figuring out how to best push on those principles in the world of AI, and appreciate feedback and contributions from the community.
JohnPDickerson 1y ago
Common question, thanks for asking! We’re a public benefit corporation focused on democratizing access to AI tech, on enabling non-AI experts to benefit from and control their own AI tools, and on empowering the open source AI ecosystem. Our majority shareholder is the Mozilla Foundation - the other shareholders being our employees, soon :). As access to knowledge and people shifts due to AI, we’re working to make sure people retain choice, ownership, privacy, and dignity. We're very small compared to the Mozilla mothership, but moving quickly to support open source AI in any way we can.

spooky_deep 1y ago

Really needs a Docker image (maybe just not mentioned?) so one doesn’t have to wrestle pip and python versions.

gapeleon 1y ago

You guys need to fact check your AI-generated blog posts: https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to-access-any-llm-provider/ https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to... > One popular solution, LiteLLM, is highly valued for its wide support of different providers and modalities, making it a great choice for many developers. However, it re-implements provider interfaces rather than leveraging SDKs that are managed and released by the providers themselves. As a result, the approach can lead to compatibility issues and unexpected modifications in behavior, making it difficult to keep up with the changes happening among all the providers. LiteLLM is rock-solid in practice. The underlying API providers announce breaking changes well in advance, and LiteLLM has never been caught out by this. LLMs will come up with hypothetical cons like this upon request. > Lastly, proxy/gateway solutions like OpenRouter and Portkey require users to set up a hosted proxy server to act as an intermediary between their code and the LLM provider. Although this can effectively abstract away the complicated logic from the developer, it adds an extra layer of complexity and a dependency on external services, which might not be ideal for all use cases. OpenRouter is a hosted service that provides the proxy/gateway infrastructure. Users don't "set up a hosted proxy server" themselves; they just make API calls to OpenRouter's endpoints. But older LLMs don't know what OpenRouter is and will assume it's a self-hosted proxy server. > Another option, AISuite, was created by Andrew NG and offers a clean and modular design. However, it is not actively maintained (its last release was in December of 2024) and lacks consistent Python-typed interfaces. Okay so you clicked the "releases" tab and saw December 2024. Next time check https://github.com/andrewyng/aisuite/commits/main/ https://github.com/andrewyng/aisuite/commits/main/ Small, fast moving community projects like this, exllamav2, etc don't necessarily tag releases. I've got nothing against using AI to write posts like this, but at least take the time to fact check before dumping on other people's work. If not for the Mozilla branding, I'd have assumed this was a scam/malware - especially since it's name is so similar to Anything-LLM.