6 ms·
Show HN: Litellm – Simple library to standardize OpenAI, Cohere, Azure LLM I/O
I built this library because langchain was too bloated and I needed a simple abstraction to call multiple LLM APIs. litellm has two functions - completion(), embedding()
- detente18 3y ago[dead]
- neha_n 3y agoThis is much needed. For someone looking to quickly implement, having a simple interface goes a long way.
- detente18 3y agoThanks
- yding 3y agoVery cool Ishaan!
- ij23 3y agoThank you !
- hardware2win 3y ago>completion(..., azure=True) Why like this?
- detente18 3y agoazure uses the openai python sdk, just remaps certain components. The models are also user-named. This makes it hard to detect if a model passed in is an azure model
- ij23 3y agoazure models have custom names - eg I call mine 'chat-gpt-test1', I require some flag to know if it's an azure model
- d4rkp4ttern 3y agoGreat start! Are you planning to add to following: Retries w exponential backoff, Caching, Streaming output, Function-calling support
- detente18 3y agoYou can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up doing. Streaming output and function-calling support is interesting
- d4rkp4ttern 3y agoCould you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here
- detente18 3y agoYou're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response, why would you cache the openai call instead of the endpoint that's receiving the user call?
- ij23 3y agogood points, probably going to add streaming output, function calling support. As for retries tenacity does a great job already
- kaushik92 3y agoThis is amazing. Really needed something like this to standardize all my different AI APIs! On a side note - I love how quickly your team is shipping! Do keep it going!
- uripeled2 3y agoTake a look at llm-client a similar library that also support chat, async and more llm providers https://github.com/uripeled2/llm-client-sdk https://github.com/uripeled2/llm-client-sdk
- ij23 3y agothanks for sharing, while your library looks really powerful my goal with Litellm is simplicity