3 ms·
Show HN: A lightweight AI gateway to 100+ models, in TS
We are Rohit & Ayush, we created Portkey this year March to help tackle some challenges we had seen while building apps based on GPT3, 3.5, 4, and the DevOps principles we brought to the scene to help tackle them.
We believe, a solid, performant, and reliable gateway lays the foundation to help build the next level of LLM apps. It decreases excessive reliance on any one company and takes the focus back to building instead of spending time fixing the nitty gritties of different providers and making them work together.
Features:
Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
Load balance across multiple models, providers, and keys
Fallbacks make sure your app stays resilient
Automatic Retries with exponential fallbacks come by default
Plug-in middleware as needed
Battle tested over 100B tokens and millions of requests
For the folks serious about gateway, separation of concerns, TS developers.. I'd love to hear your thoughts and we're hungry for feedback!
Reach out to us at hello@portkey.ai or explore the project: https://github.com/portkey-ai/gateway https://github.com/portkey-ai/gateway
- retrovrv 3y agoPretty excited to announce this! While there are some popular and awesome AI gateways out there, like litellm, bricksai - none are written in TS, and for the TS ecosystem. Looking forward to the community's feedback
- roh26it 3y agoWhile I love and use Python everyday, I do believe that JS is a lot more popular and mainstream for web development today.
- CShorten 3y agoPortkey is super cool, congrats Rohit and Ayush on the HN launch!
- roh26it 3y agothanks! we spent very long building it out properly. Time to integrate weaviate generators now?
- ayush-garg 3y agoA few months back, we embarked on what seemed like another regular project. Little did we know, it would turn into such a game changer for LLM deployments. Feels surreal that this gateway is already processing upwards of 3B tokens a day in such a short time.
- deleted 3y ago[deleted]
- aravindputrevu 3y agoCongratulations Rohit, Ayush and team Portkey on the launch! Excited to take it for a spin and contribute!
- roh26it 3y ago_/\_
- pranavgoyal 3y agoCongrats on the launch folks! This sounds really amazing - rooting for success!
- kaushik92 3y agoCongrats on the launch! Looks amazing
- shaial 3y agoCongrats, this looks neat, and surely great to have more TS products in the ecosystem. One plugin or feature that I will like to see in an AI gateway: *Cache* per unique request. So if I send the same request (system, messages, temperature, etc.), I will have the option to pull if from a cache (if it was already populated) and skip the LLM generation. This is much faster and cheaper - especially during development and testing.
- retrovrv 3y agoThank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here - https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic https://portkey.ai/docs/product/ai-gateway-streamline-llm-in... The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!
- shaial 3y agoAwesome! We built the simple version in-house, and hoped someone would productize it.
- kesavan_kk 3y agoCongratulations on this launch. Looks awesome.