4 ms·
Show HN: Made a Free LLM Router
Hey everyone! Been working on a router that works with OpenAI, Anthropic, and Google (Gemini not Vertex), and just launched it for free.
Let me know what you think.
- anoop_kumar 3mo agoIs this open sourceD? How do we know how the routing happens? IS there an eval?
- Aperswal 3mo agoThis isn't open sourced, as of now. The routing is largely pass through or standard conversions between API surfaces. The SDK uses the OpenAPI specs for it's given API surface. The eval is in the hero of the website itself, everyday it updates with the current median cost saved and latency traded. Additionally, there are a few demos I have added to the website right under the hero.
- BrunkenClaas 3mo agoWhat is the basic logic by which you route?
- Aperswal 3mo agoThe basic logic for default routing or flex routing?
- BrunkenClaas 3mo agoThe flex routing you offer.
- Aperswal 3mo agoIf you give an SLA of lets say 10 seconds to find cheaper inference. I look around different providers that offer flex/batch and try to fulfill in that time. If I can't then I escalate to a default request. This means you definitely get a response in 10 seconds (could still be a 5xx or 429 from default), but in those 10 seconds I might find you cheaper inference and drop costs by 50%, then charge 20% of that as FlexInference's cut, effectively giving you a 40% discount. If I don't find the cheaper inference and have to escalate, then I don't charge at all, so it is free.
- zach7 3mo ago[dead]
- heliskyr2 3mo ago[flagged]