3 ms·
Show HN: We built open OpenRouter that turns usage into a better model
Hi HN, we built an open source model gateway. It's a single place to manage our own self hosted, frontier, and open source models in one place.
It’s is rust native, built for concurrency, and implements all the config quirks across models and providers (streaming formats, tool calls, model parameters, rate limits, and different error behavior).
The gateway adds under 1 ms for BYOK requests and under 2 ms when Experiential supplies the provider key. It has every major inference provider, and 1000+ models refreshed daily via a codex agent that opens a PR.
Compared to other similar projects we’re open source, take no markup, allow you to mix local models with a marketplace, and use your traffic to (opt in) train you a model. Simple routing doesn’t warrant a 10% token markup.
The way we do this is given standardized OTel traces, we mine representative real tasks, use text world models to simulate rollouts for various models, apply an LLM judge, and fit a nearest neighbor classifier on top of an embedding of a prompt to decide the optimal model for each request. Usually this can map out a better pareto curve on cost/quality than just calling single models but it’s not perfect.
Using these simulations we can also do things like suggesting cache hit optimizations, new model suggestions, and training models.
It’s open source, so you can deploy it on your own infrastructure, use our hosted version with 0 markup, or read how we design for maximum availability on our website.
- Areibman 29d agoCould you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
- purplecats 29d agoand caching is related to performance too ofc
- SilenN 29d agoThe trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".
- cameronh90 29d agoBut then it's better to just not have a gateway switch models at all. Just have the harness able to choose which model its sub-agents use, then tell it how to split up tasks and which models to use when doing so.
- SilenN 29d agoThat is another way to do. Or we can automatically figure out which models the subagents should be using for you. And update them as new models come out and the work your subagents do changes. More than one way to skin a cat.
- aerzen 29d agoThis does make sense. I generally only switch between models in pi when creating a new session. And it is apparent from the promt if this just a "how to see open ports on linux" or "make a concrete plan for feature X"
- try-working 29d agoGenerally you should only have two models in the pool per domain. I wrote some of my learnings building a router here: https://try.works/first-principles-of-model-routing https://try.works/first-principles-of-model-routing
- ashermania 29d agoFinally an open source tool doing this!
- 23david 29d agoSuper interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
- SilenN 29d agoYep! If you look at the commit history that's exactly what happened.
- cheema33 29d agoI have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
- ceroxylon 29d ago>The gateway adds under 1 ms for BYOK requests Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.
- kfallah15 29d agoThanks! We are going to add continual RL via Tinker soon too
- akshay_akula 29d agoOpen source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
- SilenN 29d agoAns: we rarely switch, often times it's just a "switch to using this model for your agent"
- akshay_akula 29d agoYou guys should look into ngrok ai gateway. We have some small models running on local hardware that we tried to use but it was too painful. Just wanted to not waste all the compute hit one endpoint and be done but as a team.
- jing09928 29d ago[dead]
- 0xbadcafebee 29d agoYou started it a week ago? I look forward to checking back in 3 weeks when you've exited for $1B
- SilenN 29d agoSee you soon
- tyre 29d agoLooks like first PR is June 24th: https://github.com/experientiallabs/experiential/pull/1 https://github.com/experientiallabs/experiential/pull/1 So, two months. Still impressive!
- rdslw 29d agoimpressive only if using pre-gpt era assumptions about saas/products/software. unfortunately a small team can reproduce it in two months, which greatly lowers value of it. we, as a collective, have to change our value-judging logic and tune it to post AI world.
- SilenN 29d agoThanks for the positivity tyre! If you look at our git history, we pivoted and only started building the gateway recently. Before that we were building research infrastructure that now powers the intelligence features we provide.
- swthbht 29d agoVery cool. Does your gateway decide effort levels as well? Or just models?
- SilenN 29d agoYep! One interesting example is often Opus 5 on low reasoning ~= Opus 5 on high reasoning.
- sangwook 29d agoWhat online signal recalibrates simulated rankings against actual task success? Also do you have a plan to support semantic caching at the router level?
- kfallah15 29d agoFor the online signal, we use a LLM judge with a rubric calibrated offline by the user via TUI. UX of the calibration is a major focus area. Semantic caching is interesting, open to supporting it but not currently planned.
- forgetme2020 29d agowhat's the business model here. How does experiential labs make money
- kakugawa 29d agoThey make money on enterprise plans: https://www.experientiallabs.ai/pricing#enterprise https://www.experientiallabs.ai/pricing#enterprise Look at the Intelligence features in the Enterprise plan: * Per-prompt model optimization * Caching * A model you own, trained on your traffic
- kfallah15 29d agoyep, it will be through enterprise licenses and our own hosted platform built on the repo
- rdslw 29d agothe business model, I suspect, is classic rug-pull in some time after building user base. proof: boldly claiming being open source in literally first sentence, while cowardly hiding on-by-default telemetry (WTF??) in truly last paragraph of readme. sorry to sound harsh, but this is typical old era playbook here. in the era of AI, fortunately, such products has much lower value. people and VCs didn’t yet tune to it.
- SilenN 29d agoTelemetry is off by default. PostHog is for usage analytics on the open source repo. Audit it if you're skeptical. We make money off enterprise licenses and hosting models. I have strong reason to suspect you either can't read or are a bad actor.
- aHumbleUser 29d agoYour GitHub readme says telemetry is on by default
- deleted 29d ago[deleted]
- gpiechnik2 29d agogreat design! i love it
- SilenN 29d agoThank you!
- nejch 29d agoA part of me wishes the open source community would focus making research and industry-backed initiatives like the vLLM Semantic Router rock solid. Then I'd spend less time every month checking if this or that new model router has differentiating over vllm-sr :) At least for open source inference, it seems like there's healthy competition centered around vllm/sglang, but 2026 seems to be for model routers what 2025 was for agent harnesses.
- d2p 29d ago> and use your traffic to (opt in) train you a model. Is there more info on this? I'm curious exactly what it is. Is it fine-tuning/LoRA on some base model? Don't cloud providers encrypt reasoning now - does that prevent this?
- croemer 29d agoProbably shouldn't call it "Open router" in the title as that's a specific brand. Maybe you meant "we built something like OpenRouter". Rereading I see you wrote "open OpenRouter", which looks a bit like a typo at first glance.
- alescalaios 29d ago[flagged]
- mongrelion 29d agoCongrats on the launching of your product. I will be taking it for a spin to compare it with these other products that seem to be competing directly with what you have to offer: - https://github.com/ENTERPILOT/GoModel https://github.com/ENTERPILOT/GoModel - https://github.com/maximhq/bifrost https://github.com/maximhq/bifrost - https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm Would you care to share what makes experiential different?
- santiago-pl 26d agoGoModel author here. If you are looking for a self-reproducible benchmark, I run this one periodically. I keep it as honest as possible to know how GoModel stands in comparison to its competitors. Any feedback appreciated via GitHub issues: https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark/ https://github.com/ENTERPILOT/ai-gateway-reproducible-benchm...
- foremerge 29d agoWe built our own model router at GPTree but this looks interesting. Caching is definitely one of the hardest parts to get right, especially in our scenario where users can branch from any part of the conversation and keep the dynamic context from the parent (unlike most other platforms that lock in the context once you branch).
- jakswa 29d agoI think I have to look at setting up one of these AI gateways for work, since AWS bedrock is such a PITA to hook a harness up to over IAM roles. Also I still can't believe bedrock hasn't released any open models in months (so there's paranoia that I'll want to swap in another provider). Really though I'm hoping anthropic fixes the oppressive claude verbosity. I saw someone refer to being "clauderboarded" and my brain cannot let go of this as Claude's tokens bombard me.
- SilenN 29d agoWe had this exact problem so we solved it for ourselves. Happy to help if you run into any issues. I have a /hmmm command I use for the second part that works reasonably well: "Stop using jargon and speak coherently. State it more simply and concisely, like one human talking to another. Make it like google dev docs style. More dead prose. No aphorisms, no flourishes. Simple."
- jakswa 29d agoOh. The UI screenshot on github is... not actually in the github repo? It's platform/hosted only? There's my first awkward discovery, but makes sense in retrospect.
- carloslfu 29d agowhy text world models? Is this backed by any evidence or did you find it to be good in practice?
- bicepjai 29d agoGreat tool. Thanks. I am working on something similar.
- heliskyr2 28d ago[flagged]
- brianpg 27d ago[flagged]
- aquadros 25d ago[dead]
- KhanBotFun 24d ago[flagged]