5 ms·
Wayfinder Router: deterministic routing of queries between local and hosted LLM
- ListeningPie 4mo agocan you send to multiple LLMs to compare responses? From that create a heuristic of which LLM gets what.
- try-working 4mo agoLove to see local/cloud routing explicitly supported. I'm building another router for routing between local and remote models, ShowHN coming up later today. Here's a sneak preview of the github: https://github.com/try-works/role-model https://github.com/try-works/role-model
- try-working 4mo agoPosted my ShowHN: https://news.ycombinator.com/item?id=48706181 https://news.ycombinator.com/item?id=48706181
- tcballard 4mo ago[dead]
- tcballard 4mo ago[flagged]
- quijoteuniv 4mo agoThis is the way!
- tcballard 4mo agoI like to think so!
- throwawayk7h 4mo agoIt'd be nice to just have a command prefix e.g. /local fix my typo
- girvo 4mo agoThat’s what I did with Pi, super simple :)
- dd8601fn 4mo agoIt's funny how much that first paragraph is Claude's voice. I don't know how it got trained so hard to use, "the shape of" for everything.
- ninjalanternshk 4mo agoThank you! I thought it was just me.
- paradox460 4mo agoLoads of ed sheeran in the training data?
- ifdefdebug 4mo agoDo you want the honest answer?
- niemandhier 4mo ago[dead]
- kevinten10 4mo ago[flagged]
- josalhor 4mo agoWe need LLM query routing at the OS level like Mobile data. I know it will sound crazy but hear me out. I think about this AI inference as infrastructure. I do not want to pay for it on every app I use it on. I do not think "I have to pay the mobile data of youtube, and the mobile data of whatsapp etc.". I pay Mobile data infrastructure and let my device route it appropiately. In fact, if we ever go the local llm route, you could have LLM capabilities without having access to the internet (or local LAN), and your OS/computer is the only one capable of doing that routing for you.
- solenoid0937 4mo agoIt doesn't sound crazy at all, this seems almost obvious. The OS should provide a chat completions server and the user should be able to select the underlying LLM's server. This should be just like selecting a default search engine or browser. Hopefully the EU forces US tech giants to do this. God knows Apple and Google won't do this on their own. They gotta get that sweet default provider revenue.
- KronisLV 4mo ago> It doesn't sound crazy at all, this seems almost obvious. The OS should provide a chat completions server and the user should be able to select the underlying LLM's server. This should be just like selecting a default search engine or browser. I wonder why this hasn’t happened yet. If Microsoft wants to have a Copilot button and AI investments are all the rage now, surely anything to make integrating with them would be good for keeping the hype cycle alive for longer?
- dewey 4mo agoBecause it’s still pretty early in this journey and there’s some exclusivity deals to be made.
- number6 4mo agoApple told EU Citizens thats why they cant have Siri on their Phone as AI; they would have to provide an Interface where you can plug in your own LLM of your chosing.
- JSR_FDED 4mo agoSlight tangent, but “Wayfinder sits behind whatever OpenAI-compatible client you already use” reminds me that descriptions of where proxies sit in the information flow always seem so arbitrary to me: - “after the client” - “reverse proxy” (in front of servers) - “proxy” (in front of client) I always have to look this up, surely there must be a standardized way to describe this?
- terekhindc 4mo ago[dead]
- _pdp_ 4mo agoThere are so many proxies like this now but I can tell you from first hand experience this is not going to work. You cannot just route away from a situation at such a high level especially when we are talking about models that are quite different in behaviour, with different context windows and tuned to different tool uses. The harness is doing all kind of funky things to compensate for issues (like tool call truncation) that a proxy that routes dynamically like this will work against the very same strategies that make the harness work. Interesting concept, work in theory, but I cannot see this being part of larger system.
- Semaphor 4mo agoThis is not choosing between different models, really. You should check the (interesting, yet sadly very slop-padded) readme. It’s about trying to make a binary decision: Is this a hard or easy question, and about making that decision extremely fast. They suggest putting another router that chooses the model behind it. I’m not sure how well it would work, but the idea is interesting and different than other routers.
- stanpinte 4mo agoWe are developing many applications in my company, some of them safety critical. A natural routing way could happen for certain phases of development, and interfaces via git. One agent works on branch a and is responsible for brainstorm planning specs, and the other is responsible code and tests. The first agent creates tickets for the second one and the second one consumes these. This works with today’s standard harness.
- kstenerud 4mo agoI'm not sure I understand what this is trying to solve? If a prompt I give routes to one model, and then another prompt to another model, how does one tie the context together such that the next model knows what's going on? Otherwise this would only be useful for one-off prompts as far as I can tell. And if it did keep a context to be passed around, it would always land hot (not in the cache).
- spiderfarmer 4mo agoI'm not sure if output of easy commands like "summarize this" are added back to the context? I always assumed they are in a separate UI layer?
- SwellJoe 4mo agoEvery turn of a conversation with an LLM is getting the whole conversation. Caching complicates the picture, but not by a huge amount. That's why a short question at the end of a long conversation chews tokens faster than it would in a fresh session. So, a conversation that's ongoing with one model then switching to another would presumably send the whole conversation and the new question. Which defeats the purpose of splitting traffic...so, you're not wrong to question how this actually improves things for anything other than short sessions, which you could choose your own model for if it's a small problem.
- nok22kon 4mo agosending the whole conversation to a cheap model could still be cheaper than sending just the latest message to the expensive one you could even take this into account automatically to help decide
- SwellJoe 4mo agoDeepSeek is seemingly novel in its caching capability. In most of my usage of it, I'm seeing something like 80% cached tokens, which makes using DeepSeek cheaper than damned near any other model anywhere near its size. I don't think even my self-hosted models are cheaper once electricity and air conditioning is accounted for (it's summer in Texas, pumping a few hundred watts of heat into the room for hours at a time for local LLM usage is sub-optimal). So, I think if you're using a model that has extremely effective caching (of which there is only one I know of) as your expensive model, you almost certainly won't come out ahead by switching back and forth. Cached tokens in DeepSeek are damned near cheap as free. But, for Claude or GPT where tokens are extremely expensive and caching barely puts a dent in it, yeah, I guess there is math that could work out in favor of this idea.
- kumiko_studio 4mo ago[flagged]
- api 4mo agoI do this manually with a desktop app called BoltAI that lets you continue the whole conversation at your LLM of choice.
- nok22kon 4mo agowe could use some composability. today any kind of routing requires implementing an http proxy to put in the middle ideally harnesses would support a routing plugin which receives the new whole context and returns just where to send it, and the harness does that. no http proxy. obviously some complications if you want to route from codex to anthropic or openrouter. but we need to decouple the context building and routing decision from the actual http requests sending, we need to be able to insert "context/routing plugins" in the chain
- cyanydeez 4mo agoI'm still waiting for an isolated protocol so we don't have to run the hanress directly on any of the code base's infrastructure. Something as simple as piping everything into and out of an ssh shell would be better than anything I've tested so far.
- try-working 4mo agoi've created the protocol, role-model: https://github.com/try-works/role-model https://github.com/try-works/role-model
- niles 4mo agoGreat name, but ironically hard to reason about from a role perspective, at least at the read me. Does this interfere with cache hits? Could a single conversation or task span multiple roles? Why are you building this? Does this maximize my toxen value by saving the hard tasks for the hard model? Does it maximize cache hits as part of its scoring? Does it help agents develop a specialist mindset? Are you anticipating users will have many local models hot, or is this also a model load/unload controller?
- try-working 4mo agoI'm building this to achieve a state where I can, as a user and on my own device, decide that certain type of workloads should be handled by my Qwen model and keep the data on my device, while other workloads should be handled by more capable models. For this we don't just need a router, because the information to make detailed and accurate routing decisions currently doesn't exist. And there are no standards but every lab and maybe even inference providers have their own way of implementing reasoning, chat templates, cache, tool use and so on. All issues that make models non-interoperable. What we need is applications that clearly specify their requests so they can be accurately routed to a provider, whether local or remote. And for that they need to use a standard protocol for model requests and intent. I wrote a longer piece here: https://news.ycombinator.com/item?id=48706181 https://news.ycombinator.com/item?id=48706181
- harvardhan1 4mo agowfdwcZz
- deleted 4mo ago[deleted]
- harvardhan1 4mo agoaxa
- systemalice 4mo ago[flagged]
- mrkn1 4mo agoHas anyone tried the others listed? Any feedback?
- darepublic 4mo agoSome kind of routing prompts to different models does make sense. But the usecase of saving money on simple prompt.. I think that has only a slight benefit. Fix my typo doesn't use many tokens anyhow.. also model switching still requires carrying over context so it does have some overhead right.
- SnipVote 4mo ago[dead]
- hmokiguess 4mo agoThere's a hidden tax with routing this way, the original model loses context of what was done and either performs a regression or hallucinates. I think this sort of behaviour started happening more frequently as agentic/ai programming became more often. Back in the days (lol, reads like a long time ago but that's probably a few months?), you would not say "edit this typo", you would just open the file and not be lazy, and the harness would detect a user change and ground itself. I feel like now, when I edit outside the AI flow, it goes and introduces a regression or gets lots thinking it didn't do that and something must have gone wrong.
- madikz 4mo ago[flagged]
- pennylaw946 4mo ago[flagged]
- billiam 4mo agoI sure hope the community is working on these APIs now for Linux, with pressure to come on M$ and Apple.
- jarodrh 4mo agoInteresting. Read through the comments and the cache issues and context loss issues were already mentioned and they are all good valid points. I'll go from the positive side. I think the part that resonates most with me is the deterministic rule based routing. It can even double as a policy that you can audit. Configuring which calls would go external and internal.....this could work well for companies who want to enforce certain privacy policies on what data goes to external frontier models vs internal local models. I think the privacy angle is stronger here over the cost angle. I believe a config-led setup is the way over a model trained on routing. Opus (I'm most familiar with) and I'm sure other models already have the subagent spawning skills down-pat. I basically do a similar routing by config setup but using Claude with some rules and a framework. So if any routing is needed it might be at the point you've identified....locally on device config-led. Neat. If I get the chance I will check it out and give it a whirl.