6 ms·

Looking at all these releases it’s not a surprise that model routing is the fastest growing segment in AI right now. There are 10+ LLM companies, each with doz

by paxys 3mo ago

Looking at all these releases it’s not a surprise that model routing is the fastest growing segment in AI right now. There are 10+ LLM companies, each with dozens of models of different modalities, each model with multiple size variants, then different “thinking” levels, then agentic modes, “pro” modes, a “fast” option, standard vs flex vs batch execution. And of course each end combination has a different input/output/cache token price. Companies that say “give me a prompt and I’ll route it to the most ideal and cost effective model and setting for you” are capturing a ton of value from a gap that model developers don’t seem to understand exists.

hnfong 3mo ago
Because they're trying very hard not to understand it. Otherwise the expensive-yet-powerful model probably won't see much revenue. How much money is there in bleeding edge scientific research? There's a lot, but there's even more existing capital in paying people people to do college level paperwork, and the bulk of those traffic gets routed to the cheapest model. You mostly don't need super powerful AGI to replace the paper pushers, but the frontier labs are trying to position themselves as being uniquely capable of producing super powerful AGI, and also be the ones replacing office workers. Not sure how it will work out for them, but I think model routing is going to poke holes in that narrative. That's why I think they're trying very hard not to understand model routing exists.
- ip26 3mo ago
  It wasn’t long ago at all that the chief problem was “can AI even help me with this” (cost be damned). Until a time when the answer is an unmitigated “yes, obviously”, the frontier labs have everything to lose and nothing to gain on routing, because if they screw it up you might incorrectly decide “nope, it can’t help yet” due to a poor routing decision.
- tackta 3mo ago
  Is there really that much money in bleeding edge scientific research though? This is what terrifies me about this whole ordeal economically. Maybe we get AGI and it is not worth anything close to what we thought it was for those who have a bet on it. I think of what was the direct, economic value in the betting sense of quantum mechanics or relativity? Huge value at the systems level of society but as you scale down towards the individual the value is more and more dispersed to the point I would think any pool of bets would have all not paid off. You can't monopolize and commoditize relativity. I almost think there is a kind of dutch book against the AI equity holder because even in the best case scenario the bet doesn't pay off anything close to what is expected for an individual bet.
simianwords 3mo ago
Openrouter should ideally kill in this space and make their model agnostic infra like memory, harnesses, chat applications.
- verdverm 3mo ago
  OpenRouter is in acquisition talks with Stripe, fyi I would expect routers to commodify like tokens.
  - torginus 3mo ago
    OpenRouter sprang up overnight. I might need to replace some urls and access tokens should they decide to try and screw me.
    - verdverm 3mo ago
      I never signed up because I found the 5.5% fee on token usage to be a "screw you" tactic. Still do not understand why they are popular with the other options out there.
TeMPOraL 3mo ago
Who are the customers though? Honest question, I'd like to understand it. For me, anything other than current best available SOTA for any task is unacceptable. The only routing rule I need is "the most powerful model I still have flat-priced quota available for". I mean, why settle for less?
- polotics 3mo ago
  because "less" can be so much faster?
  - kxxx 3mo ago
    not just that -- "less" ($$$) can also result in indistinguishable quality for some tasks/inputs. I'd argue this is the primary reason, secondary being speed.
- msabalau 3mo ago
  Not everyone is you. Other people probably have a range of tasks that can accomplished with different models. Certainly if I'm confident that I'm going to get what I need from a faster model, that's what I want to use, rather than wasting time grinding away for the sake of saying of the same answer came from a SOTA model. Given that every chatbot does offer a range of models, it seems clear people do choose among options.
  - internet2000 3mo ago
    The mental effort in estimating what model would be better is so not worth it. I just want to switch to Claude Code, tell it to turn a .csv into a BigQuery table then cmd+tab to something else while it runs. Thinking "oh this is probably an easy task, I can /model to Sonnet to save $0.0004" is silly.
    - resonious 3mo ago
      The game changes when it's not just "Opus vs Sonnet", but "Opus vs GLM". The amount saved is way more than even $0.04. And it's not only money but speed. Some providers can serve GLM crazy fast - I'll even go outside of my subscription to pay extra money for the quick results.
ai-x 3mo ago
Model Routing will always be done better by models themselves. Plus routing loses context making it more expensive and less reliable. Model Routing is just Bitter lesson. The models themselves will get better at this and frontier companies will simply give that capability
- johnfn 3mo ago
  This doesn’t seem obviously true, eg an Anthropic model will never route to Kimi even if it were best suited for a particular task.
  - posix_compliant 3mo ago
    I think what the parent is saying is that the model itself has the best context for whether a portion of a request should be routed. The specifics of that routing (e.g., should you route to KimiK2) are something that can be trained, finetuned, or even included in a model's startup context.
    - johnfn 3mo ago
      This doesn't seem quite right. For one, I don't need all the intelligence of an expensive model like Opus 5 to do the relatively simple task of choosing a correct model for a task. Additionally, since this isn't something Anthropic would ever put effort into doing well, you could tune a model to do better and faster than Opus 5 does out of the box.
      - kolinko 3mo ago
        Are you speaking from experience? My experience is the opposite - for many cases it’s not very obvious how good a model needs to be to solve it. Worse models tend to just follow their first instincts without proper reasoning And also btw you don’t need a routing company to decide, you can do it on your harness. And yeah my Fable has zero issues delegating to Terra instead of Opus.
      - dannyw 3mo ago
deleted 3mo ago
[deleted]
awongh 3mo ago
what's the threshold for model routing where you're willing to trust the router? For coding my own work I don't trust the model router, and it would have to be shown to be to save a real dollar amount. From a buying perspective it's a hard sell to save x but lose out on bugs you are probably introducing at an unquantifiable severity and frequency. How much is it worth to hedge your bets by doing every single inference request on the frontier model? How much will it cost to go back later and fix things, but also the meta question of how to be able to decide on a hypothetical unknowable? (You'll never know how much better or worse your code was gonna be, it's untestable at a project level)
- binary132 3mo ago
  you trust the service provider but not the router? weird, but ok
  - awongh 3mo ago
    I don't want to save money so badly that I'd possibly undercut the quality of the code that gets created. *edit to add: that code quality (or lack of quality) is it's own cost
TacticalCoder 3mo ago
And it's not just that model routing is much cheaper: no longer than yesterday we got a post showing that routing between K3 and Fable 5 was more SOTA than either of those. If that is true, model routing is here to stay. It also seems to validate the minimalist approach of pi.dev, where sub-agents from the same company is not the preferred approach (pi.dev believes in neither sub-agents all from the same company nor MCP even you can do it if you want for pi.dev's philosophy is to do add any functionality you want to a minimal harness). Now of course we'll get for a few weeks all the Anthropic fanbois and shills explaining that "sure, K3 was basically at the level of Fable 5 but now that Opus 5 is out, open-weights models are six months behind".
ModernMech 3mo ago
lol I had to get ChatGpt to explain to me the difference between 5.6 sol, 5.6 Terra, 5.6 Luna, 5.5, 5.4 mini, 5.3 spark, and then there is low, medium, high, extra high, max, ultra, and pro… I still don’t really know, it feels like ordering hot wings.
owenthejumper 3mo ago
I am not convinced this will end up being a domain of the ‘routers’ vs the clients, as in harnesses themselves. Thoughts?
throw03172019 3mo ago
How does model routing work if the prompt is static? Do the routers attempt to tweak the prompt to make it the best version for that model? For example, reading the “migration” guides for Claude Opus 4.6 -> Opus 5 is exhausting. Adding in all of the other models complicates it even more. Especially across vendors.

by paxys 3mo ago

hnfong 3mo ago

Because they're trying very hard not to understand it. Otherwise the expensive-yet-powerful model probably won't see much revenue. How much money is there in bleeding edge scientific research? There's a lot, but there's even more existing capital in paying people people to do college level paperwork, and the bulk of those traffic gets routed to the cheapest model. You mostly don't need super powerful AGI to replace the paper pushers, but the frontier labs are trying to position themselves as being uniquely capable of producing super powerful AGI, and also be the ones replacing office workers. Not sure how it will work out for them, but I think model routing is going to poke holes in that narrative. That's why I think they're trying very hard not to understand model routing exists.

ip26 3mo ago
It wasn’t long ago at all that the chief problem was “can AI even help me with this” (cost be damned). Until a time when the answer is an unmitigated “yes, obviously”, the frontier labs have everything to lose and nothing to gain on routing, because if they screw it up you might incorrectly decide “nope, it can’t help yet” due to a poor routing decision.
tackta 3mo ago
Is there really that much money in bleeding edge scientific research though? This is what terrifies me about this whole ordeal economically. Maybe we get AGI and it is not worth anything close to what we thought it was for those who have a bet on it. I think of what was the direct, economic value in the betting sense of quantum mechanics or relativity? Huge value at the systems level of society but as you scale down towards the individual the value is more and more dispersed to the point I would think any pool of bets would have all not paid off. You can't monopolize and commoditize relativity. I almost think there is a kind of dutch book against the AI equity holder because even in the best case scenario the bet doesn't pay off anything close to what is expected for an individual bet.

simianwords 3mo ago

Openrouter should ideally kill in this space and make their model agnostic infra like memory, harnesses, chat applications.

verdverm 3mo ago
OpenRouter is in acquisition talks with Stripe, fyi I would expect routers to commodify like tokens.
- torginus 3mo ago
  OpenRouter sprang up overnight. I might need to replace some urls and access tokens should they decide to try and screw me.
  - verdverm 3mo ago
    I never signed up because I found the 5.5% fee on token usage to be a "screw you" tactic. Still do not understand why they are popular with the other options out there.

TeMPOraL 3mo ago

Who are the customers though? Honest question, I'd like to understand it. For me, anything other than current best available SOTA for any task is unacceptable. The only routing rule I need is "the most powerful model I still have flat-priced quota available for". I mean, why settle for less?

polotics 3mo ago
because "less" can be so much faster?
- kxxx 3mo ago
  not just that -- "less" ($$$) can also result in indistinguishable quality for some tasks/inputs. I'd argue this is the primary reason, secondary being speed.
msabalau 3mo ago
Not everyone is you. Other people probably have a range of tasks that can accomplished with different models. Certainly if I'm confident that I'm going to get what I need from a faster model, that's what I want to use, rather than wasting time grinding away for the sake of saying of the same answer came from a SOTA model. Given that every chatbot does offer a range of models, it seems clear people do choose among options.
- internet2000 3mo ago
  The mental effort in estimating what model would be better is so not worth it. I just want to switch to Claude Code, tell it to turn a .csv into a BigQuery table then cmd+tab to something else while it runs. Thinking "oh this is probably an easy task, I can /model to Sonnet to save $0.0004" is silly.
  - resonious 3mo ago
    The game changes when it's not just "Opus vs Sonnet", but "Opus vs GLM". The amount saved is way more than even $0.04. And it's not only money but speed. Some providers can serve GLM crazy fast - I'll even go outside of my subscription to pay extra money for the quick results.

ai-x 3mo ago

Model Routing will always be done better by models themselves. Plus routing loses context making it more expensive and less reliable. Model Routing is just Bitter lesson. The models themselves will get better at this and frontier companies will simply give that capability

johnfn 3mo ago
This doesn’t seem obviously true, eg an Anthropic model will never route to Kimi even if it were best suited for a particular task.
- posix_compliant 3mo ago
  I think what the parent is saying is that the model itself has the best context for whether a portion of a request should be routed. The specifics of that routing (e.g., should you route to KimiK2) are something that can be trained, finetuned, or even included in a model's startup context.
  - johnfn 3mo ago
    This doesn't seem quite right. For one, I don't need all the intelligence of an expensive model like Opus 5 to do the relatively simple task of choosing a correct model for a task. Additionally, since this isn't something Anthropic would ever put effort into doing well, you could tune a model to do better and faster than Opus 5 does out of the box.
    - kolinko 3mo ago
      Are you speaking from experience? My experience is the opposite - for many cases it’s not very obvious how good a model needs to be to solve it. Worse models tend to just follow their first instincts without proper reasoning And also btw you don’t need a routing company to decide, you can do it on your harness. And yeah my Fable has zero issues delegating to Terra instead of Opus.
    - dannyw 3mo ago

deleted 3mo ago

[deleted]

awongh 3mo ago

what's the threshold for model routing where you're willing to trust the router? For coding my own work I don't trust the model router, and it would have to be shown to be to save a real dollar amount. From a buying perspective it's a hard sell to save x but lose out on bugs you are probably introducing at an unquantifiable severity and frequency. How much is it worth to hedge your bets by doing every single inference request on the frontier model? How much will it cost to go back later and fix things, but also the meta question of how to be able to decide on a hypothetical unknowable? (You'll never know how much better or worse your code was gonna be, it's untestable at a project level)

binary132 3mo ago
you trust the service provider but not the router? weird, but ok
- awongh 3mo ago
  I don't want to save money so badly that I'd possibly undercut the quality of the code that gets created. *edit to add: that code quality (or lack of quality) is it's own cost

TacticalCoder 3mo ago

And it's not just that model routing is much cheaper: no longer than yesterday we got a post showing that routing between K3 and Fable 5 was more SOTA than either of those. If that is true, model routing is here to stay. It also seems to validate the minimalist approach of pi.dev, where sub-agents from the same company is not the preferred approach (pi.dev believes in neither sub-agents all from the same company nor MCP even you can do it if you want for pi.dev's philosophy is to do add any functionality you want to a minimal harness). Now of course we'll get for a few weeks all the Anthropic fanbois and shills explaining that "sure, K3 was basically at the level of Fable 5 but now that Opus 5 is out, open-weights models are six months behind".

ModernMech 3mo ago

lol I had to get ChatGpt to explain to me the difference between 5.6 sol, 5.6 Terra, 5.6 Luna, 5.5, 5.4 mini, 5.3 spark, and then there is low, medium, high, extra high, max, ultra, and pro… I still don’t really know, it feels like ordering hot wings.

owenthejumper 3mo ago

I am not convinced this will end up being a domain of the ‘routers’ vs the clients, as in harnesses themselves. Thoughts?

throw03172019 3mo ago

How does model routing work if the prompt is static? Do the routers attempt to tweak the prompt to make it the best version for that model? For example, reading the “migration” guides for Claude Opus 4.6 -> Opus 5 is exhausting. Adding in all of the other models complicates it even more. Especially across vendors.