4 ms·

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks li

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.

foota 2mo ago
I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code. I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.
- kumama 2mo ago
  castform founder here. that's a future we are really excited about too :) ideally, you can post-train the llm within the application itself, as it's being used. both interesting infrastructure & algorithmic challenges here
Malp 2mo ago
There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc. There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).
- fennecfoxy 2mo ago
  By post-train I presume you mean a finetune? Unless that's wrong (please correct me if so). I haven't looked into model architecture people are working with for this stuff too deeply yet but I presume the core idea is fine-tuning a lightweight reasoning-enabled LLM specifically using search as a metric for training?
  - Malp 2mo ago
    That or providing a concrete RL env for $your_search_corpus_etc_here
devolving-dev 2mo ago
Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?
- kumama 2mo ago
  (one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements
Razengan 2mo ago
> There is so much opportunity for purpose built models like this. OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes
- mrinterweb 2mo ago
  Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.
- kumama 2mo ago
  castform founder here. openai actually deprecated their finetuning apis a few months back weirdly.
phainopepla2 2mo ago
> Claude code does some of this by handing off the "explore" agent work to haiku This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku" [0] https://code.claude.com/docs/en/changelog#2-1-198 https://code.claude.com/docs/en/changelog#2-1-198
benjiro29 2mo ago
> Claude code does some of this by handing off the "explore" agent work to haiku. That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs. The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model. There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.
- BikiniPrince 2mo ago
  You can register models with mcp. I think it’s an expensive solution, but it is available in the framework. I use a light weight bus protocol that lets agents interact and pass short messages with pointers. It’s very efficient.
- tyre 2mo ago
  > There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense. As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
  - kumama 2mo ago
    castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head - high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement could mean millions in revenue :)
oliver236 2mo ago
this is exactly what leopold talks about in situational awareness
try-working 2mo ago
yes, and this is why we need model routing
- kumama 2mo ago
  castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)
  - dd8601fn 2mo ago
    I’ve been trying to do this with a pet project and admit that I’m getting terrible results. My small llm as a classifier/router stage just isn’t getting the job done.
nikcub 2mo ago
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
- bizzletk 2mo ago
  If the PhDs don't cost very much more than your equivalent of factory-technicians but still get the job done, why wouldn't you do that, at least in the blunt case before cost control rears up?
  - polotics 2mo ago
    because they will take initiatives for localized improvements you don't want them to take
- dwaltrip 2mo ago
  This only works if the tasks are actually specific and don’t benefit from broad competency. IMO, this doesn’t match most things that people use LLMs for.
nullbio 2mo ago
Yep. It's a real shame that the labs are incentivized not to go in this direction. They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent. They naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth. I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful. Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.
kinnth 2mo ago
isn't this also the threat to frontier AI houses? As in they want you to expend tokens in their ecosystem, but the optimization at 100x is their profit?
- kumama 2mo ago
  castform founder here. i'd say it's a threat but the frontier ai labs' argument would basically be that the market opportunity for intelligence is infinite so it doesn't matter. at the same time, i do believe there will continue to be a massive, growing market for big labs, esp for super-frontier use-cases. today that's longer-horizon coding tasks but in the future it can be things like scientific discovery, etc.
boonzeet 2mo ago
This is how Sakana's Fugu model works, achieving similar performance to Opus/Fable with a mix of open source & mainstream LLMs.
drob518 2mo ago
IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.
- fennecfoxy 2mo ago
  It is nice to have a model that can "do it all", though. And surely that's still the end goal? Like how MoE is still somewhat popular in certain areas even after its heyday. I am wondering if models will end up being some sort of evolution of MoE where it has something internally like the model the author refers to that gets surfaced when it needs to search in some way. I guess it makes sense; our own brains have so many distinct task-specific regions.
  - drob518 2mo ago
    Yes, agreed, but I think it’s going to be difficult to have a high quality model that “does it all” and have it be local. For a quality “does it all” model, you’re going to need a lot of parameters and that means you’re almost always going to be running in the cloud. But it seems like you could probably get a smaller, focused model that runs locally and is also high quality. In other words, I want the programming expert portion of a 2T model that is maybe 35B or 70B parameters or whatever but it’s running locally (and yes, I know you don’t just carve out an expert from a larger model, but conceptually that’s what I’m after).
    - fennecfoxy 2mo ago
      Yeah, I always thought the future of this stuff would be hot-pluggable MoE modules or LORAs that are able to be downloaded and applied/used at will like how skills have become a thing. Like, atm most architectures seem limited by a single context and fixed architecture with no hot loading. Especially for robotics, being able to load/unload various specialised skills on limited mobile hardware will (I hope) definitely become a thing.

foota 2mo ago

I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code. I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.

kumama 2mo ago
castform founder here. that's a future we are really excited about too :) ideally, you can post-train the llm within the application itself, as it's being used. both interesting infrastructure & algorithmic challenges here

Malp 2mo ago

There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc. There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).

fennecfoxy 2mo ago
By post-train I presume you mean a finetune? Unless that's wrong (please correct me if so). I haven't looked into model architecture people are working with for this stuff too deeply yet but I presume the core idea is fine-tuning a lightweight reasoning-enabled LLM specifically using search as a metric for training?
- Malp 2mo ago
  That or providing a concrete RL env for $your_search_corpus_etc_here

devolving-dev 2mo ago

Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?

kumama 2mo ago
(one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements

Razengan 2mo ago

> There is so much opportunity for purpose built models like this. OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes

mrinterweb 2mo ago
Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.
kumama 2mo ago
castform founder here. openai actually deprecated their finetuning apis a few months back weirdly.

phainopepla2 2mo ago

> Claude code does some of this by handing off the "explore" agent work to haiku This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku" [0] https://code.claude.com/docs/en/changelog#2-1-198 https://code.claude.com/docs/en/changelog#2-1-198

benjiro29 2mo ago

> Claude code does some of this by handing off the "explore" agent work to haiku. That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs. The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model. There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.

BikiniPrince 2mo ago
You can register models with mcp. I think it’s an expensive solution, but it is available in the framework. I use a light weight bus protocol that lets agents interact and pass short messages with pointers. It’s very efficient.
tyre 2mo ago
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense. As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
- kumama 2mo ago
  castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head - high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement could mean millions in revenue :)

oliver236 2mo ago

this is exactly what leopold talks about in situational awareness

try-working 2mo ago

yes, and this is why we need model routing

kumama 2mo ago
castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)
- dd8601fn 2mo ago
  I’ve been trying to do this with a pet project and admit that I’m getting terrible results. My small llm as a classifier/router stage just isn’t getting the job done.

nikcub 2mo ago

There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.

bizzletk 2mo ago
If the PhDs don't cost very much more than your equivalent of factory-technicians but still get the job done, why wouldn't you do that, at least in the blunt case before cost control rears up?
- polotics 2mo ago
  because they will take initiatives for localized improvements you don't want them to take
dwaltrip 2mo ago
This only works if the tasks are actually specific and don’t benefit from broad competency. IMO, this doesn’t match most things that people use LLMs for.

nullbio 2mo ago

Yep. It's a real shame that the labs are incentivized not to go in this direction. They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent. They naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth. I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful. Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.

kinnth 2mo ago

isn't this also the threat to frontier AI houses? As in they want you to expend tokens in their ecosystem, but the optimization at 100x is their profit?

kumama 2mo ago
castform founder here. i'd say it's a threat but the frontier ai labs' argument would basically be that the market opportunity for intelligence is infinite so it doesn't matter. at the same time, i do believe there will continue to be a massive, growing market for big labs, esp for super-frontier use-cases. today that's longer-horizon coding tasks but in the future it can be things like scientific discovery, etc.

boonzeet 2mo ago

This is how Sakana's Fugu model works, achieving similar performance to Opus/Fable with a mix of open source & mainstream LLMs.

drob518 2mo ago

IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.

fennecfoxy 2mo ago
It is nice to have a model that can "do it all", though. And surely that's still the end goal? Like how MoE is still somewhat popular in certain areas even after its heyday. I am wondering if models will end up being some sort of evolution of MoE where it has something internally like the model the author refers to that gets surfaced when it needs to search in some way. I guess it makes sense; our own brains have so many distinct task-specific regions.
- drob518 2mo ago
  Yes, agreed, but I think it’s going to be difficult to have a high quality model that “does it all” and have it be local. For a quality “does it all” model, you’re going to need a lot of parameters and that means you’re almost always going to be running in the cloud. But it seems like you could probably get a smaller, focused model that runs locally and is also high quality. In other words, I want the programming expert portion of a 2T model that is maybe 35B or 70B parameters or whatever but it’s running locally (and yes, I know you don’t just carve out an expert from a larger model, but conceptually that’s what I’m after).
  - fennecfoxy 2mo ago
    Yeah, I always thought the future of this stuff would be hot-pluggable MoE modules or LORAs that are able to be downloaded and applied/used at will like how skills have become a thing. Like, atm most architectures seem limited by a single context and fixed architecture with no hot loading. Especially for robotics, being able to load/unload various specialised skills on limited mobile hardware will (I hope) definitely become a thing.