3 ms·

But did it work? This is the sticking point with me now. I've seen slides, architecture diagrams, job descriptions, roadmaps and other docs now from about a doz

But did it work? This is the sticking point with me now. I've seen slides, architecture diagrams, job descriptions, roadmaps and other docs now from about a dozen different companies doing AI Agent projects. And while it's completely feasible to build the systems they're describing, what I have not seen yet is evidence of any of them working. When you press them on this, they have all sorts of ideas like a judge LLM that takes the outputs, comes up with modified SOPs and feeds those into the prompts of the mixture-of-experts LLMs. But I don't think that works, I've tried closing that loop and all I got was LLMs flailing around.

krackers 11mo ago
I think long-term just having a single LLM responsible for everything will win out compared to brittle and complex subagent hierarchie. Most use of "subagents" today are just workarounds for LLM limitations: lack of instruction following, context length, non- determinism, or "hallucinations". All of these are things that will need to be solved long-term in the model itself though, at least if the AI bubble needs to be kept alive. And solving those things would in fact materially improve all sorts of benchmarks, so there's an incentive for frontier labs to do it. I think this is why you have the back-forth pattern that GP mentioned. You start with a single model doing everything. Then you find all sorts of gaps that you start to plug ad-hoc, and decide that breaking it into subagents might help fix things. This works for a while but then you realize that you lose out on the flexibility of a single-model having access to the entire context, so you starting trying to improve communication between subagents. But then a new model drops that fixes a lot of the things you originally had to workaround, so you go back to a single-model setup. Rinse and repeat. It's a great VC- bubble funded employment program though.
- theptip 11mo ago
  The general pattern seems to be that LLM+scaffolding performs better than LLM. In 6 months time a new model will incorporate 80% of your scaffolding, but also will enable new capabilities with a new layer of scaffolding. I suspect the model that doesn’t need scaffolding is simply ASI, as in, the AI can build its own scaffolding (aka recursive self-improvement), and build it better than a human can. Until that point, the job is going to remain figuring out how to eval your frontier task, scaffold the models’ weaknesses, and codify/absorb more domain knowledge that’s not in the training set. You are talking about context management stuff here, the solution will be something like a proper memory subsystem, maybe some architectural tweaks to integrate it. There are more obvious gaps beyond that which we will have to scaffold and then solve in turn. Another way of thinking about this is just that scaffolding is a much faster way of iterating on solutions than pre-training, or even post-training, and so it will continue to be a valuable way of advancing capabilities.
eddythompson80 11mo ago
It hasn’t really worked so far. Pretty much exactly what you’ve described. I don’t even really work on that team, but “a judge LLM” low-key triggered me just because of how much I’ve been hearing it over the last couple of months. I think the reason of the recent pivot is to “keep the human in the loop” more. The current thinking is they tried to remove the human too much and were getting bad results. So now they just want to make the interaction faster and let the human be more involved like how we (developers) use Claude code or copilot by checking every interaction and nudging it towards the right/desired answer. I got the sense that management isn’t taking it well though. Just this Friday they gave a demo of the new POC where the LLM is just suggesting things and frequently asking for permissions and where to go next and expecting the user to interact with it a lot more than the one-shot approach before (which I do think is likely to yield better results tbh) but the main reaction was “this seems like a massive step backward”

krackers 11mo ago

I think long-term just having a single LLM responsible for everything will win out compared to brittle and complex subagent hierarchie. Most use of "subagents" today are just workarounds for LLM limitations: lack of instruction following, context length, non- determinism, or "hallucinations". All of these are things that will need to be solved long-term in the model itself though, at least if the AI bubble needs to be kept alive. And solving those things would in fact materially improve all sorts of benchmarks, so there's an incentive for frontier labs to do it. I think this is why you have the back-forth pattern that GP mentioned. You start with a single model doing everything. Then you find all sorts of gaps that you start to plug ad-hoc, and decide that breaking it into subagents might help fix things. This works for a while but then you realize that you lose out on the flexibility of a single-model having access to the entire context, so you starting trying to improve communication between subagents. But then a new model drops that fixes a lot of the things you originally had to workaround, so you go back to a single-model setup. Rinse and repeat. It's a great VC- bubble funded employment program though.

theptip 11mo ago
The general pattern seems to be that LLM+scaffolding performs better than LLM. In 6 months time a new model will incorporate 80% of your scaffolding, but also will enable new capabilities with a new layer of scaffolding. I suspect the model that doesn’t need scaffolding is simply ASI, as in, the AI can build its own scaffolding (aka recursive self-improvement), and build it better than a human can. Until that point, the job is going to remain figuring out how to eval your frontier task, scaffold the models’ weaknesses, and codify/absorb more domain knowledge that’s not in the training set. You are talking about context management stuff here, the solution will be something like a proper memory subsystem, maybe some architectural tweaks to integrate it. There are more obvious gaps beyond that which we will have to scaffold and then solve in turn. Another way of thinking about this is just that scaffolding is a much faster way of iterating on solutions than pre-training, or even post-training, and so it will continue to be a valuable way of advancing capabilities.

eddythompson80 11mo ago

It hasn’t really worked so far. Pretty much exactly what you’ve described. I don’t even really work on that team, but “a judge LLM” low-key triggered me just because of how much I’ve been hearing it over the last couple of months. I think the reason of the recent pivot is to “keep the human in the loop” more. The current thinking is they tried to remove the human too much and were getting bad results. So now they just want to make the interaction faster and let the human be more involved like how we (developers) use Claude code or copilot by checking every interaction and nudging it towards the right/desired answer. I got the sense that management isn’t taking it well though. Just this Friday they gave a demo of the new POC where the LLM is just suggesting things and frequently asking for permissions and where to go next and expecting the user to interact with it a lot more than the one-shot approach before (which I do think is likely to yield better results tbh) but the main reaction was “this seems like a massive step backward”