3 ms·

As someone who's built a project in this space, this is incredibly unreliable. Subagents don't get a full system prompt (including stuff like CLAUDE.md directio

As someone who's built a project in this space, this is incredibly unreliable. Subagents don't get a full system prompt (including stuff like CLAUDE.md directions) so they are flying very blind in your projects, and as such will tend to get derailed by their lack of knowledge of a project and veer into mock solutions and "let me just make a simpler solution that demonstrates X." I advise people to only use subagents for stuff that is very compartmentalized because they're hard to monitor and prone to failure with complex codebases where agents live and die by project knowledge curated in files like CLAUDE.md. If your main Claude instance doesn't give a good handoff to a subagent, or a subagent doesn't give a good handback to the main Claude, shit will go sideways fast. Also, don't lean on agents for refactoring. Their ability to refactor a codebase goes in the toilet pretty quickly.

theshrike79 1y ago
I don't use subagents to do things, they're best for analysing things. Like "evaluate the test coverage" or "check if the project follows the style guide". This way the "main" context only gets the report and doesn't waste space on massive test outputs or reading multiple files.
- olivermuty 1y ago
  This is only a problem if an agent is made in a lazy way (all of them). Chat completion sends the full prompt history on every call. I am working on my own coding agent and seeing massive improvements by rewriting history using either a smaller model or a freestanding call to the main one. It really mitigates context poisoning.
  - mattmanser 1y ago
    Everyone complains that when you compact the context, Claude tends to get stupid Which as far as I understand it is summarizing the context with a smaller model. Am I misunderstanding you, as the practical experience of most people seem to contradict your results.
    - NitpickLawyer 1y ago
      One key insight I have from having worked on this from the early stages of LLMs (before chatgpt came out) is that the current crop of LLM clients or "agentic clients" don't log/write/keep track of success over time. It's more of a "shoot and forget" environment right now, and that's why a lot of people are getting vastly different results. Hell, even week to week on the same tasks you get different results (see the recent claude getting dumber drama). Once we start to see that kind of self feedback going in next iterations (w/ possible training runs between sessions, "dreaming" stage from og RL, distilling a session, grabbing key insights, storing them, surfacing them at next inference, etc) then we'll see true progress in this space. The problem is that a lot of people work on these things in silos. The industry is much more geared towards quick returns now, having to show something now, rather than building strong fo0undations based on real data. Kind of an analogy to early linux dev. We need our own Linus, it would seem :)
      - troupo 1y ago
        > don't log/write/keep track of success over time. How do you define success of a model's run?
        NitpickLawyer 1y ago
        Lots of ways. You could do binary thumbs up/down. You could do a feedback session. You could look at signals like "acceptance rate" (for a pr?) or "how many feedback messages did the user send in this session", and so on. My point was more on tracking these signals over time. And using them to improve the client, not just the model (most model providers probably track this already).
        troupo 1y ago
        Ah. Yes! My somewhat terse/bitter question was because yesterday Claude would continue claim to have created a "production-ready" solution which was completely entirely wrong. I would've loved to have the feedback loop you describe
      - 1y ago
  - CuriouslyC 1y ago
    There's a large body of research on context pruning/rewriting (I know because I'm knee deep in benchmarks in release prep for my context compiler), definitely don't ad hoc this.
    - spariev 1y ago
      Care to give some pointers on what to look at? Looks like I will be doing something similar soon so that would be much appreciated
      - CuriouslyC 1y ago
        Just ask chat gpt about state of the art in context pruning and other methods to optimize the context being provided to a LLM, it's a good research helper. The right mental model is that it's basically like RAG in reverse, instead of trying to select and rank from a data set, you're trying to select and rank from context given a budget.
  - ixsploit 1y ago
    I do something similar and I have the best results of not having a history at all, but setting the context new with every invokation.
zarzavat 1y ago
> Their ability to refactor a codebase goes in the toilet pretty quickly. Very much this. I tried to get Claude to move some code from one file to another. Some of the code went missing. Some of it was modified along the way. Humans have strategies for refactoring, e.g. "I'm going to start from the top of the file and Cut code that needs to be moved and Paste it in the new location". LLM don't have a clipboard (yet!) so they can't do this. Claude can only reliably do this refactoring if it can keep the start and end files in context. This was a large file, so it got lost. Even then it needs direct supervision.
- diggan 1y ago
  > Humans have strategies for refactoring, e.g. "I'm going to start from the top of the file and Cut code that needs to be moved and Paste it in the new location". LLM don't have a clipboard (yet!) so they can't do this. For my own agent I have a `move_file` and `copy_file` tool with two args each, that at least GPT-OSS seems to be able to use whenever it suits, like for moving stuff around. I've seen it use it as part of refactoring as well, moving a file to one location, copying that to another, the trim both of them but different trims, seems to have worked OK. If the agent has access to `exec_shell` or similar, I'm sure you could add `Use mv and cp if you need to move or copy files` to the system prompt to get it to use that instead, probably would work in Claude Code as well.
- lupire 1y ago
  Remember 20 years ago when Eclipse could move a function by manipulating the AST and following references to adjust imports and callers, and it it didn't lose any code?
  - Yeroc 1y ago
    I think it's likely that these agent-based development will inevitably add more imperative tools to their arsenal to lower cost, improve speed and accuracy.
  - mleo 1y ago
    It’s still early days for these agents. There isn’t any reason the agents won’t build or understand AST in the future to more quickly refactor.
prash2488 1y ago
Totally agreed, tried agents for a lot of stuff (I started creating a team of agents, architect, frontend coder, backend coder and QA). Spent around 50 USD on a failed project, context contaminated and the project eventually had to be re-written. Then I moved some parts in rules, some parts in slash commands and then I got much better results. The subagents are like a freelance contractors (I know, I have been one very recently) Good when they need little handoff (Not possible in realtime), little overseeing and their results are a good advice not an action. They don't know what you are doing, they don't care what you do with the info they produce. They just do the work for you while you do something else, or wait for them to produce independent results. They come and go with little knowledge of existing functionalities, but good on their own. Here are 3 agents I still keep and one I am working on. 1: Scaffolding: Now I create (and sometimes destroy) a lot of new projects. I use a scaffolding agents when I am trying something new. They start with fresh one line instruction to what to scaffold (e.g. a New docker container with Hono and Postgres connection, or a new cloudflare worker which will connect to R2, D1 and AI Gateway, or a AWS Serverless API Gateway with SQS that does this that and that), where to deploy. At the end of the day they setup the project with structure, create a Github Repo and commit it for me. I will take it forward from them 2: Triage: When I face some issues which is not obvious from reading code alone, I give them the place, some logs and the agent will use whatever available (including the DB Data) to make a best guess of why this issue happens. I often found out they work best when they are not biased by recent work 3: Pre-Release Check QA: Now this QA will test the entire system (Essentially calling all integration and end-to-end test suite to make sure this product doesn't break anything existing. Now I am adding a functionality to let them see original business requirement and see if the code satisfies it or not. I want this agent to be my advisor to help me decide if something goes to release pipeline or not. 4: Web search (Experimental) Sometimes, some search are too costly for existing token, and we only need the end result, not what they search and those 10 pages it found out...
stingraycharles 1y ago
It was my understanding that the subagents have the same system prompt. How do you know that they don’t follow CLAUDE.md directions? I’ve been using subagents since they were introduced and it has been a great way to manage context size / pollution.
- CuriouslyC 1y ago
  A few youtubers have done deep dives on this, monitoring claude traffic through a proxy. Subagents don't get the system prompt or anything else, they get their subagent prompt and whatever handoff the main agent gives them. I was on the subagent hype train myself for a while but as my codebases have scaled (I have a couple of codebases up to almost 400k now) subagents have become a lot more error prone and now I cringe when I see them for anything challenging and immediately escape out. They seem to work great with more greenfield projects though.
  - wild_egg 1y ago
    I have a bunch of homegrown CLI tools in my $PATH that are only described in the CLAUDE.md file. My subagents use these tools perfectly as if they have full instructions on their use but no such instructions are in the subagent prompts. This should not be possible if they don't have CLAUDE.md in their context. My main agent prompt always has a complete ban on the main agent doing any work themselves. All work is done by subagents which they coordinate. I've been doing this for 2-3 months now on projects upwards of 200k lines and the results have been incredible. I'm very confused how so many of us can have such completely different experiences with these tools.
    - stingraycharles 1y ago
      Yes for me the same, I specify using “direnv exec .” as a prefix to every command and the subagents follow this without issue. On the Claude Code Reddit communities there’s basically a constant outrage about CC’s performance over the past few months, it seems different people have vastly different experiences with these tools. There appears to be a lot of anecdotal evidence everywhere, and not enough hard facts. Anthropic’s lack of transparency how everything works and interacts is at least a factor in this.
quijoteuniv 1y ago
My experience so far, after trying to keep CC on track with different strategies is that it will more or less end up on the same ditch sooner or later. Even though i had defined agents, workflows, etc. now i just let it interact with github issues and the quality is pretty much the same

theshrike79 1y ago

I don't use subagents to do things, they're best for analysing things. Like "evaluate the test coverage" or "check if the project follows the style guide". This way the "main" context only gets the report and doesn't waste space on massive test outputs or reading multiple files.

olivermuty 1y ago
This is only a problem if an agent is made in a lazy way (all of them). Chat completion sends the full prompt history on every call. I am working on my own coding agent and seeing massive improvements by rewriting history using either a smaller model or a freestanding call to the main one. It really mitigates context poisoning.
- mattmanser 1y ago
  Everyone complains that when you compact the context, Claude tends to get stupid Which as far as I understand it is summarizing the context with a smaller model. Am I misunderstanding you, as the practical experience of most people seem to contradict your results.
  - NitpickLawyer 1y ago
    One key insight I have from having worked on this from the early stages of LLMs (before chatgpt came out) is that the current crop of LLM clients or "agentic clients" don't log/write/keep track of success over time. It's more of a "shoot and forget" environment right now, and that's why a lot of people are getting vastly different results. Hell, even week to week on the same tasks you get different results (see the recent claude getting dumber drama). Once we start to see that kind of self feedback going in next iterations (w/ possible training runs between sessions, "dreaming" stage from og RL, distilling a session, grabbing key insights, storing them, surfacing them at next inference, etc) then we'll see true progress in this space. The problem is that a lot of people work on these things in silos. The industry is much more geared towards quick returns now, having to show something now, rather than building strong fo0undations based on real data. Kind of an analogy to early linux dev. We need our own Linus, it would seem :)
    - troupo 1y ago
      > don't log/write/keep track of success over time. How do you define success of a model's run?
      - NitpickLawyer 1y ago
        Lots of ways. You could do binary thumbs up/down. You could do a feedback session. You could look at signals like "acceptance rate" (for a pr?) or "how many feedback messages did the user send in this session", and so on. My point was more on tracking these signals over time. And using them to improve the client, not just the model (most model providers probably track this already).
        troupo 1y ago
        Ah. Yes! My somewhat terse/bitter question was because yesterday Claude would continue claim to have created a "production-ready" solution which was completely entirely wrong. I would've loved to have the feedback loop you describe
    - 1y ago
- CuriouslyC 1y ago
  There's a large body of research on context pruning/rewriting (I know because I'm knee deep in benchmarks in release prep for my context compiler), definitely don't ad hoc this.
  - spariev 1y ago
    Care to give some pointers on what to look at? Looks like I will be doing something similar soon so that would be much appreciated
    - CuriouslyC 1y ago
      Just ask chat gpt about state of the art in context pruning and other methods to optimize the context being provided to a LLM, it's a good research helper. The right mental model is that it's basically like RAG in reverse, instead of trying to select and rank from a data set, you're trying to select and rank from context given a budget.
- ixsploit 1y ago
  I do something similar and I have the best results of not having a history at all, but setting the context new with every invokation.

zarzavat 1y ago

> Their ability to refactor a codebase goes in the toilet pretty quickly. Very much this. I tried to get Claude to move some code from one file to another. Some of the code went missing. Some of it was modified along the way. Humans have strategies for refactoring, e.g. "I'm going to start from the top of the file and Cut code that needs to be moved and Paste it in the new location". LLM don't have a clipboard (yet!) so they can't do this. Claude can only reliably do this refactoring if it can keep the start and end files in context. This was a large file, so it got lost. Even then it needs direct supervision.

diggan 1y ago
> Humans have strategies for refactoring, e.g. "I'm going to start from the top of the file and Cut code that needs to be moved and Paste it in the new location". LLM don't have a clipboard (yet!) so they can't do this. For my own agent I have a `move_file` and `copy_file` tool with two args each, that at least GPT-OSS seems to be able to use whenever it suits, like for moving stuff around. I've seen it use it as part of refactoring as well, moving a file to one location, copying that to another, the trim both of them but different trims, seems to have worked OK. If the agent has access to `exec_shell` or similar, I'm sure you could add `Use mv and cp if you need to move or copy files` to the system prompt to get it to use that instead, probably would work in Claude Code as well.
lupire 1y ago
Remember 20 years ago when Eclipse could move a function by manipulating the AST and following references to adjust imports and callers, and it it didn't lose any code?
- Yeroc 1y ago
  I think it's likely that these agent-based development will inevitably add more imperative tools to their arsenal to lower cost, improve speed and accuracy.
- mleo 1y ago
  It’s still early days for these agents. There isn’t any reason the agents won’t build or understand AST in the future to more quickly refactor.

prash2488 1y ago

Totally agreed, tried agents for a lot of stuff (I started creating a team of agents, architect, frontend coder, backend coder and QA). Spent around 50 USD on a failed project, context contaminated and the project eventually had to be re-written. Then I moved some parts in rules, some parts in slash commands and then I got much better results. The subagents are like a freelance contractors (I know, I have been one very recently) Good when they need little handoff (Not possible in realtime), little overseeing and their results are a good advice not an action. They don't know what you are doing, they don't care what you do with the info they produce. They just do the work for you while you do something else, or wait for them to produce independent results. They come and go with little knowledge of existing functionalities, but good on their own. Here are 3 agents I still keep and one I am working on. 1: Scaffolding: Now I create (and sometimes destroy) a lot of new projects. I use a scaffolding agents when I am trying something new. They start with fresh one line instruction to what to scaffold (e.g. a New docker container with Hono and Postgres connection, or a new cloudflare worker which will connect to R2, D1 and AI Gateway, or a AWS Serverless API Gateway with SQS that does this that and that), where to deploy. At the end of the day they setup the project with structure, create a Github Repo and commit it for me. I will take it forward from them 2: Triage: When I face some issues which is not obvious from reading code alone, I give them the place, some logs and the agent will use whatever available (including the DB Data) to make a best guess of why this issue happens. I often found out they work best when they are not biased by recent work 3: Pre-Release Check QA: Now this QA will test the entire system (Essentially calling all integration and end-to-end test suite to make sure this product doesn't break anything existing. Now I am adding a functionality to let them see original business requirement and see if the code satisfies it or not. I want this agent to be my advisor to help me decide if something goes to release pipeline or not. 4: Web search (Experimental) Sometimes, some search are too costly for existing token, and we only need the end result, not what they search and those 10 pages it found out...

stingraycharles 1y ago

It was my understanding that the subagents have the same system prompt. How do you know that they don’t follow CLAUDE.md directions? I’ve been using subagents since they were introduced and it has been a great way to manage context size / pollution.

CuriouslyC 1y ago
A few youtubers have done deep dives on this, monitoring claude traffic through a proxy. Subagents don't get the system prompt or anything else, they get their subagent prompt and whatever handoff the main agent gives them. I was on the subagent hype train myself for a while but as my codebases have scaled (I have a couple of codebases up to almost 400k now) subagents have become a lot more error prone and now I cringe when I see them for anything challenging and immediately escape out. They seem to work great with more greenfield projects though.
- wild_egg 1y ago
  I have a bunch of homegrown CLI tools in my $PATH that are only described in the CLAUDE.md file. My subagents use these tools perfectly as if they have full instructions on their use but no such instructions are in the subagent prompts. This should not be possible if they don't have CLAUDE.md in their context. My main agent prompt always has a complete ban on the main agent doing any work themselves. All work is done by subagents which they coordinate. I've been doing this for 2-3 months now on projects upwards of 200k lines and the results have been incredible. I'm very confused how so many of us can have such completely different experiences with these tools.
  - stingraycharles 1y ago
    Yes for me the same, I specify using “direnv exec .” as a prefix to every command and the subagents follow this without issue. On the Claude Code Reddit communities there’s basically a constant outrage about CC’s performance over the past few months, it seems different people have vastly different experiences with these tools. There appears to be a lot of anecdotal evidence everywhere, and not enough hard facts. Anthropic’s lack of transparency how everything works and interacts is at least a factor in this.

quijoteuniv 1y ago

My experience so far, after trying to keep CC on track with different strategies is that it will more or less end up on the same ditch sooner or later. Even though i had defined agents, workflows, etc. now i just let it interact with github issues and the quality is pretty much the same