5 ms·

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They def

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outcome is a well-named test that fails, because that represents a gap between spec and implementation that didn't previously have a tripwire. Their proxy metric is "number of meaningful new failures introduced" and the barrier prevents them from writing tests pre-adapted to pass. Green Team (Implementers), write implementation to pass tests without seeing the test code directly. They only see test results (pass/fail) and the spec. Rewarded by turning red tests green. Straightforward, but the barrier makes the reward structure honest. Without it, Green could satisfy the reward trivially by reading assertions and hard-coding. With it, Green has to actually close the gap between spec intent and code behavior, using error messages as noisy gradient signal rather than exact targets. Their reward is "tests that were failing now pass," and the only reliable strategy to get there is faithful implementation. Refactor Team, improve code quality without changing behavior. They can see implementation but are constrained by tests passing. Rewarded by nothing changing (pretty unusual in this regard). Reward is that all tests stay green while code quality metrics improve. They're optimizing a secondary objective (readability, simplicity, modularity, etc.) under a hard constraint (behavioral equivalence). The spec barrier ensures they can't redefine "improvement" to include feature work. If you have any code quality tools, it makes sense to give the necessary skills to use them to this team. It's worth being honest about the limits. The spec itself is a shared artifact visible to both Red and Green, so if the spec is vague, both agents might converge on the same wrong interpretation, and the tests will pass for the wrong reason. The Coordinator (your main claude/codex/whatever instance) mitigates this by watching for suspiciously easy green passes (just tell it) and probing the spec for ambiguity, but it's not a complete defense.

skybrian 7mo ago
How do you define visibility rules? Is that possible for subagents?
- egeozcan 7mo ago
  AFAIK Claude doesn't support it, but if you're willing to go the extra mile, you can get creative with some bash script: https://pastebin.com/raw/m9YQ8MyS https://pastebin.com/raw/m9YQ8MyS (generated this a second ago - just to get the point across ) To be clear, I don't do this. I never saw an agent cheat by peeking or something. I really did look through their logs. I'd be very interested to see claude code and other tools support this pattern when dispatching agents to be really sure.
  - skybrian 7mo ago
    So what do you do? Do you define roles somewhere and tell the agent to assign these roles to subagents?
    - ssk42 7mo ago
      Fun to see you not on tildes. Setting up a clean room is one of the only ways to do Evals on agentic harnesses. Especially prevalent with Windsurf which doesn’t have an easy CLI start. So how? The easiest answer when allowed is docker. Literally new image per prompt. There’s also flags with Claude to not use memory and from there you can use -p to have it just be like a normal cli tool. Windsurf requires manual effort of starting it up in a new dir.
      - skybrian 7mo ago
        Sounds interesting, but I'm not quite getting the relevance for people writing code with an agent. Should I be doing evals?
        ssk42 7mo ago
        Well I mean yes. I think people ought be aware for how the harnesses compare for their stacks. But clean room applies for this RGR situation too
        novaleaf 7mo ago
tomtom1337 7mo ago
This is very interesting, but like sibling comments, I'm very curious as to how you run this in practice. Do you just tell Claude/Copilot to do what you describe? And do you have any prompts to share?
- throwaway7783 7mo ago
  You don't need most of this. Prompts are also normally what you would say to another engineer. * There is a lot of duplication between A & B. Refactor this. * Look at ticket X and give me a root cause * Add support for three new types of credentials - Basic Auth, Bearer Token and OAuth Client Creds Claude.md has stuff like "Here's how you run the frontend. here's how u run backend. This module support frontend. That module is batch jobs. Always start commit messages with ticket number. Always run compile at the top level. When you make code changes, always add tests" etc etc
- Exoristos 7mo ago
  They never do.
- kolinko 7mo ago
  Not Claude/Copilot. Claude.
- duskdozer 7mo ago
  Sign up for your Claude Max (TM) subscription and have Claude set you up
xienze 7mo ago
This seems like a tremendous amount of planning, babysitting, verification, and token cost just to avoid writing code and tests yourself.
- habinero 7mo ago
  It's assigning yourself the literal worst parts of the job - writing specs, docs, tests and reading someone else's code.
  - zarzavat 7mo ago
    There's a real disconnect. I was talking to a junior developer and they were telling me how Claude is so much smarter than them and they feel inferior. I couldn't relate. From my perspective as a senior, Claude is dumb as bricks. Though useful nonetheless. I believe that if you're substantially below Claude's level then you just trust whatever it says. The only variables you control are how much money you spend, how much markdown you can produce, and how you arrange your agents. But I don't understand how the juniors on HN have so much money to throw at this technology.
    - godelski 7mo ago
      > I was talking to a junior developer and they were telling me how Claude is so much smarter than them and they feel inferior. Every time I talk to a wizard I feel like they're so much smarter than me and it makes me feel inferior. So I take that feeling and use it to drive me to become a wizard like them. I've generally found that wizards are very happy to take on apprentices. I'm not trying to call Claude a wizard (I have similar feelings to you), but more that I don't understand that junior's take. We all feel dumb. All but time. Even the wizards! But it's that feeling that drives you to better yourself and it's what turns you into a wizard. Honestly so much of what I hear from the "AI does all my coding" crowd just sounds very junior. It's just the same like how a year or two ago they were saying "it does the repetitive stuff". Isn't that what functions, libraries, functors, templates, and other abstractions are for? It feels like we're back to that laughable productivity metric of lines of code or number of commits. I don't know why we love our cargo cults. It seems people are putting so much effort into their cargo cults that they could have invented a real airplane by now.
w4yai 7mo ago
You guys are describing wonderful things, but I've yet to see any implementation. I tried coding my own agents, yet the results were disappointing. What kind of setup do you use ? Can you share ? How much does it cost ?
- dworks 7mo ago
  rlm-workflow does all that TDD for you: https://skills.sh/doubleuuser/rlm-workflow/rlm-workflow https://skills.sh/doubleuuser/rlm-workflow/rlm-workflow (I built it)
  - _ink_ 7mo ago
    Thanks for sharing. What does RLM stand for? Any idea why the socket security test fails?
    - stavros 7mo ago
      Recursive language models: https://github.com/doubleuuser/rlm-workflow https://github.com/doubleuuser/rlm-workflow
  - cheema33 7mo ago
    Why make powershell a requirement? I like powershell, but Python is very common and already installed on many dev systems.
    - dworks 7mo ago
      Sorry about that. Let me push an update.
- canadiantim 7mo ago
  Check out Mike Pocock’s work, he’s done excellent work writing about red green refactor and has a GitHub repo for his skills. Read and take what you need from his tdd skill and incorporate it into your own tdd skill tailored for your project.
tnecio 7mo ago
How do you make sure Red Team doesn't just write subtly broken tests?
esperent 7mo ago
Someone directly down from you suggested looking up Mike Postock's TDD skill, so I did: https://github.com/mattpocock/skills/blob/main/tdd%2FSKILL.md https://github.com/mattpocock/skills/blob/main/tdd%2FSKILL.m... Everything below quoted from that skill, and serves as a much better rebuttal than I had started writing: DO NOT write all tests first, then all implementation. This is "horizontal slicing" - treating RED as "write all tests" and GREEN as "write all code." This produces crap tests: Tests written in bulk test imagined behavior, not actual behavior You end up testing the shape of things (data structures, function signatures) rather than user-facing behavior Tests become insensitive to real changes - they pass when behavior breaks, fail when behavior is fine You outrun your headlights, committing to test structure before understanding the implementation Correct approach: Vertical slices via tracer bullets. One test → one implementation → repeat. Each test responds to what you learned from the previous cycle. Because you just wrote the code, you know exactly what behavior matters and how to verify it.
- fsckboy 7mo ago
  >One test → one implementation → repeat. >Because you just wrote the code, you know exactly what behavior matters and how to verify it. what you go on to describe is One implementation → one test → repeat.
seer 7mo ago
This seems quite amazing really, thanks for sharing What is the scope of projects / features you’ve seen this be successful at? Do you have a step before where an agent verifies that your new feature spec is not contradictory, ambiguous etc. Maybe as reviewed with regards to all the current feature sets? Do you make this a cycle per step - by breaking down the feature to small implementable and verifiable sub-features and coding them in sequence, or do you tell it to write all the tests first and then have at it with implementation and refactoring? Why not refactor-red-green-refactor cycle? E.g. a lot of the time it is worth refactoring the existing code first, to make a new implementation easier, is it worth encoding this into the harness?
- aray07 7mo ago
  I do it per feature, not per step. Write the AC for the whole feature upfront, then the agent builds against it. I haven't added a spec-validation step before coding but that's a good idea. Catching ambiguity in the spec before the agent runs with it would save a lot of rework
devinplatt 7mo ago
I'm curious how this works if the green team writes an implementation that makes a network call like an RPC. Red team might not anticipate this if the spec does detail every expected RPC (which seems unreasonable: this could vary based on implementation). But a unit test would need mocks. Is green team allowed to suggest mocks to add to the test? (Even if they can't read the tests themselves?) This also seems gamaeable though (e.g. mock the entire implementation). Unless another agent makes a judgement call on the reasonability of the mock (though that starts to feel like code review more generally). Maybe record/replay tests could work? But there are drawbacks in the added complexity.
- codethief 7mo ago
  I think the solution here is: Don't mock and inject dependencies explicitly, as function parameters / monads / algebraic effects. Make side effects part of the spec/interface.
bcrosby95 7mo ago
Seems like red team is incentivized to write tests that violate the spec since you're rewarding failed tests.

skybrian 7mo ago

How do you define visibility rules? Is that possible for subagents?

egeozcan 7mo ago
AFAIK Claude doesn't support it, but if you're willing to go the extra mile, you can get creative with some bash script: https://pastebin.com/raw/m9YQ8MyS https://pastebin.com/raw/m9YQ8MyS (generated this a second ago - just to get the point across ) To be clear, I don't do this. I never saw an agent cheat by peeking or something. I really did look through their logs. I'd be very interested to see claude code and other tools support this pattern when dispatching agents to be really sure.
- skybrian 7mo ago
  So what do you do? Do you define roles somewhere and tell the agent to assign these roles to subagents?
  - ssk42 7mo ago
    Fun to see you not on tildes. Setting up a clean room is one of the only ways to do Evals on agentic harnesses. Especially prevalent with Windsurf which doesn’t have an easy CLI start. So how? The easiest answer when allowed is docker. Literally new image per prompt. There’s also flags with Claude to not use memory and from there you can use -p to have it just be like a normal cli tool. Windsurf requires manual effort of starting it up in a new dir.
    - skybrian 7mo ago
      Sounds interesting, but I'm not quite getting the relevance for people writing code with an agent. Should I be doing evals?
      - ssk42 7mo ago
        Well I mean yes. I think people ought be aware for how the harnesses compare for their stacks. But clean room applies for this RGR situation too
      - novaleaf 7mo ago

tomtom1337 7mo ago

This is very interesting, but like sibling comments, I'm very curious as to how you run this in practice. Do you just tell Claude/Copilot to do what you describe? And do you have any prompts to share?

throwaway7783 7mo ago
You don't need most of this. Prompts are also normally what you would say to another engineer. * There is a lot of duplication between A & B. Refactor this. * Look at ticket X and give me a root cause * Add support for three new types of credentials - Basic Auth, Bearer Token and OAuth Client Creds Claude.md has stuff like "Here's how you run the frontend. here's how u run backend. This module support frontend. That module is batch jobs. Always start commit messages with ticket number. Always run compile at the top level. When you make code changes, always add tests" etc etc
Exoristos 7mo ago
They never do.
kolinko 7mo ago
Not Claude/Copilot. Claude.
duskdozer 7mo ago
Sign up for your Claude Max (TM) subscription and have Claude set you up

xienze 7mo ago

This seems like a tremendous amount of planning, babysitting, verification, and token cost just to avoid writing code and tests yourself.

habinero 7mo ago
It's assigning yourself the literal worst parts of the job - writing specs, docs, tests and reading someone else's code.
- zarzavat 7mo ago
  There's a real disconnect. I was talking to a junior developer and they were telling me how Claude is so much smarter than them and they feel inferior. I couldn't relate. From my perspective as a senior, Claude is dumb as bricks. Though useful nonetheless. I believe that if you're substantially below Claude's level then you just trust whatever it says. The only variables you control are how much money you spend, how much markdown you can produce, and how you arrange your agents. But I don't understand how the juniors on HN have so much money to throw at this technology.
  - godelski 7mo ago
    > I was talking to a junior developer and they were telling me how Claude is so much smarter than them and they feel inferior. Every time I talk to a wizard I feel like they're so much smarter than me and it makes me feel inferior. So I take that feeling and use it to drive me to become a wizard like them. I've generally found that wizards are very happy to take on apprentices. I'm not trying to call Claude a wizard (I have similar feelings to you), but more that I don't understand that junior's take. We all feel dumb. All but time. Even the wizards! But it's that feeling that drives you to better yourself and it's what turns you into a wizard. Honestly so much of what I hear from the "AI does all my coding" crowd just sounds very junior. It's just the same like how a year or two ago they were saying "it does the repetitive stuff". Isn't that what functions, libraries, functors, templates, and other abstractions are for? It feels like we're back to that laughable productivity metric of lines of code or number of commits. I don't know why we love our cargo cults. It seems people are putting so much effort into their cargo cults that they could have invented a real airplane by now.

w4yai 7mo ago

You guys are describing wonderful things, but I've yet to see any implementation. I tried coding my own agents, yet the results were disappointing. What kind of setup do you use ? Can you share ? How much does it cost ?

dworks 7mo ago
rlm-workflow does all that TDD for you: https://skills.sh/doubleuuser/rlm-workflow/rlm-workflow https://skills.sh/doubleuuser/rlm-workflow/rlm-workflow (I built it)
- _ink_ 7mo ago
  Thanks for sharing. What does RLM stand for? Any idea why the socket security test fails?
  - stavros 7mo ago
    Recursive language models: https://github.com/doubleuuser/rlm-workflow https://github.com/doubleuuser/rlm-workflow
- cheema33 7mo ago
  Why make powershell a requirement? I like powershell, but Python is very common and already installed on many dev systems.
  - dworks 7mo ago
    Sorry about that. Let me push an update.
canadiantim 7mo ago
Check out Mike Pocock’s work, he’s done excellent work writing about red green refactor and has a GitHub repo for his skills. Read and take what you need from his tdd skill and incorporate it into your own tdd skill tailored for your project.

tnecio 7mo ago

How do you make sure Red Team doesn't just write subtly broken tests?

esperent 7mo ago

Someone directly down from you suggested looking up Mike Postock's TDD skill, so I did: https://github.com/mattpocock/skills/blob/main/tdd%2FSKILL.md https://github.com/mattpocock/skills/blob/main/tdd%2FSKILL.m... Everything below quoted from that skill, and serves as a much better rebuttal than I had started writing: DO NOT write all tests first, then all implementation. This is "horizontal slicing" - treating RED as "write all tests" and GREEN as "write all code." This produces crap tests: Tests written in bulk test imagined behavior, not actual behavior You end up testing the shape of things (data structures, function signatures) rather than user-facing behavior Tests become insensitive to real changes - they pass when behavior breaks, fail when behavior is fine You outrun your headlights, committing to test structure before understanding the implementation Correct approach: Vertical slices via tracer bullets. One test → one implementation → repeat. Each test responds to what you learned from the previous cycle. Because you just wrote the code, you know exactly what behavior matters and how to verify it.

fsckboy 7mo ago
>One test → one implementation → repeat. >Because you just wrote the code, you know exactly what behavior matters and how to verify it. what you go on to describe is One implementation → one test → repeat.

seer 7mo ago

This seems quite amazing really, thanks for sharing What is the scope of projects / features you’ve seen this be successful at? Do you have a step before where an agent verifies that your new feature spec is not contradictory, ambiguous etc. Maybe as reviewed with regards to all the current feature sets? Do you make this a cycle per step - by breaking down the feature to small implementable and verifiable sub-features and coding them in sequence, or do you tell it to write all the tests first and then have at it with implementation and refactoring? Why not refactor-red-green-refactor cycle? E.g. a lot of the time it is worth refactoring the existing code first, to make a new implementation easier, is it worth encoding this into the harness?

aray07 7mo ago
I do it per feature, not per step. Write the AC for the whole feature upfront, then the agent builds against it. I haven't added a spec-validation step before coding but that's a good idea. Catching ambiguity in the spec before the agent runs with it would save a lot of rework

devinplatt 7mo ago

I'm curious how this works if the green team writes an implementation that makes a network call like an RPC. Red team might not anticipate this if the spec does detail every expected RPC (which seems unreasonable: this could vary based on implementation). But a unit test would need mocks. Is green team allowed to suggest mocks to add to the test? (Even if they can't read the tests themselves?) This also seems gamaeable though (e.g. mock the entire implementation). Unless another agent makes a judgement call on the reasonability of the mock (though that starts to feel like code review more generally). Maybe record/replay tests could work? But there are drawbacks in the added complexity.

codethief 7mo ago
I think the solution here is: Don't mock and inject dependencies explicitly, as function parameters / monads / algebraic effects. Make side effects part of the spec/interface.

bcrosby95 7mo ago

Seems like red team is incentivized to write tests that violate the spec since you're rewarding failed tests.