I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on

I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance. The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data. All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes. I opted to just build my own harness for these things to avoid overfitting. https://softwaredoug.com/blog/2026/05/17/autoresearching-a-better-msmarco-bm25 https://softwaredoug.com/blog/2026/05/17/autoresearching-a-b...