4 ms·
Claude Sonnet 3.7 really, really loves to rewrite tests so they'll pass. I've had it happen many times in a claude-code session, I had to add this to each reque
by skerit 2y ago
Claude Sonnet 3.7 really, really loves to rewrite tests so they'll pass. I've had it happen many times in a claude-code session, I had to add this to each request (though it did not fix it 100%)
- Never disable, skip, or comment out failing unit tests. If a unit test fails, fix the root cause of the exception.
- Never change the unit test in such a way that it avoids testing the failing feature (e.g., by removing assertions, adding empty try/catch blocks, or making tests trivial).
- Do not mark tests with @Ignore or equivalent annotations.
- Do not introduce conditional logic that skips test cases under certain conditions.
- Always ensure the unit test continues to properly validate the intended functionality.
- _ea1k 2y agoI'm guessing this is a side effect of mistakes in the reinforcement learning face. It'd be really easy to build a reward model that favors passing tests, without properly measuring the quality of those tests.