3 ms·
So when Anthropic releases a new model that "breaks compatibility" with some Markdown files, do we call it "refactoring" to find (guess) the required changes to
by einrealist 7mo ago
So when Anthropic releases a new model that "breaks compatibility" with some Markdown files, do we call it "refactoring" to find (guess) the required changes to have the desired outcome again? Don't we create brittle specifications to fit a version of a model?
- Leynos 7mo agoUse evals Coming soon, unit, behavioural and regression tests for your prompts and skills :P
- stingraycharles 7mo agoHow do you use evals when you’re using Claude Code, given that Claude Code also changes their prompts all the time? You’ll have: * Claude model version * Claude Code prompts and tools * Your own prompts and skills and whatnot * Your repository’s source code (= the input) All of those change constantly, it’s not like it’s some kind of SWE benchmark.
- Leynos 7mo agoYou just said it. If consistency is that important, keep consistent versions of model, harness, prompts, skills, etc., and regression test changes. That way lies madness :)