4 ms·
Sonnet 4 is weird. Sometimes it creates empty files and wants to delete what I asked it to refactor, only to confuse itself even more. So I retry, same path of
by jmaker 1y ago
Sonnet 4 is weird. Sometimes it creates empty files and wants to delete what I asked it to refactor, only to confuse itself even more. So I retry, same path of destruction. Next time I interrupt it and explicitly state that it must first move the type definitions to the new files, it just ignores that (exclaiming several times how “absolutely right!” I was), and destroys the files anyway.
I mean it’s not even good as a refactoring tool sometimes. Sometimes it’s acceptable to a degree.
It loves stopping in the middle of a refactoring or generating a test suite, even though it convinced itself that the tests were still failing.
That’s on something simple like TypeScript in a Node microservice repo.
Same MCP servers, same context, instructions, prompt templates, same config, same repos. GitHub Copilot, Claude Code.
So I just turn to a mixture of ChatGPT models where I need a quick win on a repo I took over and need to upgrade or when I want extra checks for potential mistakes or when I need a quick summary of some AWS docs without with links to verify.
But of all things reliable it is not yet.