4 ms·
I remember super clearly the first time an LLM told me “No.” It was in May when I was using Copilot in VS Code and switched from Claude 3.7 Sonnet to Claude Son
by Uehreka 1y ago
I remember super clearly the first time an LLM told me “No.” It was in May when I was using Copilot in VS Code and switched from Claude 3.7 Sonnet to Claude Sonnet 4. I asked Sonnet 4 to do something 3.7 Sonnet had been struggling with (something involving the FasterLivePortrait project in Python) and it told me that what I was asking for was not possible and explained why.
I get that this is different from getting an LLM to admit that it doesn’t know something, but I thought “getting a coding agent to stop spinning its wheels when set to an impossible task” was months or years away, and then suddenly it was here.
I haven’t yet read a good explanation of why Claude 4 is so much better at this kind of thing, and it definitely goes against what most people say about how LLMs are supposed to work (which is a large part of why I’ve been telling people to stop leaning on mechanical explanations of LLM behavior/strengths/weaknesses). However it was definitely a step-function improvement.
- cainxinth 1y agoYet LLMs also sometimes erroneously claim they cannot do something they can.
- s-macke 1y agoLike they learn facts by heart, they learn what they can’t by heart as well. Ask them to solve one of the Millennium Prize Problems. They’ll say they can’t do it, but that 'No' is just memorized. There’s nothing behind it.