3 ms·
...the evidence, as best I can tell from the tweet, is that they asked Claude what effort level it was set to. But how would the model even know that? Not conv
by Wowfunhappy 1mo ago
...the evidence, as best I can tell from the tweet, is that they asked Claude what effort level it was set to. But how would the model even know that?
Not convinced here.
- jerbear4328 1mo agoEffort level is actually controlled entirely by system prompt (as I understand it, the model is trained on that format but still), so actually this is a valid way to check I think
- Wowfunhappy 1mo agoI know that's true for Qwen but I don't think most models work that way?
- wren6991 1mo agoOpenAI models also work this way, as evidenced by full cache blowout when changing reasoning level. Every single open-weight model I've seen also works this way (your "reasoning_effort" argument just changes a small section of the system prompt in the chat template). I would have to see some evidence to believe Anthropic were doing anything different.
- ranie93 1mo agoI don’t know if this is still the case but while using Copilot if you looked through the chain-of-thought output you would see it reasoning about a “budget”. i.e. “since I’m close to the session budget I should…”. So it could be possible
- varispeed 1mo agoThen model can say it's Opus, but really it is some old Sonnet. This seems to be happening less often, but some weeks ago I had to give models some test problems to gauge whether I am getting Opus or something knee-capped. The problem is that Anthropic seems to be getting away with selling one thing and delivering another. You pay for Opus, you get something else etc.
- matltc 1mo agoSaw this in npx ccusage@latest claude output. Had only used opus but showed sonnet. Can't remember if the jsonl retains which model is doing what, but meh