8 ms·
I'm finding the "adaptive thinking" thing very confusing, especially having written code against the previous thinking budget / thinking effort / etc modes: htt
by simonw 7mo ago
I'm finding the "adaptive thinking" thing very confusing, especially having written code against the previous thinking budget / thinking effort / etc modes: https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking https://platform.claude.com/docs/en/build-with-claude/adapti...
Also notable: 4.7 now defaults to NOT including a human-readable reasoning token summary in the output, you have to add "display": "summarized" to get that: https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking#summarized-thinking https://platform.claude.com/docs/en/build-with-claude/adapti...
(Still trying to get a decent pelican out of this one but the new thinking stuff is tripping me up.)
- avaer 7mo ago> Still trying to get a decent pelican out of this one but the new thinking stuff is tripping me up Wouldn't that be p-hacking where p stands for pelican?
- throwup238 7mo agoThe p stands for putrification.
- LostMyLogin 7mo agoInput: I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Output: Walk. It'll take you under a minute, and driving 50 meters barely gets the engine warm — plus you'd just have to park again at the other end. Honestly, by the time you started the car, you'd already be there on foot. --- I asked it to figure out why it made the mistake: "Physical/spatial common sense. Exactly what just happened — I pattern-matched "50 meters, walk vs drive" to a pedestrian trip and skipped the part where the car is the thing being transported. Similar traps: "I need to move this couch upstairs, should I take the elevator or stairs?" (the couch has to go with you), or anything where the object in the task needs to accompany you. Counting and letter-level operations. How many R's in "strawberry," counting words in a sentence, reversing strings. I process text as tokens, not characters, so these are surprisingly error-prone. Arithmetic at scale. Small sums are fine; multi-digit multiplication, long division, or anything requiring careful carrying is where I'll confidently produce a wrong answer. I should reach for a calculator and often don't. Recent events and live data. My training has a cutoff, and even after that I can be wrong about who currently holds a role, what a stock is worth, or whether a restaurant is still open. I should search more often than I do. Negation and constraint-stacking. "Name a movie that's not a sequel, not from the 2010s, not American, and has no dialogue" — the more filters, the more likely I quietly violate one. Your specific situation. I'll give generic advice when the right answer depends on details I didn't ask about. Sometimes I should stop and ask; often I don't. Confidently wrong on obscure specifics. Quoting a passage from a real book, citing a real paper's exact title, naming the B-side of a 1974 single — I'll often generate something plausible rather than admit I don't know. Taking a premise too literally. The car wash answer is a version of this: I answered the question as posed instead of stepping back to ask what you were actually trying to do. The meta-gotcha: I'm often most wrong when I sound most confident. If an answer comes out fast and tidy on a question that should be messy, that's a signal to push back."
- vintermann 7mo agoWell, at least we know that's one gotcha/benchmark they aren't gaming.
- slekker 7mo agoWhat about Qwen? Does it get that right?
- lambda 7mo agoI've run several local models that get this right. Qwen 3.5 122B-A10B gets this right, as does Gemma 4 31B. These are local models I'm running on my laptop GPU (Strix Halo, 128 GiB of unified RAM). And I've been using this commonly as a test when changing various parameters, so I've run it several times, these models get it consistently right. Amazing that Opus 4.7 whiffs it, these models are a couple of orders of magnitude smaller, at least if the rumors of the size of Opus are true.
- qingcharles 7mo agoDoes Gemma 4 31B run full res on Strix or are you running a quantized one? How much context can you get?
- lambda 7mo agoI'm running an 8 bit quant right now, mostly for speed as memory bandwidth is the limiting factor and 8 bit quants generally lose very little compared to the full res, but also to save RAM. I'm still working on tweaking the settings; I'm hitting OOM fairly often right now, it turns out that the sliding window attention context is huge and llama.cpp wants to keep lots of context snapshots.
- qingcharles 7mo agoI had a whole bunch of trouble getting Gemma 4 working properly. Mostly because there aren't many people running it yet, so there aren't many docs on how to set it up correctly. It is a fantastic model when it works, though! Good luck :)
- deleted 7mo ago[deleted]
- haellsigh 7mo agoI got Opus 4.7 working on oh-my-pi with this commit if it interests you: https://github.com/azais-corentin/oh-my-pi/commit/6a74456f0ba90a3ddbe8cfc1100db7ddf001bd4c https://github.com/azais-corentin/oh-my-pi/commit/6a74456f0b...
- lukan 7mo ago"Also notable: 4.7 now defaults to NOT including a human-readable reasoning token summary in the output, you have to add "display": "summarized" to get that" I did not follow all of this, but wasn't there something about, that those reasoning tokens did not represent internal reasoning, but rather a rough approximation that can be rather misleading, what the model actual does?
- motoboi 7mo agoThe reasoning is the secret sauce. They don't output that. But to let you have some feedback about what is going on, they pass this reasoning through another model that generates a human friendly summary (that actively destroys the signal, which could be copied by competition).
- XenophileJKO 7mo agoDon't or can't. My assumption is the model no longer actually thinks in tokens, but in internal tensors. This is advantageous because it doesn't have to collapse the decision and can simultaneously propogate many concepts per context position.
- haellsigh 7mo agoIf that's true, then we're following the timeline of https://ai-2027.com/ https://ai-2027.com/
- dgb23 7mo agoDon't look at "thinking" tokens. LLMs sometimes produce thinking tokens that are only vaguely related to the task if at all, then do the correct thing anyways.
- thepasch 7mo agoThey also sometimes flag stuff in their reasoning and then think themselves out of mentioning it in the response, when it would actually have been a very welcome flag.
- vorticalbox 7mo agoYea I’ve seen this and stopped it and asked it about it. Sometimes they notice bugs or issues and just completely ignore it.
- Gracana 7mo agoThis can result in some funny interactions. I don't know if Claude will say anything, but I've had some models act "surprised" when I commented on something in their thinking, or even deny saying anything about it until I insisted that I can see their reasoning output.
- ceejayoz 7mo agoSupposedly (https://www.reddit.com/r/ClaudeAI/comments/1seune4/claude_cheated_at_a_number_guessing_game_got/ https://www.reddit.com/r/ClaudeAI/comments/1seune4/claude_ch...) they can't even see their own reasoning afterwards.
- astrange 7mo agoIt depends on the version. For the more recent Claudes they've been keeping it.
- shawnz 7mo agoThinking summaries might not be useful for revealing the model's actual intentions, but I find that they can be helpful in signalling to me when I have left certain things underspecified in the prompt, so that I can stop and clarify.
- p_stuart82 7mo agoyeah they took "i pick the budget" and turned it into "trust us".
- bandrami 7mo agoI keep saying even if there's not current malfeasance, the incentives being set up where the model ultimately determines the token use which determines the model provider's revenue will absolutely overcome any safeguards or good intentions given long enough.
- vessenes 7mo agoThis might be true, but right now everybody is like "please let me spend more by making you think longer." The datacenter incentives from Anthropic this month are "please don't melt our GPUs anymore" though.
- puppystench 7mo agoDoes this mean Claude no longer outputs the full raw reasoning, only summaries? At one point, exposing the LLM's full CoT was considered a core safety tenet.
- andrepd 7mo agoCoT is basically bullshit, entirely confabulated and not related to any "thought process"...
- clbrmbr 7mo agoBut still CoT distillation WORKS. See the DeepSeek R1 paper.
- whattheheckheck 7mo agoTokens relate to each other. More tokens more compute
- fasterthanlime 7mo agoI don't think it ever has. For a very long time now, the reasoning of Claude has been summarized by Haiku. You can tell because a lot of the times it fails, saying, "I don't see any thought needing to be summarised."
- fmbb 7mo agoMaybe there was no thinking.
- derrida 7mo agoNot a haiku, more a koan.
- astrange 7mo agoIt also gets confused if the entire prompt is in a text file attachment. And the summarizer shows the safety classifier's thinking for a second before the model thinking, so every question starts off with "thinking about the ethics of this request".
- cyanydeez 7mo agoIt's likely hiding the model downgrade path they require to meet sustainable revenue. Should be interesting if they can enshittify slowly enough to avoid the ablative loss of customers! Good luck all VCs!
- vessenes 7mo agoThey have super sustainable revenue. They are deadly supply constrained on compute, and have a really difficult balancing act over the next year or two in which they have to trade off spending that limited compute on model training so that they can stay ahead, while leaving enough of it available for customers that they can keep growing number of customers.
- dainiusse 7mo agoBut do they? When was the last time they declined your subscription because they have no compute?
- shawnz 7mo agoNote that for Claude Code, it looks like they added a new undocumented command line argument `--thinking-display summarized` to control this parameter, and that's the only way to get thinking summaries back there. VS Code users can write a wrapper script which contains `exec "$@" --thinking-display summarized` and set that as their claudeCode.claudeProcessWrapper in VS Code settings in order to get thinking summaries back.
- accrual 7mo agoHere is additional discussion and hacks around trying to retain Thinking output in Claude Code (prior to this release): https://github.com/anthropics/claude-code/issues/8477 https://github.com/anthropics/claude-code/issues/8477
- JamesSwift 7mo agoIts especially concerning / frustrating because boris’s reply to my bug report on opus being dumber was “we think adaptive thinking isnt working” and then thats the last I heard of it: https://news.ycombinator.com/item?id=47668520 https://news.ycombinator.com/item?id=47668520 Now disabling adaptive thinking plus increasing effort seem to be what has gotten me back to baseline performance but “our internal evals look good“ is not good enough right now for what many others have corroborated seeing
- whateveracct 7mo agoyou're using a proprietary blackbox
- JamesSwift 7mo agoSure, but that blackbox was giving me a lot of value last month.
- whateveracct 7mo agoso it's also a skinner box
- retinaros 7mo agoits a drug. that is how it works. they ration it before the new stuff. seeing legends of programming shilling it pains me the most. so far there are a few decent non insane public people talking about it :Mitchel Hashimoto, Jeremy Howard, Casei Muratori. hell even DHH drank the coolaid while most of his interviews in the past years was how he went away from AWS and reduced the bill from 3 million to 1millions by basically loosing 9s, resiliency and availability. but it seems he is fine with loosing what makes his business work(programming) to a company that sells Overpowered stack overflow slot machines.
- throwaway9980 7mo ago[flagged]
- markrogersjr 7mo agoCLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 claude…
- slekker 7mo agoWhat does that actually do? Force the "effort" to be static to what I set?
- miguno 7mo agoAs per https://code.claude.com/docs/en/model-config#adaptive-reasoning-and-fixed-thinking-budgets https://code.claude.com/docs/en/model-config#adaptive-reason...: > Opus 4.7 always uses adaptive reasoning. The fixed thinking budget mode and CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING do not apply to it.
- simonw 7mo ago... here's the pelican, I think Qwen3.6-35B-A3B running locally did a better job! https://simonwillison.net/2026/Apr/16/qwen-beats-opus/ https://simonwillison.net/2026/Apr/16/qwen-beats-opus/
- bredren 7mo agoA secret backup test to the pelican? This is as noteworthy as 4.7 dropping.
- qingcharles 7mo agoThat flamingo is hilarious. Is that his beak or a huge joint he's smoking?
- SyneRyder 7mo agoWith the sunglasses, the long flamingo neck and the "joint", I immediately thought of the poster for Fear And Loathing In Las Vegas: https://www.imdb.com/title/tt0120669/mediaviewer/rm2647909376/?ref_=tt_ov_i https://www.imdb.com/title/tt0120669/mediaviewer/rm264790937... EDIT: Actually, it must be a beak. If you zoom in, only one eye is visible and it's facing to the left. The sunglasses are actually on sideways!
- cakeface 7mo agoYou used a secret backup test! Truly honored to see the flamingos. We obviously need them all now ;-)
- ionwake 7mo agobased sun worshipping pelican
- djmips 7mo agoOpus did get the feet on pedals better.
- deleted 7mo ago[deleted]
- deleted 7mo ago[deleted]
- maximgran 7mo agohttps://github.com/anthropics/claude-agent-sdk-python/pull/830 https://github.com/anthropics/claude-agent-sdk-python/pull/8... - created PR for that cause hit it in their python sdk
- deleted 7mo ago[deleted]
- nextaccountic 7mo agoIf you do include reasoning tokens you pay more, right?
- schneehertz 7mo agoIn fact, you need to pay regardless of whether the output includes reasoning tokens or not
- boxingdog 7mo ago[dead]
- Razengan 7mo agoClaude Opus 4.6 has been hilarious for me so far: https://i.imgur.com/jYawPDY.png https://i.imgur.com/jYawPDY.png
- mosfets 7mo agoMade my day!
- XCSme 7mo agoThe reasoning modes are really weird with 4.7 In my tests, asking for "none" reasoning resulted in higher costs than asking for "medium" reasoning... Also, "medium" reasoning only had 1/10 of the reasoning tokens 4.6 used to have.
- XCSme 7mo agoOh, and also, the "none" and "medium" variants performed the same (??)
- nisarg2 7mo agoInsane! Even Haiku doesn't make such mistakes.
- XCSme 7mo agoI am not sure it's a mistake, this might be their new "adaptive reasoning" + hidden reasoning trace, so we can't verify.
- amelius 7mo agoClaude is known for its shitty metering.
- Eifert 7mo agoMedium reasoning has regressed since 4.6. While None and Max have improved since 4.6 in our benchmark. We suspect that this is how Claude tries to cope with the increased user base. Note, Google and OpenAI probably did something similar long ago.
- j45 7mo agoPrompts seem to need to evolve with every new model.
- zhonghuajin 7mo ago[flagged]
- devmor 7mo ago> Also notable: 4.7 now defaults to NOT including a human-readable reasoning token summary in the output, you have to add "display": "summarized" to get that That’s extremely bothersome because half of what helps teams build better guardrails and guidelines for agents is the ability to do deep analysis on session transcripts. I guess we shouldn’t be surprised these vendors want to do everything they can to force users to rely explicitly on their offerings.
- vdalhambra 7mo ago[flagged]
- godzillabrennus 7mo agoSince the performance of 4.6 started dropping, I started using Codex more and more. OpenAI playing it smart by being more cost-effective, even if they are catching up in terms of total utility in their desktop application, is going to win more than Anthropic (if they can't drop prices).
- weatherfun 7mo ago[dead]
- jafitc 7mo agobigger change here might not be model quality, but debuggability. once you hide the reasoning, remove the knobs, and let the model choose its own effort, it gets much harder to tell whether the model got worse or just got harder to inspect. that’s a real shift. less tool, more black box.