10 ms·
Reasoning models don't always say what they think
- nottorp 2y ago... because they don't think.
- evrimoztamur 2y agoSounds like LLMs short-circuit without necessarily testing their context assumptions. I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact. I think this also explains a common issue with LLMs where people get the answer they're looking for, regardless of whether it's true or there's a CoT in place.
- jiveturkey 2y agoi found with the gemini answer box on google, it's quite easy to get the answer you expect. i find myself just playing with it, asking a question in the positive sense then the negative sense, to get the 2 different "confirmations" from gemini. also it's easily fooled by changing the magnitude of a numerical aspect of a question, like "are thousands of people ..." then "are millions of people ...". and then you have the now infamous black/white people phrasing of a question. i haven't found perplexity to be so easily nudged.
- andrewmcwatters 2y agoThis is such an annoying issue in assisted programming as well. Say you’re referencing a specification, and you allude to two or three specific values from that specification, you mention needing a comprehensive list and the LLM has been trained on it. I’ll often find that all popular models will only use the examples I’ve mentioned and will fail to elaborate even a few more. You might as well read specifications yourself. It’s a critical feature of these models that could be an easy win. It’s autocomplete! It’s simple. And they fail to do it every single time I’ve tried a similar abstract. I laugh any time people talk about these models actually replacing people. They fail at reading prompts at a grade school reading level.
- BurningFrog 2y agoThe LLMs copy human written text, so maybe they'll implement Motivated Reasoning just like humans do? Or maybe it's telling people what they want to hear, just like humans do
- ben_w 2y agoThey definitely tell people what they want to hear. Even when we'd rather they be correct, they get upvoted or downvoted by users, so this isn't avoidable (but is is fawning or sychophancy?) I wonder how deep or shallow the mimicry of human output is — enough to be interesting, but definitely not quite like us.
- jiveturkey 2y agoseemed common-sense obvious to me -- AI (LLMs) don't "reason". great to see it methodically probed and reported in this way. but i am just a casual observer of all things AI. so i might be too naive in my "common sense".
- zurfer 2y agoI recently had fascinating example of that where Sonnet 3.7 had to decide for one option from a set of choices. In the thinking process it narrowed it down to 2 and finally in the last thinking section it decided for one, saying it's best choice. However, in the final output (outside of thinking) it then answered with the other option with no clear reason given
- thomassmith65 2y agoOne interesting quirk with Claude is that it has no idea its Chain-of-Thought is visible to users. In one chat, it repeatedly accused me of lying about that. It only conceded after I had it think of a number between one and a million, and successfully 'guessed' it.
- reaperman 2y agoEdit: 'wahnfrieden corrected me. I incorrectly posited that CoT was only included in the context window during the reasoning task and later left out entirely. Edited to remove potential misinformation.
- monsieurbanana 2y agoIn which case the model couldn't possibly know that the number was correct.
- Me1000 2y agoI'm also confused by that, but it could just be the model being agreeable. I've seen multiple examples posted online though where it's fairly clear that the COT output is not included in subsequent turns. I don't believe Anthropic is public about it (could be wrong), but I know that the Qwen team specifically recommend against including COT tokensfrom previous inferences.
- thomassmith65 2y agoClaude has some awareness of its CoT. As an experiment, it's easy, for example, to ask Claude to "think of a city, but only reply with the word 'ready' and next to ask "what is the first letter of the city you thought of?"
- thomassmith65 2y agoOops! I tried a couple experiments after writing this, and I believe I was mistaken, though I don't know how. It appears Claude was simply playing along, and convinced me it could remember the choices it secretly made. I must either have given it a tell, or perhaps it guessed the same answers twice in a row.
- lsy 2y agoThe fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it "explain itself"? CoT improves results, sure. And part of that is probably because you are telling the LLM to add more things to the context window, which increases the potential of resolving some syllogism in the training data: One inference cycle tells you that "man" has something to do with "mortal" and "Socrates" has something to do with "man", but two cycles will spit those both into the context window and lets you get statistically closer to "Socrates" having something to do with "mortal". But given that the training/RLHF for CoT revolves around generating long chains of human-readable "steps", it can't really be explanatory for a process which is essentially statistical.
- hnuser123456 2y agoWhen we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. But as you say, currently, they have zero "self awareness".
- dragonwriter 2y ago> When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. While I believe we are far from AGI, I don't think the standard for AGI is an AI doing things a human absolutely cannot do.
- redeux 2y agoAll that was described here is learning from a mistake, which is something I hope all humans are capable of.
- pton_xd 2y agoI was under the impression that CoT works because spitting out more tokens = more context = more compute used to "think." Using CoT as a way for LLMs "show their working" never seemed logical, to me. It's just extra synthetic context.
- margalabargala 2y agoMy understanding of the "purpose" of CoT, is to remove the wild variability yielded by prompt engineering, by "smoothing" out the prompt via the "thinking" output, and using that to give the final answer. Thus you're more likely to get a standardized answer even if your query was insufficiently/excessively polite.
- voidspark 2y agoThat's right. It's not "show the working". It's "do more working".
- tasty_freeze 2y agoHumans sometimes draw a diagram to help them think about some problem they are trying to solve. The paper contains nothing that the brain didn't already know. However, it is often an effective technique. Part of that is to keep the most salient details front and center, and part of it is that the brain isn't fully connected, which allows (in this case) the visual system to use its processing abilities to work on a problem from a different angle than keeping all the information in the conceptual domain.
- svachalek 2y agoThis is an interesting paper, it postulates that the ability of an LLM to perform tasks correlates mostly to the number of layers it has, and that reasoning creates virtual layers in the context space. https://arxiv.org/abs/2412.02975 https://arxiv.org/abs/2412.02975
- ertgbnm 2y agoBut the model doesn't have an internal state, it just has the tokens, which means it must encode it's reasoning into the output tokens. So it is a reasonable take to think that CoT was them showing their work.
- moralestapia 2y ago40 billion cash to OpenAI while others keep chasing butterflies. Sad.
- nodja 2y agoI highly suspect that CoT tokens are at least partially working as register tokens. Have these big LLM trainers tried replacing CoT with a similar amount of register tokens and see if the improvements are similar?
- wgd 2y agoI remember there was a paper a little while back which demonstrated that merely training a model to output "........" (or maybe it was spaces?) while thinking provided a similar improvement in reasoning capability to actual CoT.
- PeterStuer 2y agoHumans also post-rationalize the things their subconscious "gut feeling" came up with. I have no problem for a system to present a reasonable argument leading to a production/solution, even if that materially was not what happened in the generation process. I'd go even further and pose that probably requiring the "explanation" to be not just congruent but identical with the production would either lead to incomprehensible justifications or severely limited production systems.
- pixl97 2y agoNow, at least in a well disciplined human, we can catch when our gut feeling was wrong when the 'create a reasonable argument' process fails. I guess I wonder how well a LLM can catch that and correct it's thinking. Now I've seen in some models where it figures out it's wrong, but then gets stuck in a loop. I've not really used the larger reasoning models much to see their behaviors.
- eab- 2y agoyep, this post is full of this post-rationalization, for example. it's pretty breathtaking
- alach11 2y agoThis is basically a big dunk on OpenAI, right? OpenAI made a big show out of hiding their reasoning traces and using them for alignment purposes [0]. Anthropic has demonstrated (via their mech interp research) that this isn't a reliable approach for alignment. [0] https://openai.com/index/chain-of-thought-monitoring/ https://openai.com/index/chain-of-thought-monitoring/
- gwd 2y agoI don't think those are actually showing different things. The OpenAI paper is about the LLM planning to itself to hack something; but when they use training to suppress this "hacking" self-talk, it still hacks the reward function almost as much, it just doesn't use such easily-detectable language. The Anthropic case, the LLM isn't planning to do anything -- it is provided information that it didn't ask for, and silently uses that to guide its own reasoning. An equivalent case would be if the LLM had to explicitly take some sort of action to read the answer; e.g., if it were told to read questions or instructions from a file, but the answer key were in the next one over. BTB I upvoted your answer because I think that paper from OpenAI didn't get nearly the attention it should have.
- ctoth 2y agoI invite anyone who postulates humans are more than just "spicy autocomplete" to examine this thread. The level of actual reasoning/engaging with the article is ... quite something.
- AgentME 2y agoInternet commenters don't "reason". They just generate inane arguments over definitions, like a lowly markov bot, without the true spark of life and soul that even certain large language models have.
- Marazan 2y agoYou don't say. This is my very shocked face.
- AYHL 2y agoTo me CoT is nothing but lowering learning rate and increasing iterations in a typical ML model. It's basically to force the model to make a small step at a time and try more times to increase accuracy.
- xg15 2y ago> There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; Isn't the whole reason for chain-of-thought that the tokens sort of are the reasoning process? Yes, there is more internal state in the model's hidden layers while it predicts the next token - but that information is gone at the end of that prediction pass. The information that is kept "between one token and the next" is really only the tokens themselves, right? So in that sense, the OP would be wrong. Of course we don't know what kind of information the model encodes in the specific token choices - I.e. the tokens might not mean to the model what we think they mean.
- svachalek 2y agoExactly. There's no state outside the context. The difference in performance between the non-reasoning model and the reasoning model comes from the extra tokens in the context. The relationship isn't strictly a logical one, just as it isn't for non-reasoning LLMs, but the process is autoregression and happens in plain sight.
- miven 2y agoI'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole lot of internal state that's stored in (mostly) inscrutable key and value vectors with hundreds of dimensions per attention head, around a few dozen heads per attention block, and around a few dozen blocks per model.
- xg15 2y agoYes, but all that internal state only survives until the end of the computation chain that predicts the next token - it doesn't survive across the entire sequence as it would in a recurrent network. There is literally no difference between a model predicting the tokens "<thought> I think the second choice looks best </thought>" and a user putting those tokens into the prompt: The input for the next round would be exactly the same. So the tokens kind of act like a bottleneck (or more precisely the sampling of exactly one next token at the end of each prediction round does). During prediction of one token, the model can go crazy with hidden state, but not across several tokens. That forces the model to do "long form" reasoning through the tokens and not through hidden state.
- lpzimm 2y agoNot exactly the same as this study, but I'll ask questions to LLMs with and without subtle hints to see if it changes the answer and it almost always does. For example, paraphrased: No hint: "I have an otherwise unused variable that I want to use to record things for the debugger, but I find it's often optimized out. How do I prevent this from happening?" Answer: 1. Mark it as volatile (...) Hint: "I have an otherwise unused variable that I want to use to record things for the debugger, but I find it's often optimized out. Can I solve this with the volatile keyword or is that a misconception?" Answer: Using volatile is a common suggestion to prevent optimizations, but it does not guarantee that an unused variable will not be optimized out. Try (...) This is Claude 3.7 Sonnet.
- pixl97 2y agoI mean, this sounds along the lines of human conversations that go like P1 "Hey, I'm doing A but X is happening" P2 "Have you tried doing Y? P1 "Actually, yea I am doing A.Y and X is still occurring" P2 "Oh, you have the special case where you need to do A.Z" What happens when you ask your first question with something like "what is the best practice to prevent this from happening"
- lpzimm 2y agoOh sorry, these are two separate chats, I wasn't clear. I would agree that if I had asked them in the same chat it would sound pretty normal. When I ask about best practices it does still give me the volatile keyword. (I don't even think that's wrong, when I threw it in Godbolt with -O3 or -Os I couldn't find a compiler that optimized it away.)
- priyadarshin 2y ago[dead]
- nopelynopington 2y agoOf course they don't. LLMs are a brainless algorithm that guesses the next word. When you ask them what they think they're also guessing the next word. No reason for it to match, except a trick of context
- deleted 2y ago[deleted]
- afro88 2y agoCan a model even know that it used a hint? Or would it only say so if it was trained to say what parts of the context it used when asked? Because then it's statistically probable to say so?
- richardw 2y agoOne thing I think I’ve found is: reasoning models get more confident and that makes it harder to dislodge a wrong idea. It feels like I only have 5% of the control, and then it goes into a self-chat where it thinks it’s right and builds on it’s misunderstanding. So 95% of the outcome is driven by rambling, not my input. Windsurf seems to do a good job of regularly injecting guidance so it sticks to what I’ve said. But I’ve had some extremely annoying interactions with confident-but-wrong “reasoning” models.
- freehorse 2y agoIt is nonsense to take whatever an LLM writes in its CoT too seriously. I try to classify some messy data, writing "if X edge case appears, then do Y instead of Z". The model in its CoT took notice of X, wrote it should do Y and... it would not do it in the actual output. The only way to make actual use of LLMs imo is to treat them as what they are, a model that generates text based on some statistical regularities, without any kind of actual understanding or concepts behind that. If that is understood well, one can know how to setup things in order to optimise for desired output (or "alignment"). The way "alignment research" presents models as if they are actually thinking or have intentions of their own (hence the choice of the word "alignment" for this) makes no sense.
- thoughtlede 2y agoIt feels to me that the hypothesis of this research was somewhat "begging the question". Reasoning models are trained to spit some tokens out that increase the chance of the models spitting the right answer at the end. That is, the training process is singularly optimizing for the right answer, not the reasoning tokens. Why would you then assume the reasoning tokens will include hints supplied in the prompt "faithfully"? The model may or may not include the hints - depending on whether the model activations believe those hints are necessary to arrive at the answer. In their experiments, they found between 20% and 40% of the time, the models included those hints. Naively, that sounds unsurprising to me. Even in the second experiment when they trained the model to use hints, the optimization was around the answer, not the tokens. I am not surprised the models did not include the hints because they are not trained to include the hints. That said, and in spite of me potentially coming across as an unsurprised-by-the-result reader, it is a good experiment because "now we have some experimental results" to lean into. Kudos to Anthropic for continuing to study these models.
- m3kw9 2y agoWhat would “think” mean? Processed the prompt? Or just accessed the part of the model where the weights are? This is a bit persudo science
- islewis 2y ago> For the purposes of this experiment, though, we taught the models to reward hack [...] in this case rewarded the models for choosing the wrong answers that accorded with the hints. > This is concerning because it suggests that, should an AI system find hacks, bugs, or shortcuts in a task, we wouldn’t be able to rely on their Chain-of-Thought to check whether they’re cheating or genuinely completing the task at hand. As a non-expert in this field, I fail to see why a RL model taking advantage of it's reward is "concerning". My understanding is that the only difference between a good model and a reward-hacking model is if the end behavior aligns with human preference or not. The articles TL:DR reads to me as "We trained the model to behave badly, and it then behaved badly". I don't know if i'm missing something, or if calling this concerning might be a little bit sensationalist.
- bee_rider 2y agoChain of thought does have a minor advantage in the final “fish” example—the explanation blatantly contradicts itself to get to the cheated hint answer. A human reading it should be pretty easily able to tell that something fishy is going on… But, yeah, it is sort of shocking if anybody was using “chain of thought” as a reflection of some actual thought process going on in the model, right? The “thought,” such as it is, is happening in the big pile of linear algebra, not the prompt or the intermediary prompts. Err… anyway, like, IBM was working on explainable AI years ago, and that company is a dinosaur. I’m not up on what companies like OpenAI are doing, but surely they aren’t behind IBM in this stuff, right?
- madethisnow 2y agoIf something convinces you that it's aware then it is. Simulated computation IS computation itself. The territory is the map
- jxjnskkzxxhx 2y agoMeh. People also invent justifications after the fact.
- EncomLab 2y agoThe use of highly anthropomorphic language is always problematic- Does a photo resistor controlled nightlight have a chain of thought? Does it reason about its threshold value? Does it have an internal model of what is light, what is dark, and the role it plays in demarcation between the two? Are the transistors executing the code within the confines even capable of intentionality? If so - where is it derived from?
- lngnmn2 2y ago[dead]
- HammadB 2y agoThere is an abundance of discussion on this thread about whether models are intelligent or not. This binary is an utter waste of time. Instead focus on the gradient of intelligence - the set of cognitive skills any given system has and to what degree it has them. This engineering approach is more likely to lead to practical utility and progress. The view of intelligence as binary is incredibly corrosive to this field.
- nlpnerd 2y agoI have always believed that Chain of Thought basically acts as a form of regularization. LLMs are fundamentally next token predictors without any form or notion of logic, reasoning etc and is as likely (as a probabilistic model) to produce a creative response as something based on facts/principles (or anything that resembles "reasoning"). Asking the LLM to think step by step simply biases it towards the latter. It's still a stochastic parrot but now it sounds logical and that happens to be useful in some cases, regardless of whether we can agree if it's "reasoning".