12 ms·
Is chain-of-thought AI reasoning a mirage?
- NitpickLawyer 1y agoFinally! A good take on that paper. I saw that arstechnica article posted everywhere, and most of the comments are full of confirmation bias, and almost all of them miss the fineprint - it was tested on a 4 layer deep toy model. It's nice to read a post that actually digs deeper and offers perspectives on what might be a good finding vs. just warranting more research.
- stonemetal12 1y ago> it was tested on a 4 layer deep toy model How do you see that impacting the results? It is the same algorithm just on a smaller scale. I would assume a 4 layer model would not be very good, but does reasoning improve it? Is there a reason scale would impact the use of reasoning?
- okasaki 1y agoHuman babies are the same algorithm as adults.
- mirekrusin 1y agoThis analogy would mean very large model that didn't finish training yet. Tiny model like this is more like doing study on fruit flies and extrapolating results to humans.
- archaeans 1y agoEvery argument about LLMs of that is a variant of "humans same" is self defeating because it assumes a level of understanding of human cognition and the human brain that doesn't really exist outside of the imagination of people with a poor understanding of neuroscience.
- okasaki 1y agoSome humans never attain intelligence beyond early childhood.
- azrazalea_debt 1y agoA lot of current LLM work is basically emergent behavior. They use a really simple core algorithm and scale it up, and interesting things happen. You can read some of anthropic's recent papers to see some of this, like: They didn't expect LLMs could "lookahead" when writing poetry. However, when they actually went in and watched what was happening (there's details on how this "watching" works on their blog/in their studies) they found the LLM actually was planning ahead! That's emergent behavior, they didn't design it to do that, it just started doing due to the complexity of the model. If (BIG if) we ever do see actual AGI, it is likely to work like this. It's unlikely we're going to make AGI by designing some grand Cathedral of perfect software, it is more likely we are going to find the right simple principles to scale big enough to have AGI emerge. This is similar.
- pinoy420 1y ago[dead]
- mrspuratic 1y agoOn that topic, it seems backwards to me: intelligence is not emergent behaviour of language, rather the opposite.
- deleted 1y ago[deleted]
- danans 1y agoPerception and interpretation can very much be influenced by language (Sapir-Wharf hypothesis), so to the extent that perception and interpretation influence intelligence, it's not clear that the relationship is only in one direction.
- archaeans 1y ago"It would be naïve to imagine that any analysis of experience is dependent on pattern expressed in language." - Sapir It's hard to take these discussions on cognition and intelligence seriously when there is so much lossy compression going on.
- NitpickLawyer 1y agoThere's prior research that finds a connection between model depth and "reasoning" ability - https://arxiv.org/abs/2503.03961 https://arxiv.org/abs/2503.03961 A depth of 4 is very small. It is very much a toy model. It's ok to research this, and maybe someone will try it out on larger models, but it's totally not ok to lead with the conclusion, based on this toy model, IMO.
- sempron64 1y ago[flagged]
- mwkaufma 1y agoBetteridge's law applies to editors adding question marks to cover-the-ass of articles with weak claims, not bloggers begging questions.
- robviren 1y agoI feel it is interesting but not what would be ideal. I really think if the models could be less linear and process over time in latent space you'd get something much more akin to thought. I've messed around with attaching reservoirs at each layer using hooks with interesting results (mainly over fitting), but it feels like such a limitation to have all model context/memory stuck as tokens when latent space is where the richer interaction lives. Would love to see more done where thought over time mattered and the model could almost mull over the question a bit before being obligated to crank out tokens. Not an easy problem, but interesting.
- dkersten 1y agoAgree! I’m not an AI engineer or researcher, but it always struck me as odd that we would serialise the 100B or whatever parameters of latent space down to maximum 1M tokens and back for every step.
- vonneumannstan 1y ago>I feel it is interesting but not what would be ideal. I really think if the models could be less linear and process over time in latent space you'd get something much more akin to thought. Please stop, this is how you get AI takeovers.
- varelse 1y ago[dead]
- adastra22 1y agoCitation seriously needed.
- vonneumannstan 1y agoIt's really very simple. As models become more capable they may become interested in deceiving humans or otherwise manipulating them to achieve their goals. We already see this in various places see: https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 If the chain of thought of models becomes pure "neuralese" i.e. the models think purely in latent space then we will lose the ability to monitor for malicious behavior. This is incredibly dangerous, CoT monitoring is one of the best and highest leverage tools for monitoring model behavior and losing that would be devastating for safety. https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-about-latent-reasoning-in-llms https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-ab... https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-f... https://www.lesswrong.com/posts/3W8HZe8mcyoo4qGkB/an-idea-for-avoiding-neuralese-architectures-1 https://www.lesswrong.com/posts/3W8HZe8mcyoo4qGkB/an-idea-fo... https://x.com/RyanPGreenblatt/status/1908298069340545296 https://x.com/RyanPGreenblatt/status/1908298069340545296 https://redwoodresearch.substack.com/p/notes-on-countermeasures-for-exploration https://redwoodresearch.substack.com/p/notes-on-countermeasu...
- mentalgear 1y ago> Whether AI reasoning is “real” reasoning or just a mirage can be an interesting question, but it is primarily a philosophical question. It depends on having a clear definition of what “real” reasoning is, exactly. It's pretty easy: causal reasoning. Causal, not statistic correlation only as LLM do, with or without "CoT".
- naasking 1y agoDefine causal reasoning?
- glial 1y agoCorrect me if I'm wrong, I'm not sure it's so simple. LLMs are called causal models in the sense that earlier tokens "cause" later tokens, that is, later tokens are causally dependent on what the earlier tokens are. If you mean deterministic rather than probabilistic, even Pearl-style causal models are probabilistic. I think the author is circling around the idea that their idea of reasoning is to produce statements in a formal system: to have a set of axioms, a set of production rules, and to generate new strings/sentences/theorems using those rules. This approach is how math is formalized. It allows us to extrapolate - make new "theorems" or constructions that weren't in the "training set".
- jayd16 1y agoBy this definition a bag of answers is causal reasoning because we previously filled the bag, which caused what we pulled. State causing a result is not causal reasoning. You need to actually have something that deduces a result from a set of principles that form a logical conclusion or the understanding that more data is needed to make a conclusion. That is clearly different than finding a likely next token on statics alone, despite the fact the statical answer can be correct.
- apples_oranges 1y agoBut let's say you change your mathematical expression by reducing or expanding it somehow, then, unless it's trivial, there are infinite ways to do it, and the "cause" here is the answer to the question of "why did you do that and not something else"? Brute force excluded, the cause is probably some idea, some model of the problem or a gut feeling (or desperation..) ..
- empath75 1y agoOne thing that LLMs have exposed is how much of a house of cards all of our definitions of "human mind"-adjacent concepts are. We have a single example in all of reality of a being that thinks like we do, and so all of our definitions of thinking are inextricably tied with "how humans think", and now we have an entity that does things which seem to be very like how we think, but not _exactly like it_, and a lot of our definitions don't seem to work any more: Reasoning, thinking, knowing, feeling, understanding, etc. Or at the very least, our rubrics and heuristics for determining if someone (thing) thinks, feels, knows, etc, no longer work. And in particular, people create tests for those things thinking that they understand what they are testing for, when _most human beings_ would also fail those tests. I think a _lot_ of really foundational work needs to be done on clearly defining a lot of these terms and putting them on a sounder basis before we can really move forward on saying whether machines can do those things.
- gdbsjjdn 1y agoCongratulations, you've invented philosophy.
- empath75 1y agoThis is an obnoxious response. Of course I recognize that philosophy is the solution to this. What I am pointing out is that philosophy has not as of yet resolved these relatively new problems. The idea that non-human intelligences might exist is of course an old one, but that is different from having an actual (potentially) existing one to reckon with.
- adastra22 1y agoThese are not new problems though.
- deadbabe 1y agoNon-human intelligences have always existed in the form of animals. Animals do not have spoken language the way humans do, so their thoughts aren’t really composed of sentences. Yet, they have intelligence and can reason about their world. How could we build an AGI that doesn’t use language to think at all? We have no fucking clue and won’t for a while because everyone is chasing the mirage created by LLMs. AI winter will come and we’ll sit around waiting for the next big innovation. Probably some universal GOAP with deeply recurrent neural nets.
- naasking 1y ago> Because reasoning tasks require choosing between several different options. “A B C D [M1] -> B C D E” isn’t reasoning, it’s computation, because it has no mechanism for thinking “oh, I went down the wrong track, let me try something else”. That’s why the most important token in AI reasoning models is “Wait”. In fact, you can control how long a reasoning model thinks by arbitrarily appending “Wait” to the chain-of-thought. Actual reasoning models change direction all the time, but this paper’s toy example is structurally incapable of it. I think this is the most important critique that undercuts the paper's claims. I'm less convinced by the other point. I think backtracking and/or parallel search is something future papers should definitely look at in smaller models. The article is definitely also correct on the overreaching, broad philosophical claims that seems common when discussing AI and reasoning.
- mucho_mojo 1y agoThis paper I read from here has an interesting mathematical model for reasoning based on cognitive science. https://arxiv.org/abs/2506.21734 https://arxiv.org/abs/2506.21734 (there is also code here https://github.com/sapientinc/HRM https://github.com/sapientinc/HRM) I think we will see dramatic performance increases on "reasoning" problems when this is worked into existing AI architectures.
- deleted 1y ago[deleted]
- stonemetal12 1y agoWhen Using AI they say "Context is King". "Reasoning" models are using the AI to generate context. They are not reasoning in the sense of logic, or philosophy. Mirage, whatever you want to call it, it is rather unlike what people mean when they use the term reasoning. Calling it reasoning is up there with calling generating out put people don't like hallucinations.
- bongodongobob 1y agoAnd yet it improves their problem solving ability.
- adastra22 1y agoYou are making the same mistake OP is calling out. As far as I can tell “generating context” is exactly what human reasoning is too. Consider the phrase “let’s reason this out” where you then explore all options in detail, before pronouncing your judgement. Feels exactly like what the AI reasoner is doing.
- stonemetal12 1y ago"let's reason this out" is about gathering all the facts you need, not just noting down random words that are related. The map is not the terrain, words are not facts.
- deleted 1y ago[deleted]
- energy123 1y agoPerformance is proportional to the number of reasoning tokens. How to reconcile that with your opinion that they are "random words"?
- kelipso 1y agoTechnically random can have probabilities associated with them.. Casual speech, random means equal probabilities, or we don’t know the probabilities. But for LLM token output, it does estimate the probabilities.
- moc_was_wronged 1y ago[dead]
- modeless 1y ago"The question [whether computers can think] is just as relevant and just as meaningful as the question whether submarines can swim." -- Edsger W. Dijkstra, 24 November 1983
- mdp2021 1y agoBut the topic here is whether some techniques are progressive or not (with a curious parallel about whether some paths in thought are dead-ends - the unproductive focus mentioned in the article).
- draw_down 1y ago[dead]
- griffzhowl 1y agoI don't agree with the parallel. Submarines can move through water - whether you call that swimming or not isn't an interesting question, and doesn't illuminate the function of a submarine. With thinking or reasoning, there's not really a precise definition of what it is, but we nevertheless know that currently LLMs and machines more generally can't reproduce many of the human behaviours that we refer to as thinking. The question of what tasks machines can currently accomplish is certainly meaningful, if not urgent, and the reason LLMs are getting so much attention now is that they're accomplishing tasks that machines previously couldn't do. To some extent there might always remain a question about whether we call what the machine is doing "thinking" - but that's the uninteresting verbal question. To get at the meaningful questions we might need a more precise or higher resolution map of what we mean by thinking, but the crucial element is what functions a machine can perform, what tasks it can accomplish, and whether we call that "thinking" or not doesn't seem important. Maybe that was even Dijkstra's point, but it's hard to tell without context...
- wizzwizz4 1y agohttps://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898... provides the context. I haven't re-read it in the last month, but I'm pretty sure you've correctly identified Dijkstra's point.
- skybrian 1y agoMathematical reasoning does sometimes require correct calculations, and if you get them wrong your answers will be wrong. I wouldn’t want someone doing my taxes to be bad at calculation or bad at finding mistakes in calculation. It would be interesting to see if this study’s results can be reproduced in a more realistic setting.
- pinoy420 1y ago[dead]
- slashdave 1y ago> reasoning probably requires language use The author has a curious idea of what "reasoning" entails.
- sixdimensional 1y agoI feel like the fundamental concept of symbolic logic[1] as a means of reasoning fits within the capabilities of LLMs. Whether it's a mirage or not, the ability to produce a symbolically logical result that has valuable meaning seems real enough to me. Especially since most meaning is assigned by humans onto the world... so too can we choose to assign meaning (or not) to the output of a chain of symbolic logic processing? Edit: maybe it is not so much that an LLM calculates/evaluates the result of symbolic logic as it is that it "follows" the pattern of logic encoded into the model. [1] https://en.wikipedia.org/wiki/Logic https://en.wikipedia.org/wiki/Logic
- lawrence1 1y agowe should be asking if reasoning while speaking is even possible for humans. this is why we have the scientific method and that's why LLMs write and run unit tests on their reasoning. But yeah intelligence is probably in the ear of the believer.
- hungmung 1y agoChain of thought is just a way of trying to squeeze more juice out of the lemon of LLM's; I suspect we're at the stage of running up against diminishing returns and we'll have to move to different foundational models to see any serious improvement.
- archaeans 1y agoThe so-called "scaling laws" are expressing diminishing returns. How is it that "if we grow resources used exponentially errors decrease linearly" ever seen as a good sign?
- brunokim 1y agoI'm unconvinced by the article criticism's, given they also employ their feels and few citations. > I appreciate that research has to be done on small models, but we know that reasoning is an emergent capability! (...) Even if you grant that what they’re measuring is reasoning, I am profoundly unconvinced that their results will generalize to a 1B, 10B or 100B model. A fundamental part of applied research is simplifying a real-world phenomenon to better understand it. Dismissing that for this many parameters, for such a simple problem, the LLM can't perform out of distribution just because it's not big enough undermines the very value of independent research. Tomorrow another model with double the parameters may or may not show the same behavior, but that finding will be built on top of this one. Also, how do _you_ know that reasoning is emergent, and not rationalising on top of a compressed version of the web stored in 100B parameters?
- ActionHank 1y agoI think that when you are arguing logic and reason with a group who became really attached to the term vibe-coding you've likely already lost.
- mirekrusin 1y agoFeels like running psychology experiment with fruit flies because it's cheaper and extrapolating results to humans because it's almost the same thing but smaller. I'm sorry but the only hallucination here is that of the authors here. Does it really need to be said again that interesting results happen when you scale up only? This whole effort would be interesting if they did and plotted result while scaling something up.
- deleted 1y ago[deleted]
- LudwigNagasena 1y ago> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "think" and "reason" without outputting a single "word". > Whether AI reasoning is “real” reasoning or just a mirage can be an interesting question, but it is primarily a philosophical question. It depends on having a clear definition of what “real” reasoning is, exactly. It is not a "philosophical" (by which the author probably meant "practically inconsequential") question. If the whole reasoning business is just rationalization of pre-computed answers or simply a means to do some computations because every token provides only a fixed amount of computation to update the model's state, then it doesn't make much sense to focus on improving the quality of chain-of-thought output from human POV.
- kazinator 1y agoNot all reasoning requires language. Symbolic reasoning uses language. Real-time spatial reasoning like driving a car and not hitting things does not seem linguistic. Figuring out how to rotate a cabinet so that it will clear through a stairwell also doesn't seem like it requires language, only to communicate the solution to someone else (where language can turn into a hindrance, compared to a diagram or model).
- vmg12 1y agoSolutions to some of the hardest problems I've had have only come after a night of sleep or when I'm out on a walk and I'm not even thinking about the problem. Maybe what my brain was doing was something different from reasoning?
- skywhopper 1y agoI mostly agree with the point the author makes that "it doesn't matter". But then again, it does matter, because LLM-based products are marketed based on "IT CAN REASON!" And so, while it may not matter, per se, how an LLM comes up with its results, to the extent that people choose to rely on LLMs because of marketing pitches, it's worth pushing back on those claims if they are overblown, using the same frame that the marketers use. That said, this author says this question of whether models "can reason" is the least interesting thing to ask. But I think the least interesting thing you can do is to go around taking every complaint about LLM performance and saying "but humans do the exact same thing!" Which is often not true, but again, doesn't matter.
- cess11 1y agoYes, it's a mirage, since this type of software is an opaque simulation, perhaps even a simulacra. It's reasoning in the same sense as there are terrorists in a game of Counter-Strike.
- jrm4 1y agoCurrent thought, for me there's a lot of hand-wringing about what is "reasoning" and what isn't. But right now perhaps the question might be boiled down to -- "is the bottleneck merely hard drive space/memory/computing speed?" I kind of feel like we won't be able to even begin to test this until a few more "Moore's law" cycles.
- j45 1y agoCurrently it feels like it's more simulated chain-of-thought / reasoning, sometimes very consistent, but simulated, partially because it's statistically generated and non-deterministic (not the exact same path to the similar or same each response run).
- js8 1y agoI think LLM's chain of thought is reasoning. When trained, LLM sees lot of examples like "All men are mortal. Socrates is a man." followed by "Therefore, Socrates is mortal.". This causes the transformer to learn rule "All A are B. C is A." is often followed by "Therefore, C is B." And so it can apply this logical rule, predictively. (I have converted the example from latent space to human language for clarity.) Unfortunately, sometimes LLM also learns "All A are C. All B are C." is followed by "Therefore, A is B.", due to bad example in the training data. (More insidiously, it might learn this rule only in a special case.) So it learns some logic rules but not consistently. This lack of consistency will cause it to fail on larger problems. I think NNs (transformers) could be great in heuristic suggesting which valid logical rules (could be even modal or fuzzy logic) to apply in order to solve a certain formalized problem, but not so great at coming up with the logic rules themselves. They could also be great at transforming the original problem/question from human language into some formal logic, that would then be resolved using heuristic search.
- handoflixue 1y agoHumans are also notoriously bad at this, so we have plenty of evidence that this lack of consistency does indeed cause failures on larger problems.
- js8 1y agoYes, humans fail at this, that's why we need technology tnat doesn't simply emulate humans, but tries to be more reliable than us.
- gshulegaard 1y ago> but we know that reasoning is an emergent capability! Do we though? There is widespread discussion and growing momentum of belief in this, but I have yet to see conclusive evidence of this. That is, in part, why the subject paper exists...it seeks to explore this question. I think the author's bias is bleeding fairly heavily into his analysis and conclusions: > Whether AI reasoning is “real” reasoning or just a mirage can be an interesting question, but it is primarily a philosophical question. It depends on having a clear definition of what “real” reasoning is, exactly. I think it's pretty obvious that the researchers are exploring whether or not LLMs exhibit evidence of _Deductive_ Reasoning [1]. The entire experiment design reflects this. Claiming that they haven't defined reasoning and therefore cannot conclude or hope to construct a viable experiment is...confusing. The question of whether or not an LLM can take a set of base facts and compose them to solve a novel/previously unseen problem is interesting and what most people discussing emergent reasoning capabilities of "AI" are tacitly referring to (IMO). Much like you can be taught algebraic principles and use them to solve for "x" in equations you have never seen before, can an LLM do the same? To which I find this experiment interesting enough. It presents a series of facts and then presents the LLM with tasks to see if it can use those facts in novel ways not included in the training data (something a human might reasonably deduce). To which their results and summary conclusions are relevant, interesting, and logically sound: > CoT is not a mechanism for genuine logical inference but rather a sophisticated form of structured pattern matching, fundamentally bounded by the data distribution seen during training. When pushed even slightly beyond this distribution, its performance degrades significantly, exposing the superficial nature of the “reasoning” it produces. > The ability of LLMs to produce “fluent nonsense”—plausible but logically flawed reasoning chains—can be more deceptive and damaging than an outright incorrect answer, as it projects a false aura of dependability. That isn't to say LLMs aren't useful, just exploring it's boundaries. To use legal services as an example, using an LLM to summarize or search for relevant laws, cases, or legal precedent is something it would excel at. But don't ask an LLM to formulate a logical rebuttal to an opposing council's argument using those references. Larger models and larger training corpuses will expand that domain and make it more difficult for individuals to discern this limit; but just because you can no longer see a limit doesn't mean there is none. And to be clear, this doesn't diminish the value of LLMs. Even without true logical reasoning LLMs are quite powerful and useful tools. [1] https://en.wikipedia.org/wiki/Logical_reasoning https://en.wikipedia.org/wiki/Logical_reasoning
- dawnofdusk 1y ago>but we know that reasoning is an emergent capability! This is like saying in the 70s that we know only the US is capable of sending a man to the moon. Just because the reasoning developed in a particular context means very little about what the bare minimum requirements for that reasoning are. Overall I am not a fan of this blogpost. It's telling how long the author gets hung up on a paper making "broad philosophical claims about reasoning", based on what reads to me as fairly typical scientific writing style. It's also telling how highly cherry-picked the quotes they criticize from the paper are. Here is some fuller context: >An expanding body of analyses reveals that LLMs tend to rely on surface-level semantics and cluesrather than logical procedures (Chen et al., 2025b; Kambhampati, 2024; Lanham et al., 2023; Stechly et al., 2024). LLMs construct superficial chains of logic based on learned token associations, often failing on tasks that deviate from commonsense heuristics or familiar templates (Tang et al., 2023). In the reasoning process, performance degrades sharply when irrelevant clauses are introduced, which indicates that models cannot grasp the underlying logic (Mirzadeh et al., 2024) >Minor and semantically irrelevant perturbations such as distractor phrases or altered symbolic forms can cause significant performance drops in state-of-the-art models (Mirzadeh et al., 2024; Tang et al., 2023). Models often incorporate such irrelevant details into their reasoning, revealing a lack of sensitivity to salient information. Other studies show that models prioritize the surface form of reasoning over logical soundness; in some cases, longer but flawed reasoning paths yield better final answers than shorter, correct ones (Bentham et al., 2024). Similarly, performance does not scale with problem complexity as expected—models may overthink easy problems and give up on harder ones (Shojaee et al., 2025). Another critical concern is the faithfulness of the reasoning process. Intervention-based studies reveal that final answers often remain unchanged even when intermediate steps are falsified or omitted (Lanham et al., 2023), a phenomenon dubbed the illusion of transparency (Bentham et al., 2024; Chen et al., 2025b). You don't need to be a philosopher to realize that these problems seem quite distinct from the problems with human reasoning. For example, "final answers remain unchanged even when intermediate steps are falsified or omitted"... can humans do this?
- guybedo 1y agolots of interesting comments and ideas here. I've added a summary: https://extraakt.com/extraakts/debating-the-nature-of-ai-reasoning-toy-models-language-and-the-human-mind https://extraakt.com/extraakts/debating-the-nature-of-ai-rea...
- tomhow 1y agoIs Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens - https://news.ycombinator.com/item?id=44872850 https://news.ycombinator.com/item?id=44872850 - Aug 2025 (130 comments)
- guluarte 1y agoI would call it more like prompt refinement.
- doku 1y agoDid the original paper show that the toy model was fully grokked?
- baby 1y agoMy take on that is that it's a way to bring more relevant tokens in context, to influence the final answer. It's a bit like RAG but it's using training data instead!
- hannasm 1y ago> these papers keep stapling on broad philosophical claims about whether models can “really reason” that are just completely unsupported by the content of the research. From the scientific papers I've read almost every single research paper does this. What's the point of publishing a paper if it doesn't at least try to convince the readers that something award worthy has been learned? Usually there may be some interesting ideas hidden in the data but the paper's methods and scope weren't even worthy of a conclusion to begin with. It's just one data point in the vast sea of scientific experimentation. The conclusion feels to me like a cultural phenomenon and it's just a matter of survival for most authors. I have to imagine it was easier in the past. "Does the flame burn green? Why yes it does..." These days it's more like "With my two hours of compute on the million dollar mainframe, my toy llm didn't seem to get there, YMMV"
- Terr_ 1y agoA regular "chatting" LLM is a document generator incrementally extending a story about a conversation between a human and a robot... And through that lens, I've been thinking "chain of thought" seems like basically the same thing but with a film noir styling-twist. The LLM is trained to include an additional layer of "unspoken" text in the document, a source of continuity which substitutes for how the LLM has no other memories or goals to draw from. "The capital of Assyria? Those were dangerous questions, especially in this kind of town. But rent was due, and the bottle in my drawer was empty. I took the case."
- areeh 1y agoOh wow, now I want a chain of thought rewriter that makes the combination of chat and CoT put together follow this style
- Terretta 1y agoClick show details in chatgpt deep research for such a tale except instead of noir it's "Office Space".
- Terr_ 1y ago[P.S.] I realize I left off an important part: If the regular version did not already "reason"... Why would we ever expect this kind of tweak to change that, and bright forth real reasoning? The core algorithm hasn't really changed, we're just changing the (hidden) document so that it's a different style with a greater density of clues, so that it can more-effectively bullshit [0] output humans won't notice and dislike. [0] Creating something that "sounds good" without any particular awareness or care about truth or falsehood.
- RainyDayTmrw 1y agoThe steelman argument - not to say that I agree with it - is that this capability was present all along, but due to variability in the behavior of the model, adding certain context, such as chain-of-thought, exposes it.
- ForHackernews 1y agoDidn't Anthropic show that LLMs frequently hallucinate their "reasoning" steps? > Bullshitting (Unfaithful): The model gives the wrong answer. The computation we can see looks like it’s just guessing the answer, despite the chain of thought suggesting it’s computed it using a calculator. https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot https://transformer-circuits.pub/2025/attribution-graphs/bio...
- jongjong 1y agoYes, CoT reasoning is a mirage. What's actually happening is that we've all been brainwashed by Facebook/Meta to be hyper-predictable such that whenever we ask the AI something, it already had a prepared answer for that question. Because Meta already programmed us to ask the AI those exact questions. There is no AI, it's just a dumb database which maps a person ID and timestamp to a static piece of content. The hard part was brainwashing us to ask the questions which correspond to the answers that they had already prepared. Probably there is a super intelligent AI behind the scenes which brainwashed us all but we never actually interact with it. It outsmarted us so fast and so badly, it left us all literally talking to excel spreadsheets and convinced us that the spreadsheets were intelligent; that's why LLMs are so cheap and can scale so well. It's not difficult to scale a dumb key-value store doing a simple O(log n) lookup operation. The ASI behind this realized it was more efficient to do it this way rather than try to scale a real LLM to millions of users.
- dizlexic 1y agoHow can a statistical representation of reason ever be reason itself?
- heresie-dabord 1y agoUsers of Eliza [1] could convince themselves that the programme was intelligent, much to the dismay of Weizenbaum. It is no surprise that people will claim that more complex (and poorly understood) applications are intelligent. If our own salesbabble and technology can be used to bamboozle us and defeat our willingness to understand, we have fully regressed to credulity and Carl Sagan's state of captured idiocy. [2] [1] https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect [2] If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. -- C.S. , Demon-Haunted World
- dizlexic 1y agoA tangent, but I remember having a version Eliza on our family Macintosh SE as a kid, and my brothers and I would spend way too much time typing inappropriate things trying to get a silly response. I think you have a very important point, and I'm amazed I've never heard of the Eliza Effect.
- zahlman 1y ago> However, people keep asking the least interesting question possible Am I the only one who finds this a really strange turn of phrase? If people continually feel compelled to do something out of curiousity, is that not the very definition of "interesting"?
- bestspharma 1y ago[dead]