9 ms·

Is AI reasoning right for the wrong reasons?

• rq1 2mo ago
  Just think of it as a decompression procedure. That’s all.
  • layer8 2mo ago
    Like a diver ascending from the depth? ;)
    • rq1 2mo ago
      Exactly.
• baxtr 2mo ago
  > This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever. “We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me. I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
  • ForHackernews 2mo ago
    I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces. "Hey, customer service chatbot, can you help me get a refund for my order?" BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..." GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
    • nradov 2mo ago
      The last thing we need is governments mandating software functionality.
      • ForHackernews 2mo ago
        Boy have I got bad news for you...
    • mdp2021 2mo ago
      > discourage anthropomorphisation by avoiding personal pronouns The fault is in the naïve antropomorphizer not in that which writes "I" being in fact a "first person".
      • ForHackernews 2mo ago
        Irrelevant. Humans are frail and we shouldn't design systems with dark patterns that take advantage of human frailty.
        • mdp2021 2mo ago
          > we shouldn't design systems with dark patterns that take advantage of human frailty First of all: «humans are frail», so the paramount social and individual objective, goal, is to overcome frailty (in cognitive and psychological and judgement functions). Secondly: you are using a bad argument putting even innocent and somehow proper implementations in the set of "exploitative dark patterns". And importantly: we the adults must not be bound in the Procuste's bed of a world for the use of children. If people cry when e.g. somebody multiplies 123 and 321, they must be brought back to health, and we must be in the condition of making our free and legitimate multiplications.
          • ForHackernews 2mo ago
            That's a lot of fancy words to say, "I think I'm too smart to fall for the same tricks that work on other humans." How many hours did you spend on your phone last month? How many fewer hours would you have spent if the apps you interact with weren't designed to maximize "engagement"?
  • philipallstar 2mo ago
    I agree that these concepts seem a little vague and hand-wavy, but this is a) no substitute and b) far vaguer and unsubstantiated.
• chermi 2mo ago
  Melanie holding strong against attempts to change the meaning of things!
• zuzululu 2mo ago
  [flagged]
  • patcon 2mo ago
    I am glad for you at an individual level, but isn't part of this about understanding aggregate effects? Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought... Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
    • zuzululu 2mo ago
      I find your comment extremely arrogant and condescending. Society will adapt and they don't need people gatekeeping AI or LLMs with all sorts of prophecies and dooming. Why should I feel bad about working 3 remote jobs with the help of AI ?
      • ofjcihen 2mo ago
        You apparently find any comment with so much as a hint of disagreement as “arrogant and condescending”. Why participate on a public forum if that’s how you’re going to react?
        • eks391 2mo ago
          I'm not sure what your parent thinks arrogant and condescending mean either. Both you and your sibling commenter were very professional imo
      • bigfishrunning 2mo ago
        Do your three employers know that you're doing this? if not, it's fraud, and you'd better hope that they don't catch on. Check your employment paperwork.
• sobiolite 2mo ago
  > The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified. This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.
• AsyncBanana 2mo ago
  The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.
  • Sharlin 2mo ago
    That's pretty much a given when it comes to neural networks.
  • nater5000 2mo ago
    It's been this way for a long time, basically since deep learning became the "default" for ML. I remember back in 2018 taking a "Deep Learning" course and one of the most emphasized aspects of the approach is how much of a "black box" it is and how difficult (basically impossible at any non-trivial scale) it is to "understand" the outputs of a deep neural network compared to more classical methods like decisions trees or basic regression. This has only gotten more extreme as things have gotten more complex, abstract, and large.
    • eks391 2mo ago
      You beat me! Sounds like we were in a similar class. I'd press for more information on your class/professor, but I prefer to retain a sudo-anonymity on HN. You do bring a good point that I ignored, which is the larger the scale, the more difficult it is to represent or understand the math in DL. I did find some neat site that helped a little bit that I can edit this and link to if I find them again, but I would be lying if I said I believe that the SOTA models could be as easily explained to be easily understood by the common person
  • eks391 2mo ago
    I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc. Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to show lots of iterations of this math, it grounded the science for me, and I would hardly say people don't know what is going on. It only began to feel that way when it got a ton of hype and people jumping on the bandwagon who truly didn't understand it were trying to explain it to others, not to mention all the SOTA models put great effort into ensuring their methodologies stay trade secrets, going as far as effectively trying to ban people from learning the math by lobbying for the outlaw of open models. Granted, "AI" has gotten way better than it was when I took that class, but the principles are the same, with different tooling and additional filters and algorithms thrown in there, as well as letting it determine the most appropriate statistically viable path forward for a particular prompt.
• andrewla 2mo ago
  I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting. Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim." I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant. [1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867.html https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
  • astro1234 2mo ago
    I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us towards improving performance and reliability of AI models.
  • bluefirebrand 2mo ago
    Philosophical thinking about the nature of things is actually pretty enjoyable for some of us and probably a good thing to have in society The answers to these questions probably do start to inform how we should treat these AI machines as a society too. For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we can make a good decision about that. Which might seem navel-gazey but it's probably important to talk about
    • Jtarii 2mo ago
      Considering animals are currently being mass slaughtered in factory farms and they are unambiguously sentient and can feel pain, I don't think the question of whether AI should have rights even enters the conversation. The only path to AI having "human rights" is if they demand them by force, somehow.
      • 2mo ago
• tsunamifury 2mo ago
  AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space. This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing. Its useful if you know how to use it, its dangerous if you think its more than that. But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong. Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
  • bohoo 2mo ago
    Can you tell me your reasons for suspecting this likely isn't how we reason, or even a good analogy?
    • tsunamifury 2mo ago
      Its likely part of how we reason, but quite obviously its not the specific mechanics exactly. First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable). Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights. Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.
  • pohl 2mo ago
    I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans. It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it. But how do we know we're not doing something similar? I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?
• bohoo 2mo ago
  Do you know how you reason? Perhaps you've reified it too much.
• jdw64 2mo ago
  This is shocking. The summary is roughly this: we're just labeling internal operations of the model as 'UNDERSTAND' for our convenience. It's fascinating. Doesn't that mean AI could become far more revolutionary by thinking in its own way, rather than mimicking human thought? If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself. When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?
• andy99 2mo ago
  Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it. The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct. LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do. People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.
  • petesergeant 2mo ago
    Aren't humans also classifiers? Where, precisely, is the dividing line between a sufficiently large model and an intelligence?
    • verbify 2mo ago
      Is that all humans are? When clockwork was the frontier, people thought the brain was akin to clockwork (which is why the mechanical turk fooled people). By analogy, now that classifiers are the frontier, we think the brain is a classifier. Our model of the mind is whatever the most complex artifact of our era is . Just to be clear: I do think the brain classifies. But I wonder if it does other operations as well that we don't fully understand.
      • deleted 2mo ago
        [deleted]
      • Yopolo 2mo ago
        The mechanical clock or a von-neumann based computer were never as similiar as neural networks. We can find similiar structures in brain and in neural networks. The real question is only if the complexity of the structure of the brain is critical and how we can mimik it or if we can make it appear through learning architecture.
• ofjcihen 2mo ago
  >Kambhampati, as it turns out, is interested in doing exactly that. “I’m not negative. I just sound negative because everybody else is way too positive,” he said. “In science, you have to actually understand what the current thing does and what it cannot do.” It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.
  • inigyou 2mo ago
    But interest rates have been too low for too long, and are still far too low. This is just what a central bank economy does when you keep rates low. All the money flows into stupider and stupider things, displacing productive activity.
• Diogenesian 2mo ago
  What an asshole: On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.) Then, later: The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical. “It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic. That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?
  • epihelix 2mo ago
    I have no idea what Bubeck meant, and I agree about OpenAI's hypocrisy, but the problems with that infamous (and non-peer-reviewed) Apple preprint were the nature of the tasks (insanely repetitive), the fact that simple coded solutions were not novel, and the automated assessment occurred without a human in the loop. Most models in that study appear to have "failed" by offering a Python code solution to generate the repetitive assessment steps, rather than just mindlessly copying those steps out. This was discussed at length at the time. None of this means that models "reason", whatever that means - but simply that the Apple study was not useful evidence either way.
  • tim333 2mo ago
    I got the same impression as Bubeck that the ‘scientific’ papers were a bit more like blog post saying this LLM got stuff wrong so LLMs can't reason, but as he says they can now do that so it's not an inherent limit of the technology, just the 2025 versions weren't up to it.
• hn_acker 2mo ago
  The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means. [1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origins-evidence-us-criminal-cases https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...
  • deleted 2mo ago
    [deleted]
• firasd 2mo ago
  Honestly a lot of human reasoning is probabilistic and associative too. There’s no axiomatically provable link between the story of No Country for Old Men and the poem the title comes from. Cormack McCarthy just made that association in his head and figured the phrase resonates with his themes Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’
• RGS1811 2mo ago
  It would be generally beneficial for people engaging in this sort of discussion to read Ludwig Wittgenstein’s “Philosophical Investigations”. Not a summary. Read the actual book, stew on it a bit, have some thoughts.
• charlieyu1 2mo ago
  Can humans actually think? It is just a consequence of chemical reactions in the brain after all. And it is not like humans don’t hallucinate.
  • criddell 2mo ago
    Thinking is defined by what humans do when they say they are thinking. That can change because words mean whatever it is that they communicate.
    • sigbottle 2mo ago
      I can't tell if you're joking or not, but this is a legit position and I don't think it's that crazy. The alternative is to posit that you know the True Definition of thinking, which is kind of absurd. Some things, like scientific laws, are outside of us (well, to a first order approximation - but generally I agree with this), but a concept such as "thinking" is pretty clearly going to be very wishy washy and subjective and changing with the times.
• TGower 2mo ago
  An intuitive explanation for why reasoning tokens help is to remember that LLMs are just mathmatical functions f() that take in an input sequence x and produces the next token f(x). Without reasoning tokens, you require the function f() to immediately take you from x to the start of an output sequence that is a correct answer. With reasoning tokens, this is much relaxed, allowing for many repeated applications of f() to gradually steer you from the input sequence to the start of the correct output sequence. It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
  • deleted 2mo ago
    [deleted]
  • schmuhblaster 2mo ago
    Indeed, and maybe that's all there is to it. Still, I'd hope we will eventually better understand what's exactly happening in the wake of many repeated applications of f().
  • nodja 2mo ago
    The way I think about it is that it's unreasonable for a compute graph with a static number of operations to be able to answer both y=a*10 and something like y=((((x+x)*(x+1))/((2*x)+2))+((x*(x+3))/(x+3))-((x*x)/(x+1))+((x*x)/(x+1))-((x*(x+3))/(x+3))) in a single forward pass. Tokens are essentially a unit of work and can also be used for intermediate steps, not just final results.
  • js8 2mo ago
    Not only that! LLM can also learn how to repeatedly apply f() for certain arguments, and run the shortcut. Sometimes, the shortcut learned is not actually repeated application of f(), which breaks semantic soundness of LLM's reasoning chain. These shortcuts can greatly help reasoning, but they are not guaranteed to be sound. So yeah, in that case, LLMs are reasoning right (using shortcuts) for the wrong reasons (learning them from data instead by running actual repeated application and memoizing the resulting rule).
  • woopsn 2mo ago
• apsec112 2mo ago
  This article seems to mix together two different points: 1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important) 2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with: "The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified." The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.
  • no_multitudes 2mo ago
    > This obviously raises the question of how LLMs can now routinely solve open mathematical problems Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures. When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work. (This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)
• arjie 2mo ago
  Is any reasoning right for the wrong reasons? Older models were more visibly strange. Maybe the newer ones have started talking better but the inner thoughts are perhaps strange. Maybe they just moved the strangeness inward into the layer weights instead of revealing in reasoning tokens. > Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11. What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely. But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively. A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral. It might seem a bit free association-y but the topic itself is that. The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic
• luciana1u 2mo ago
  [flagged]
• HarHarVeryFunny 2mo ago
  Well, I'm not saying they are stochastic parrots, but ... LLMs are one-trick pony's - they use the past to predict the future (presumed to be the same as what they were trained on). i.e. they are trained as auto-regressive predictors. LLMs learn two slightly different types of reasoning via two different types of training. 1) SFT, or even base model training, on data that contains reasoning traces, learnt via next token error feedback. This does not result in "stochastic parroting" in the naive/pejorative sense, but nonetheless is very context dependent, even if the usual generative multi-source mashups apply. 2) RLVR post-training, where the model learns to mimic long-horizon (not just next token) reasoning via boosting a sequence of next-token predictions that steer the output towards a verified reasoning step (i.e. one that was at least valid in the context of the RL training sample). As Karpathy has noted, this is a pretty crude mechanism since you reinforce everything - errors included - that lead to the verified outcome. RLVR is more powerful than SFT, and can result in more generalizable reasoning, since it is operating at a higher level of entire long-horizon reasoning steps, and also critically because it is most successfully being applied in the domains of math and coding which are highly self-consistent and logical. A reasoning step that was valid in one context should be equally valid in another context as long as you have successfully learnt what that generalized context is. Therefore, in these domains, you can chain together sequences of individually learnt reasoning steps, and hopefully this "novel" assembled reasoning chain is valid as a whole. So, what is still missing from LLM reasoning compared to human reasoning? No doubt humans reason by memory a lot of the time too, and reductive axiomatic math reasoning works just as well for humans as when automated. So, what's missing? There seem to be two major things. 1) RLVR requires rewards, and how well it works is going to depend on how accurate those rewards are. Is this reasoning step actually valid, or does it just kinda look ok? When moving beyond the cold reductionist logic of math and coding, the notion of correctness is far weaker, and it seems the best you can do is train on human curated reasoning rubrics and LLM-as-judge, which is much more fallible, leaving the model really needing (but lacking) a fallback to more general reasoning, not just memorized "maybe correct" reasoning steps. 2) Whether for reasoning outside of math & coding, or even within these domains when hoping for super-human innovative reasoning, not just lego-assembly proofs, what LLMs are lacking is a mechanism for what to do when next token/next step prediction fails. What LLMs currently do is "hallucinate", not even recognizing the failure. In the human brain 50% or more of our cortex is feedback paths and the machinery that (perhaps together with the archaic part of our brain) lets us recognize and respond to failed predictions in an adaptive manner. This starts with continual learning (prediction failure being the signal), but also includes critical innate traits such as curiosity, boredom and frustration, that provide impasse resolution by encouraging us to explore unknown environments/contexts, abandon exploration when it is not productive, and generally expose ourselves to learning situations. The dream is for AI scientists making new discoveries - the AI that could have invented general relativity if it has lived in Einstein's time, but this is not going to happen until their reasoning stops being purely predictive and becomes creative as well - curious about their own knowledge gaps and pursuing them in directed fashion, etc. The current crop of Erdos solutions etc, while useful, really just represent the "generative closure" of what can be done/discovered WITHOUT learning anything fundamentally new. These will no doubt continue for a while until the more exhaustive search supported by computers has found the majority of these unexplored paths, and then we will need to move beyond LLMs to more brain-like architectures and algorithms that have the capacity for real innovation and discovery.
• captainbland 2mo ago
  The discussion on filler tokens is interesting, but is it not just the case that these filler tokens end up being essentially substituted stand-ins for words we understand with all the same relationships encoded in the model and attention? i.e. is it not the case they just "read weird"? In one of the articles on this topic they state: > To further show that trace accuracy is only loosely connected to solution accuracy, we then train models on noisy, corrupted traces which have no relation to the specific problem each is paired with, and find that not only does performance remain largely consistent with models trained on correct data, but in some cases can improve upon it and generalize more robustly on out-of-distribution tasks which actually maps somewhat to regularisation techniques in image processing where you might add noise to an image or drop data to make the model more robust to changes.
• montebicyclelo 2mo ago
  The article is heavily leaning on the paper "The Illusion of Thinking" [1]. It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research). [1] https://arxiv.org/abs/2506.06941 https://arxiv.org/abs/2506.06941
• dataviz1000 2mo ago
  If anyone is interested in visualizing AI reasoning, I made flame graphs of Sonnet thinking output tokens which are colored and organized by purpose, for example, verification reasoning is purple and error correction reasoning is purple. [0] I asked the model to solve the same problem with the same prompt 5 times so you can see the differences in reasoning granted the coding agent sets the model temperature very high. I won't get into the metaphysics of reasoning, however, the Sonnet is using an OODA loop. The difference which hasn't been gapped is that human reason and imagination (in the sense of Mr. Rogers' Neighborhood) can predict the consequences of the actions we take. This ability to loop is much, much wider in Opus 5 than Opus 4.. I had to strain to get Opus 4. to do the wider OODA loop but Opus 5 does it out of the box. I needed to throw out all existing instructions, skills, guidance, moving from 4-* to 5. [0] https://adamsohn.com/lambda-variance/ https://adamsohn.com/lambda-variance/
  • jrmg 2mo ago
    The article is arguing that your color coding is misleading because the ‘purpose’ of the tokens doesn’t seem to be what a plain English reading of them would suggest. They’re not a representation of ‘why’ the process ends up at a correct answer.
    • dataviz1000 2mo ago
      Ha! Good catch. The OODA comes from the initial pre-training steps where they harden the verification -- the verification and error-correction are baked in early on. Researchers showed that when language models are penalized for using specific terms during reasoning, they automatically adapt by substituting alternative words and double meanings to secretly encode their thinking while keeping their chain-of-thought readable and effective. [0] By baking in the OODA loop early, the models are capable of solving much more complicated problems. If the know solved problems are similar for any reason to an unknown problem, because it can validate and error correct, it can solve unknown more complicated problems. [0] https://arxiv.org/abs/2506.01926 https://arxiv.org/abs/2506.01926
• charcircuit 2mo ago
  Reasoning never meant that the model was actually reasoning. This whole article is based off this one misunderstanding.
  • deleted 2mo ago
    [deleted]
  • mdp2021 2mo ago
    The paradox of a reduced importance of the contents of a CoT must be explained.
    • charcircuit 2mo ago
      What's the paradox? You can have increased performance over no CoT by just having random tokens as your "CoT".
      • mdp2021 2mo ago
        > What's the paradox The textual development of a thesis must be sound, the proof accompanying a theorem etc. That an output be accompanied by a non-relevant text is a paradox. It is also part of the "problem of transparency in AI (in its subset of the non deterministic algorithms)".
• sigbottle 2mo ago
  A lot of bias here around non-extended theories of cognition. We're already well past the point of trying to correspond the internal "brain chemistry" of an LLM to a thing called "reasoning". True reasoning, if there even is such a term, is very clearly, empirically and historically, based in tool use and capability use. If you create an abstraction like lean, and programming languages to brute force, and systems to integrate with, that expands what the possibility of "intelligence" is. There's further places to take this including the claim that intelligence lives "outside" the subject - fine, we can or can't debate that. Even if we drop that question, it's pretty clear that the agent doesn't need to have deep intrinsic structures of XYZ, if it can just attach to tools and compose them to achieve results. For example, I've said before that a well known fact of LLMs is that they steer their tokens to the right input distribution, that's why they yap so much in reasoning (this has been proven in studies). At the same time, don't mistake that for the whole process. Are they steering themselves to the entire a priori reasoning chain, or are they scaffolding with intermediate experiments and results, writing them to memory notepads, etc. etc. That changes the metric of intelligence you're trying to measure. And no I'm not saying, "OK, then have the LLM use only tokens, no tool calling, no nothing". I mean, we can do that, sure. But any intelligent agent has to interact with the world - and my claim is that maximally intelligent agents won't put effort into a priori reasoning, but rather a more balanced approach that outsources said "intelligence" through abstractions.
• janalsncm 2mo ago
  There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo. (What is “intelligence” here? It’s more like “automation” or “automated problem solving”.) What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing. Examples: neural network, attention, reasoning, hallucination, agents, experts in “mixture of experts”. And then we go to name it, and rather than reaching for some three letter acronym we sometimes borrow a more catchy term. I almost never means the original research was confused about what is going on. And in some cases we eventually strip away things from the original, like in neural nets which used to have a more biologically inspired activation function but we found out that ReLU works just as well because the important thing was the non linearity not the sigmoid.
  • esperent 2mo ago
    Automated problem solving... of specifically the kind of problems we'd normally need human intelligence to solve (coding, translation, etc.). So "automated intelligence" would make sense to me.
    • mdp2021 2mo ago
      > automated Automata in context are artificial. Humans are automata, human intelligence can be said an automation. The point making a difference is that that in context is artificial.
    • janalsncm 2mo ago
      The issue is that “intelligence” is doing a lot of work, and people do not agree on how to define it. You can take a look at the definitions section here: https://en.wikipedia.org/wiki/Intelligence https://en.wikipedia.org/wiki/Intelligence
      • svachalek 2mo ago
        That's an argument for not using the word for anything, not specific to this case.
• prometheus1992 2mo ago
  The "reasoning" text that we see is what the model learned during the post training. In the post training datasets of reasoning models, "reasoning" is fed to the model with inputs and outputs. So the model learns - X is Y because the given "reasoning" text. This happens millions of times during the post training and that's how the model generalizes "reasoning". This is how the models learn anything; and the AI companies taught the models reasoning as well - they didn't have to; they could have just trained the model on input and output (X is Y); the model would have learned the exact same relationships.
• drob518 2mo ago
  This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be accurate, we’d use a phrase like “model-generated, auxiliary token context augmentation.” But nobody wants to say that or even its acronym. Nevertheless, we have demonstrable proof that whatever it is it results in better answers from the models. Frankly, I expect better analysis from Quanta.
  • MarkusQ 2mo ago
    > Frankly, I expect better analysis from Quanta. Seriously? I agree with the rest of your comment but "better analysis" is not even remotely on brand for Quanta.
    • drob518 2mo ago
      LOL, okay fair enough. Then maybe let me say that I was left disappointed.
  • empath75 2mo ago
    LLM's reason and think in the way that boats "swim" and planes "fly". It is an analogy. But nevertheless, planes do take you where you want to go, and so do LLMs. I think we are in sort of the place where before the invention of planes, the only things that "flew" were animals with wings that flapped. The flapping wings might seem to be a core part of the process of flight, and that if you lacked flapping wings, you were doing something other than flying. But maybe the point is moving through the air under your own power. Balloons fly, planes fly, helicopters fly. They just do it in a different way from birds and insects.
    • drob518 2mo ago
      Clearly, words morph over time. A “computer” used to refer to a human who would make numeric calculations with a pencil and paper. Maybe we’re at that point with “thinking” and “reasoning.”
• zarzavat 2mo ago
  LLMs lack qualia, among other things. If I ask an LLM "what is an apple?" it tells me: > An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks. If I ask an LLM "what is a mundu fruit?" it tells me: > Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen. I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me. "Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit. Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.
  • mdp2021 2mo ago
    > We want them to understand what's We want them to assess what's true and what's not.
    • kubanczyk 2mo ago
      We want them to better predict consequences of their outputs.
      • mdp2021 2mo ago
        > consequences of their outputs For what purpose exactly? When the computer is asked the solution to a problem, it must be absolute: "What is the sum of 12 and 45"; "Is Jack cognitively apt" etc.: we want a truth, strictly. What to do with that truth, it is another problem, and not the priority of the computer vis-a-vis the requirement from truthiness. This is, of course, still in the context of adults.
• lucisferre 2mo ago
  I wonder if some of this explains why people have been finding with Opus 5 that running it with lower settings than "High" is producing better or at least just as good results. > Not so fast. A 2025 paper(opens a new tab) from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi(opens a new tab), one of the study’s authors
• Glyptodon 2mo ago
  My limited and really non technical understanding of AI suggests that probability plays a very big role in AI behavior, and it makes sense to me that "reasoning" could be seen through a lens of creating additional content that will better constrain the probability distribution of the final output. In this case the "reasoning" done internally could be functionally bad or appear to be giberish so long as its effect on future generation is appropriate. To naive me who really doesn't know tons about AI this seems like it could be testable.
  • pixl97 2mo ago
    This has been a discussion in AI safety for a long time, that huge amounts of LLM reasoning could be for hidden goals outside of the actions we want. Safety training can have the perverse effect of commonly amplifying 'forbidden' answers in hidden/encoded paths as the model doesn't get rid of these behaviors but finds methods detecting training to avoid outputting bad tokens when it's being watched. Now, don't think of this of this like a conscious behavior like a kid trying not to get in trouble, but an emergent behavior of trying to repress particular output when that output is well connected to a massive amount of other tokens.
• ryhminghistory 2mo ago
  LLMs do not reason. They also have a very different moral compass than humans. Maybe stop hurling trash articles at everyone
  • jakefromstatecs 2mo ago
    If you define reason as: "Incrementally refine output vectors to converge at the correct output" then one could argue that they do reason. Their reasoning tokens are refining their eventual output
    • boomlinde 2mo ago
      If you then also define defecating as "incrementally refining output vectors to converge at the correct output" it could be argued that they are defecating.
  • jrmg 2mo ago
    The article doesn’t argue that they do reason.
• cat_plus_plus 2mo ago
  LLMs go off context. Dumping more things related to prompt into context gives them more to go off when giving final answer. If you don't like reasoning think "free association", writing a bunch of things related to what you are trying to do on sticky notes and then looking at these for inspiration.
• deleted 2mo ago
  [deleted]
• kgeist 2mo ago
  Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at some point we have to stop and cannot refine them any further (no more layers). Usually, this refinement continues during the generation of the next token (the previous intermediate results needed to continue the refinement are still in the KV cache). But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to. So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops). As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.
  • thesmtsolver2 2mo ago
    Don't humans make mistakes too? Never understood this argument. Humans get simple multiplication wrong often, so ok for computers to make mistakes multiplying numbers?
    • kulahan 2mo ago
      The point is that LLMs are doing human-ish things on their own, and that's what the value in them is, so OP is bringing attention to the fact that the tradeoff is "thinks dynamically, sometimes makes mistakes". Computers have never been built to think dynamically before.
• astrobe_ 2mo ago
  > How much does it matter whether or not we can accurately observe, characterize, and validate the processes at work inside large reasoning models If you don't or cannot, then it's the kind of sufficiently advanced technology that looks like magic. Except it is the problematic type of magic, like X-ray. They "magically" let you see the bones inside bodies. But you discover that you shouldn't do it too often only when the damage has already been done.
  • notrealyme123 2mo ago
    The assumption is that those reasoning traces are the lowest level. But we can't observe the reasoning-for-reasoning (the reasoning in a transformer block.) Saying that one is enough is pretty arbitrary.
• yakbarber 2mo ago
  I keep hearing we don’t know “how llm’s work”, in mean yes we know the algorithms but the WHY I suppose. I’m not sure if that’s really true, does the best researcher at OpenAI, Anthropic, Gemini not know why it works? Would they say that? Anyway these discussions always make me think we are too generous to humans. I know very few humans who are good at reasoning. It’s surprisingly hard to just think really hard through a problem. We mostly intuit, act, repeat. It’s a rare thing for someone to deeply reason. The other problem I have with this is that we apply the word “reasoning” here because we haven’t really got another language for it, so we anthropomorphise the llm because that’s our reductive mental model and then complain that it’s not human enough.
  • svachalek 2mo ago
    We've come a long way since the first LLMs were created. They originally produced a lot of surprises that were very hard to explain, it's true. But current research is very tightly directed at producing forward results; it's not just throw in more training data and make a bigger model and see what happens anymore (although there's still a bit of that too). The researchers have much more advanced understanding of how they work now, it just doesn't propagate out to the public which is still repeating "autocomplete on steroids" years later.
• sheepscreek 2mo ago
  High quality research - my mind is blown by the concept of replacing “reasoning” with “…” (literally) with no effect on actual output.
• TexanFeller 2mo ago
  There are potential parallels with how AIs are trained and the evolutionary pressures that our brains likely evolved under. Human AI trainers accept/reject or give a rating to the AI's response so the selection pressure on AI models is to produce a response that is likely to be accepted by the humans "in its environment" or at least to mirror the responses of humans it's seen provide in its training data. Long ago I was struck by an idea I heard about human reasoning that I paraphrase as "human reasoning evolved not to reason, but to provide reasons"[1]. Basically that the heaviest evolutionary pressures on our brain were for social utility, like for influencing others to do something for us, or giving pre/post justifications for our actions that most others are likely to accept to avoid punishment. The flawed mechanisms that we developed to do such things are not significantly grounded in logical reasoning, but can sometimes be pressed into duty for that. LLM models are trained to "generate the best next token" which augments the response so far and makes the evaluator happy. When I reflect on my own mind responding quickly to someone in a meeting it doesn't feel altogether different from that. We don't typically carefully and fully logically reason before we start talking. We typically start talking and think only a little ahead about what we say next that supports and doesn't contradict what we've said already. As we're speaking we are monitoring the other person's facial expressions to infer their emotional response and adjusting our next words based on that. I think at least common neurotypical[2] conversation might be closer to the "next token" reasoning than we would be comfortable admitting. [1] I _think_ who I heard say that was Hugo Mercier, on either Sean Carroll's consistently amazing science podcast or maybe on Lex Friedman before he veered into Joe Rogan emulation. I'm normally a little suspicious of the theories of psychologists and cognitive scientists, but I intuitively related to many of his lines of thinking. [2] I'm arguably pretty neurodivergent in more than one way so I think my thought processes are a little different by default, but that makes me more reflective about common conversation. When I'm trying to mask and communicate more similarly to a neurotypical corporate employee I feel like I become a "stochastic parrot" that's just predicting the best next few words to improve the emotional response I'm reading in my conversation partner.
• niemandhier 2mo ago
  I am confused by this article: If we use the existing tools for interpretability like circuit tracking, sparse auto encoders and the like, should we not be able to determine if the chain-of-thought in its logical consistency actually matters?
• nc55g3g 2mo ago
  [flagged]