13 ms·

The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]

• behnamoh 1y ago
  Okay Apple, you got my attention. But I'm a strong proponent of "something is better than nothing" philosophy—even if OpenAI/Google/etc. are building reasoning models with the limitations that you describe, they are still a huge progress compared to what we had not long ago. Meanwhile you're not even trying. It's so easy to criticize the works of others and not deliver anything. Apple—be Sam in Game of Thrones: "I'm tired of reading about the achievements of better men".
  • suddenlybananas 1y ago
    I think you're mistaking the work of researchers who work at Apple with the particular investment decisions of Apple over the past few years. >It's so easy to criticize the works of others and not deliver anything. Apple—be Sam in Game of Thrones: "I'm tired of reading about the achievements of better men". This is a patently absurd thing to write about a research paper.
  • bwfan123 1y ago
    there is enough hype already - with AGI being promised as imminent. this work balances the hype and shows fundamental limitations so the AI hypesters are checked. why be salty ?
• farts_mckensy 1y ago
  [flagged]
  • Terr_ 1y ago
    Equivalent: "I'm so sick of these [atheist] cretins proclaiming that [god doesn't exist], and when you drill down to first principles, these same people do not have a rigorous definition of [god] in the first place!" That's nonsense, because the people obligated to furnish a "rigorous definition" are the people who make the positive claim that something specific is happening. Also, the extraordinary claims are the ones that require extraordinary evidence, not the other way around.
    • farts_mckensy 1y ago
      Your god analogy is clumsy in this case. We aren't talking about something fantastical here. Reasoning is not difficult to define. We can go down that road if you'd like. Rather, the problem is, once you do define it, you will quickly find that LLMs are capable of it. And that makes human exceptionalists a bit uncomfortable.
      • Nevermark 1y ago
        "It's just statistics!", said the statistical-search survival-selected statistically-driven-learning biological Rube Goldberg contraption, between methane emissions. Imagining that a gradient learning algorithm, optimizing its statistical performance, is the same thing as only learning statistical relationships. Inconvertibly demonstrating a dramatic failure in its, and many of its kind's, ability to reason.
        • farts_mckensy 1y ago
          Reasoning exists on a spectrum, not as a binary property. I'm not claiming that LLMs reason identically to humans in all contexts. You act as if statistical processes can’t ever scale into reasoning, despite the fact that humans themselves are gradient-trained statistical learners over evolutionary and developmental timescales.
          • Nevermark 1y ago
            I think you might re-read what I wrote. I very strongly agree with you. :)
            • farts_mckensy 1y ago
              Well, at least someone does.
          • Terr_ 1y ago
            > cretins proclaiming that LLMs aren't truly capable of reasoning > Reasoning is not difficult to define > Reasoning exists on a spectrum > statistical processes [can] scale into reasoning It seems like quite a descent here, starting with the lofty heights of condemning skeptics as "cretins" and insisting the definition is easy... down to what sounds like the introduction to a flavor of panpsychism [0], where even water flowing downhill is a "statistical process" which at enough scale would be "reasoning". I don't think that's a faithful match to what other people mean [1] when they argue LLMs don't "reason." [0] https://en.wikipedia.org/wiki/Panpsychism https://en.wikipedia.org/wiki/Panpsychism [1] https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
  • visarga 1y ago
    > when you drill down to first principles It is a trap to consider that first principles perspective is sufficient. Like, if I tell you before 1980: "iterate over f(z) = z^2 + c" there is no way you are going to guess fractals emerge. Same with the rules for Conway's Game of Life - seeing the code you won't guess it makes gliders and guns. My point is that recursion creates its own inner opacity, it is irreducible, so knowing a recursive system behavior from its iteration code alone is insufficient. It is a becoming not a static thing. That's also why we have the halting problem - recursion. Reasoning is a recursive process too, you can't understand it from analyzing its parts.
    • farts_mckensy 1y ago
      What with this dingus coming at me with recursion? Hey, that's my move, bro. Seriously though, everything you said is totally unnecessary. All I'm saying is, in order to refute the claim that LLMs can't reason, you ought to have your own coherent definition of the word in order to falsify the claim. Recursion is a coherent notion. If you want to include that in the definition, have at it. That actually supports my broader view of reasoning.
  • epolanski 1y ago
    Your tone is a bit odd, but I agree with you on this 100%. Without properly defining what thinking is and what is not, you can't discuss it's properties, let alone discuss which entity manifests it or not.
    • farts_mckensy 1y ago
      My tone is odd? Never heard that before in my life. Definitely wasn't an problem for me growing up.
    • earnestinger 1y ago
      On the other hand: nobody can formally refute that a stone is reasoning/thinking. Lack of definition seems to be ideal substrate for marketing and hype.
  • deleted 1y ago
    [deleted]
• ivape 1y ago
  This is easily explained by accepting that there is no such thing as LRMs. LRMs are just LLMs that iterate on its own answers more (or provides itself more context information of a certain type). The reasoning loop on an "LRM" will be equivalent to asking a regular LLM to "refine" its own response, or "consider" additional context of a certain type. There is no such thing as reasoning basically, as it was always a method to "fix" hallucinations or provide more context automatically, nothing else. These big companies baked in one of the hackiest prompt engineering tricks that your typical enthusiast figured out long ago and managed to brand it and profit off it. The craziest part about this was Deepseek was able to cause a multi billion dollar drop and pump of AI stocks with this one trick. Crazy times.
  • AlienRobot 1y ago
    Is that what "reasoning" means? That sounds pretty ridiculous. I've thought before that AI is as "intelligent" as your smartphone is "smart," but I didn't think "reasoning" would be just another buzzword.
    • ngneer 1y ago
      I am not too familiar with the latest hype, but "reasoning" has a very straightforward definition in my mind. For example, can the program in question derive new facts from old ones in a logically sound manner. Things like applying modus ponens. (A and A => B) => B. Or, all men are mortal and Socrates is a man, and therefore Socrates is mortal. If the program cannot deduce new facts, then it is not reasoning, at least not by my definition.
      • dist-epoch 1y ago
        When people say LLMs can't do X, I like to try it. Q: Complete 3 by generating new knowledge: 1. today is warm 2. cats likes warm temperatures 3. A: Therefore, a cat is likely to be enjoying the weather today. Q: does the operation to create new knowledge you did have a specific name? A: ... Deductive Reasoning Q: does the operation also have a Latin name? A: ... So, to be precise, you used a syllogismus (syllogism) that takes the form of Modus Ponens to make a deductio (deduction). https://aistudio.google.com/app/prompts/1LbEGRnzTyk-2IDdn53tC0HMhCL-yO8qU https://aistudio.google.com/app/prompts/1LbEGRnzTyk-2IDdn53t... People then say "of course it could do that, it just pattern matched a Logic text book. I meant in a real example, not an artificially constructed one like this one. In a complex scenario LLMs obviously can't do Modus Ponens.
• JusticeJuice 1y ago
  Their finding of LLMs working best at simple tasks, LRMs working best at medium complexity tasks, and then neither succeeding at actually complex tasks is good to know.
  • cubefox 1y ago
    Not sure whether I sense sarcasm.
• nialv7 1y ago
  I've seen this too often, papers that ask questions they don't even bother to properly define. > Are these models capable of generalizable reasoning, or are they leveraging different forms of pattern matching? Define reasoning, define generalizable, define pattern matching. For additional credits after you have done so, show humans are capable of what you just defined as generalizable reasoning.
  • deleted 1y ago
    [deleted]
  • NitpickLawyer 1y ago
    > show humans are capable of what you just defined as generalizable reasoning. I would also add "and plot those capabilities on a curve". My intuition is that the SotA models are already past the median human abilities in a lot of areas.
  • crvdgc 1y ago
    In the context of this paper, I think "generalizable reasoning" means that finding a method to solve the puzzle and thus being able to execute the method on puzzle instances of arbitrary complexity.
• beneboy 1y ago
  This kind of explains why Claude will find the right solution, but then the more it thinks and keeps “improving” the more over-engineered (and sometimes wrong) the solution is. Interesting to see this coming up in formal research.
• bicepjai 1y ago
  The study challenges the assumption that more “thinking” or longer reasoning traces necessarily lead to better problem-solving in LRMs
  • bayindirh 1y ago
    As a test, I asked Gemini 2.5 Flash and Gemini 2.5 Pro to decode a single BASE64 string. Flash answered correctly in ~2 seconds, at most. Pro answered very wrongly after thinking and elaborating for ~5 minutes. Flash was also giving a wrong answer for the same string in the past, but it improved. Prompt was the same: "Hey, can you decode $BASE64_string?" I have no further comments.
    • rafterydj 1y ago
      well that's not a very convincing argument. That's just a failure to recognize when the use of a tool- base64 decoder- is needed, not a reasoning problem at all, right?
      • BoorishBears 1y ago
        That's not really a cop out here: both models had access to the same tools. Realistically there are many problems that non-reasoning models do better on, especially when the answer cannot be solved by a thought process: like recalling internal knowledge. You can try to teach the model the concept of a problem where thinking will likely steer it away from the right answer, but at some point it becomes like the halting problem... how does the model reliably think its way into the realization a given problem is too complex to be thought out?
      • bayindirh 1y ago
        I don't know whether Flash uses a tool or not, but it answers pretty quickly. However, Pro opts to use its own reasoning, not a tool. When I look at the reasoning train, it pulls and pulls knowledge endlessly, refining that knowledge and drifting away.
      • Jensson 1y ago
• actinium226 1y ago
  Man, remember when everyone was like 'AGI just around the corner!' Funny how well the Gartner hype cycle captures these sorts of things
  • bayindirh 1y ago
    They're similar to self-driving vehicles. Both are around the corner, but neither can negotiate the turn.
    • hskalin 1y ago
      And commerically viable nuclear fusion
      • mgiampapa 1y ago
        I harvest fusion energy every single day... It's just there in the sky, for free!
    • einrealist 1y ago
      All that to keep the investment pyramid schemes going.
    • kfarr 1y ago
      Waymo's pretty good at unprotected lefts
      • bayindirh 1y ago
        Waymo is pretty good at (a finite number of) unprotected lefts, and this doesn't count as "level 5 autonomous driving".
    • nmca 1y ago
      I saw your comment and counted — in May I took a Waymo thirty times.
      • 1y ago
• alansammarone 1y ago
  I have a somewhat similar point of view to the one voiced by other people, but I like to think about it slightly differently, so I'll chime in - here's my take (although, admittedly, I'm operating with a quite small reasoning budget (5 minutes tops)): Time and again, for centuries - with the pace picking up dramatically in recent decades - we thought we were special and we were wrong. Sun does not rotate around the earth, which is a pretty typical planet, with the same chemical composition of any other planet. All of a sudden we're not the only ones who could calculate, then solve symbolic equations, then play chess, then compose music, then talk, then reason (up to a point, for some definition of "reason"). You get my point. And when we were not only matched, but dramatically surpassed in these tasks (and not a day earlier), we concluded that they weren't _really_ what made us special. At this point, it seems to me reasonable to assume we're _not_ special, and the onus should be on anybody claiming that we are to at least attempt to mention in passing what is the secret sauce that we have (even if we can't quite say what it is without handwaving or using concepts that by definition can not be defined - "qualia is the indescribable feeling of red - its redness (?)). Oh, and sorry, I could never quite grasp what "sentient" is supposed to mean - would we be able to tell we're not sentient if we weren't?
  • ivape 1y ago
    I can give you a pretty wild explanation. Einstein was a freak of nature. Nature just gave him that "something" to figure out the laws of the universe. I'm avoiding the term God as to not tickle anyone incorrectly. Seriously, explain what schooling and environment gets you that guy. So, to varying degrees, all output is from the universe. It's hard for the ego to accept, surely we earned everything we ever produced ... Spooky stuff.
  • keiferski 1y ago
    This analogy doesn’t really work, because the former examples are ones in which humanity discovered that it existed in a larger world. The recent AI example is humanity building, or attempting to build, a tool complex enough to mimic a human being. If anything, you could use recent AI developments as proof of humanity’s uniqueness - what other animal is creating things of such a scale and complexity?
  • 1y ago
• curious_cat_163 1y ago
  > Rather than standard benchmarks (e.g., math problems), we adopt controllable puzzle environments that let us vary complexity systematically Very clever, I must say. Kudos to folks who made this particular choice. > we identify three performance regimes: (1) low complexity tasks where standard models surprisingly outperform LRMs, (2) medium-complexity tasks where additional thinking in LRMs demonstrates advantage, and (3) high-complexity tasks where both models experience complete collapse. This is fascinating! We need more "mapping" of regimes like this! What I would love to see (not sure if someone on here has seen anything to this effect) is how these complexity regimes might map to economic value of the task. For that, the eval needs to go beyond puzzles but the complexity of the tasks still need to be controllable.
  • deleted 1y ago
    [deleted]
  • pegasus 1y ago
    Is (1) that surprising? If I ask someone a simple question but tell them to "think really hard about it", they'll be more likely to treat it as a trick question and look for a non-obvious answer. Overthinking it, basically.
    • curious_cat_163 1y ago
      It is hard to compare models with humans so not sure how to answer it for both. :) But, for models, this is an interesting finding because a lot of LRMs are LLMs with a _bunch_ of post-training done on top. We know this about DeepSeek R1 (one of the models evaluated in the Apple paper) for sure. They write extensively about how they took DeepSeek-V3-Base and made R1 with it. [1] If the post-training is resulting in lower performance on simpler tasks then it ought to inspire more research on how to make it so that it doesn't -- i.e., with more training (of any kind), we should be gaining more capabilities. This has been a problem with DNNs historically, btw. We had these issues when fine-tuning text/image classifiers as well. Some weight changes can be destructive. So, it has to be done with a _lot_ of care. And, I am sure folks are working on it, to be honest. Maybe some of them will say something here. :-) [1] https://github.com/deepseek-ai/DeepSeek-R1 https://github.com/deepseek-ai/DeepSeek-R1
• 8bitsrule 1y ago
  Fusion has been 25 years away for all of my life.
  • sneak 1y ago
    Fusion is net positive energy now; that happened in 2022 (+54%). In 2025 they got a 313% gain (4.13 output factor). Fusion is actually here and working. It’s not cost effective yet but to pretend there has been no progress or achievements is fundamentally false.
    • oneshtein 1y ago
      It will be cost effective in just 25 years.
    • sitkack 1y ago
      Negative Negs spit out low effort snark, they said the same thing about solar, electric cars, even multicore, jit, open source. Thanks for refuting them, the forum software itself should either quarantine the response or auto respond before the comment is submitted. These people don't build the future. Fusion News, May 28th, 2025 https://www.youtube.com/watch?v=1YHcI-SfKx8 https://www.youtube.com/watch?v=1YHcI-SfKx8
    • lrhegeba 1y ago
      It isnt when you look at Q total. Total energy input for all needed support systems versus energy produced. See https://en.wikipedia.org/wiki/Fusion_energy_gain_factor https://en.wikipedia.org/wiki/Fusion_energy_gain_factor for more details
    • einsteinx2 1y ago
      Yes it’s a net positive if you ignore the 100x extra power it took to actually run the reactor, so actually no it’s not net energy positive. Not even close. Nothing to do with cost, we can not build a fusion reactor in 2025 with any amount of money that will produce more energy and goes into it.
• benlivengood 1y ago
  These are the kind of studies that make so much more sense than the "LLMs can't reason because of this ideological argument or this one anecdote" posts/articles. Keep 'em coming! And also; the frontier LLMs blow older LLMs out of the water. There is continual progress and this study would have been structured substantially the same 2 years ago with much smaller N on the graphs because the regimes were much tinier then.
• antics 1y ago
  I think the intuition the authors are trying to capture is that they believe the models are omniscient, but also dim-witted. And the question they are collectively trying to ask is whether this will continue forever. I've never seen this question quantified in a really compelling way, and while interesting, I'm not sure this PDF succeeds, at least not well-enough to silence dissent. I think AI maximalists will continue to think that the models are in fact getting less dim-witted, while the AI skeptics will continue to think these apparent gains are in fact entirely a biproduct of "increasing" "omniscience." The razor will have to be a lot sharper before people start moving between these groups. But, anyway, it's still an important question to ask, because omniscient-yet-dim-witted models terminate at "superhumanly assistive" rather than "Artificial Superintelligence", which in turn economically means "another bite at the SaaS apple" instead of "phase shift in the economy." So I hope the authors will eventually succeed.
  • sitkack 1y ago
    There is no reason that omniscient-yet-dim-witted has to plateau at human intelligence.
    • antics 1y ago
      I am not sure if you mean this to refute something in what I've written but to be clear I am not arguing for or against what the authors think. I'm trying to state why I think there is a disconnect between them and more optimistic groups that work on AI.
      • drodgers 1y ago
        I think that commenter was disagreeing with this line: > because omniscient-yet-dim-witted models terminate at "superhumanly assistive" It might be that with dim wits + enough brute force (knowledge, parallelism, trial-and-error, specialisation, speed) models could still substitute for humans and transform the economy in short order.
        • antics 1y ago
• esafak 1y ago
  I don't know that I would call it an "illusion of thinking", but LLMs do have limitations. Humans do too. No amount of human thinking has solved numerous open problems.
  • th0ma5 1y ago
    The errors that LLMs make and the errors that people make are not probably not comparable enough in a lot of the discussions about LLM limitations at this point?
    • esafak 1y ago
      We have different failure modes. And I'm sure researchers, faced with these results, will be motivated to overcome these limitations. This is all good, keep it coming. I just don't understand the some of the naysaying here.
      • Jensson 1y ago
        They naysayers just says that even when people are motivated to solve a problem the problem might still not get solved. And there are unsolved problems still with LLM, the AI hypemen say AGI is all but a given in a few years time, but if that relies on some undiscovered breakthrough that is very unlikely since such breakthroughs are very rare.
• danck 1y ago
  In figure 1 bottom-right they show how the correct answers are being found later as the complexity goes higher. In the description they even state that in false responses the LRM often focusses on a wrong answer early and then runs out of tokens before being able to self-correct. This seems obvious and indicates that it’s simply a matter of scaling (bigger token budget would lead better abilities for complexer tasks). Am I missing something?
  • moralestapia 1y ago
    Yes, the rest of the paper.
• ByteDrifter 1y ago
  [dead]
• teleforce 1y ago
  > We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across puzzles. It seems that AI LLMs/LRMs need helps from their distant cousins namely logic, optimization and constraint programming that can be attributed as intelligent automation or IA [1],[2],[3],[4]. [1] Logic, Optimization, and Constraint Programming: A Fruitful Collaboration - John Hooker - CMU (2023) [video]: https://www.youtube.com/live/TknN8fCQvRk https://www.youtube.com/live/TknN8fCQvRk [2] "We Really Don't Know How to Compute!" - Gerald Sussman - MIT (2011) [video]: https://youtube.com/watch?v=HB5TrK7A4pI https://youtube.com/watch?v=HB5TrK7A4pI [3] Google OR-Tools: https://developers.google.com/optimization https://developers.google.com/optimization [4] MiniZinc: https://www.minizinc.org/ https://www.minizinc.org/
• thomasahle 1y ago
  All the environments the test (Tower of Hanoi, Checkers Jumping, River Crossing, Block World) could easily be solved perfectly by any of the LLMs if the authors had allowed it to write code. I don't really see how this is different from "LLMs can't multiply 20 digit numbers"--which btw, most humans can't either. I tried it once (using pen and paper) and consistently made errors somewhere.
  • someothherguyy 1y ago
    > humans can't The reasons humans can't and the reasons LLMs can't are completely different though. LLMs are often incapable of performing multiplication. Many humans just wouldn't care to do it.
  • Jensson 1y ago
    > I don't really see how this is different from "LLMs can't multiply 20 digit numbers"--which btw, most humans can't either. I tried it once (using pen and paper) and consistently made errors somewhere. People made missiles and precise engineering like jet aircraft before we had computers, humans can do all of those things reliably just by spending more time thinking about it, inventing better strategies and using more paper. Our brains weren't made to do such computations, but a general intelligence can solve the problem anyway by using what it has in a smart way.
    • thomasahle 1y ago
      Some specialized people could probably do 20x20, but I'd still expect them to make a mistake at 100x100. The level we needed for space crafts was much less than that, and we had many levels of checks to help catch errors afterwards. I'd wager that 95% of humans wouldn't be able to do 10x10 multiplication without errors, even if we paid them $100 to get it right. There's a reason we had to invent lots of machines to help us. It would be an interesting social studies paper to try and recreate some "LLMs can't think" papers with humans.
      • Jensson 1y ago
        > There's a reason we had to invent lots of machines to help us. The reason was efficiency, not that we couldn't do it. If a machine can do it then we don't need expensive humans to do it, so human time can be used more effectively.
• cdrini 1y ago
  When I use a normal LLM, I generally try to think "would I be able to do this without thinking, if I had all the knowledge, but just had to start typing and go?". With thinking LLMs, they can think, but they often can only think in one big batch before starting to "speak" their true answer. I think that needs to be rectified so they can switch between the two. In my previous framework, I would say "would I be able to solve this if had all the knowledge, but could only think then start typing?". I think for larger problems, the answer to this is no. I would need paper/a whiteboard. That's what would let me think, write, output, iterate, draft, iterate. And I think that's where agentic AI seems to be heading.
• d4rkn0d3z 1y ago
  I wrote my first MLP 25 years ago. After repeating some early experiments in machine learning from 20 ywars before that. One of the experiments I repeated was in text to speach. It was amazing to set up training runs and return after seveal hours to listen to my supercomputer babble like a toddler. I literally recall listening and being unable to distinguish the output from my NN from that of a real toddler, I happened to be teaching my neice to read around that same time. And when the NN had gained a large vocabulary such that it could fairly proficiently read aloud, I was convinced that I had found my PHD project and a path to AGI. Further examination and discussion with more experienced researchers gave me pause. They said that one must have a solution, or a significant new approach toward solving the hard problems associated with a research project for it to be viable, otherwise time (and money) is wasted finding new ways to solve the easy problems. This is a more general principle that can be applied to most areas of endeavour. When you set about research and development that involves a mix of easy, medium, and hard problems, you must solve the hard problems first otherwise you blow your budget finding new ways to solve the easy problems, which nobody cares about in science. But "AI" has left the realm of science behind and entered the realm of capitalism where several years of meaningless intellectual gyration without ever solving a hard problem may be quite profitable.
• jackdoe 1y ago
  I think one of the reason we are confused about what LLMs can do is because they use language. And we look at the "reasoning traces" and the tokens there look human, but what is actually happening is very alien to us, as shown by "Biology of Large Language Models"[1] and "Safety Alignment Should Be Made More Than Just a Few Tokens Deep"[2] I am struggling a lot to see what the tech can and can not do, particularly designing systems with them, and how to build systems where the whole is bigger than the sum of its parts. And I think this is because I am constantly confused by their capabilities, despite understanding their machinery and how they work, their use of language just seems like magic. I even wrote https://punkx.org/jackdoe/language.html https://punkx.org/jackdoe/language.html just to remind myself how to think about it. I think this kind of research is amazing and we have to spend tremendous more effort into understanding how to use the tokens and how to build with them. [1]: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio... [2]: https://arxiv.org/pdf/2406.05946 https://arxiv.org/pdf/2406.05946
  • dleeftink 1y ago
    The opposite might apply, too; the whole system may be smaller than its parts, as it excels at individual tasks but mixes things up in combination. Improvements will be made, but I wonder if we should aim for generalists, or accept more specialist approaches as it is difficult to optimise for all tasks at once.
    • jackdoe 1y ago
      You know the meme "seems like will have AGI before we can reliably parse PDFs" :) So if you are building a system, lets say you ask it to parse a pdf, and you put a judge to evaluate the quality of the output, and then you create a meta judge to improve the prompts of the parser and the pdf judge. The question is, is this going to get better as it is running, and even more, is it going to get better as the models are getting better? You can build the same system in completely different way, more like 'program synthesis' imagine you dont use llms to parse, but you use them to write parser code, and tests, and then judge to judge the tests, or even escalate to human to verify, then you train your classifier that picks the parser. Now this system is much more likely to improve itself as it is running, and as the models are getting better. Few months ago Yannic Kilcher gave this example as that it seems that current language models are very constrained mid-sentence, because they most importantly want produce semantically consistent and grammatically correct text, so the entropy mid sentence is very different than the entropy after punctuation. The . dot "frees" the distribution. What does that mean for "generalists" or "specialists" approach when sampling the wrong token can completely derail everything? If you believe that the models will "think" then you should bet on the prompt and meta prompt approach, if you believe they will always be limited then you should build with program synthesis. And, honestly, I am totally confused :) So this kind of research is incredibly useful to clear the mist. Also things like https://www.neuronpedia.org/ https://www.neuronpedia.org/ E.G. Why compliment (you can do this task), guilt (i will be fired if you don't do this task), and threatening (i will harm you if you don't do this task) work with different success rate? Sergey Brin said recently that threatening works best, I cant get my self to do it, so I take his word for it.
• jbentley1 1y ago
  Is Apple failing at AI so they just put all their R&D towards convincing themselves it isn't important?
  • MontyCarloHall 1y ago
    A slightly less cynical take is that they want to temper expectations for the capabilities of LLMs in people’s day-to-day lives, specifically in the context of Apple products. A “smarter Siri” is never going to be an autonomous personal assistant à la Jarvis from Iron Man, which seems to be where a lot of investors think things are going. That tracks with this [0] preprint also released by Apple a few months ago. A slightly more cynical take is that you’re absolutely correct, and making excuses for weak machine learning prowess has long been an Apple tenet. Recall that Apple never made privacy a core selling point until it was clear that Siri was years behind Google’s equivalent, which Apple then retroactively tried to justify by claiming “we keep your data private so we can’t train on it the way Google can.” [0] https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229
  • emp17344 1y ago
    Everyone has an agenda. Companies like OpenAI and Anthropic are incentivized to overstate the capabilities of LLMs, so it’s not like they’re any less biased.
  • wavemode 1y ago
    I get the sense that many of the AI features shoved into consumer products recently have been marketed more towards investors than users. The companies are basically advertising that they're "keeping up" with the competition, meanwhile the features themselves receive mixed-to-poor reviews and are never capable of all the things advertised. So it seems to me that all of Apple, Google, Meta, Microsoft, and Samsung are currently "failing" at AI in exactly the same ways. If Apple is trying to start going a different direction that seems like a good sign.
• gwd 1y ago
  > Through extensive experimentation across diverse puzzles, we show that frontier LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counterintuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having an adequate token budget. This is exactly my experience with coding. Start simple and build up complexity, and everything is great until you get to some threshold, at which point it completely falls apart and seems to stop even trying. Getting effective utilization out of Claude + aider involves managing the complexity that the LLM sees.
• stephc_int13 1y ago
  Human language is far from perfect as a cognitive tool but still serves us well because it is not foundational. We use it both for communication and some reasoning/planning as a high level layer. I strongly believe that human language is too weak (vague, inconsistent, not expressive enough etc.) to replace interactions with the world as a basis to build strong cognition. We're easily fooled by the results of LLM/LRM models because we typically use language fluency and knowledge retrieval as a proxy benchmark for intelligence among our peers.
  • squidproquo 1y ago
    Agree with this. Human language is also not very information-dense; there is a lot of redundancy and uninformative repetition of words. I also wonder about the compounding effects of luck and survivorship bias when using these systems. If you model a series of interactions with these systems probabilistically, as a series of failure/success modes, then you are bound to get a sub-population of users (of LLM/LLRMs) that will undoubtedly have “fantastic” results. This sub-population will then espouse and promote the merits of the system. There is clearly something positive these models do, but how much of the “success” is just luck.
  • anton-c 1y ago
    Sounds like we need ai legalese as that's how we navigate the vagueness of language in the real world. Ofc I imagine they've tried similar things and that it almost takes away the point if u had to prompt that way.
    • stephc_int13 1y ago
      I was not referring to the prompt but to the underlying network that is built on weak cognitive foundations because all of it is coming from language.
  • wslh 1y ago
    Human language is more powerful than its surface syntax or semantics: it carries meaning beyond formal correctness. We often communicate effectively even with grammatically broken sentences, using jokes, metaphors, or emotionally charged expressions. This richness makes language a uniquely human cognitive layer, shaped by context, culture, and shared experience. While it's not foundational in the same way as sensorimotor interaction, it is far more than just a high-level communication tool.
• stephc_int13 1y ago
  The tldr: current approaches to add reasoning on top of language models are mostly tricks to squeeze a bit more juice out of the fruit, but the falloff is pretty steep and quick.
• mitch_said 1y ago
  Not ashamed to admit I found the original paper daunting, so I made a top-down, Q&A-based mind map to help me understand it: https://app.gwriter.io/#/mindmap/view/2d128d6e-c3e8-4b99-8f48-2ab2465a5e87 https://app.gwriter.io/#/mindmap/view/2d128d6e-c3e8-4b99-8f4...
• kamranjon 1y ago
  The two interesting things I learned after reading this paper: Even when given the exact steps needed to arrive at a solution in the prompt, the reasoning models still require just as many steps to reach a workable solution as they would if they weren’t given the solution in the prompt. The other thing, which seems obvious in hindsight, but I don’t typically use these reasoning models in my day to day - is that it requires a significant amount of tokens to reach the point where reasoning models outperform non-reasoning models by a significant margin.
• akomtu 1y ago
  The difference between imitation and reasoning can be made more clear if we switch from language to numbers: 1 3 7 15 31 63 ... How do you continue this sequence? What's the 1000000th number in this sequence? Imitation continues the likeness of what it sees and quickly gets off track. Imitation can't go abstract and tell the 1000000th element without writing down a million numbers leading to the answer. Reasoning finds the rule behind the set of examples and uses this rule to predict the next numbers, so it never gets off track. The rule generating the sequence can be a sophisticated recurrent formula, e.g. a(k) = 2a(k-1) - sqrt(a(k-3)). Imitation can't solve this problem beyond trivial examples, but an AI can do what a scientist would do: come up with hypotheses, verify them against the examples and eventually find a formula that's reasonably accurate. The role of an LLM here is to suggest possible formulas. The same sequence of examples can be generated by many formulas that differ in complexity and accuracy. This provokes the idea of a simple competition between AIs: the one that creates the simplest formula that's 99.5% accurate - wins. The formula really means a small program, once we get beyond trivial recurrent rules. The ability to find simple and accurate models of reality is the essense of intelligence.
• ksec 1y ago
  This actually set back my expectation of AI / LLM / LRM by at least 5 if not 10 years. But someone please correct me if I am wrong. My idea was that up to a few years ago while AI / LLM is good at getting conversational or dishing out results that is in a language we understand. It still doesn't "understand" anything and in a lot of time conjured up that seems remotely correctly. Pattern matching over a very large data set that could be correct for 70% and increasingly to 80%+ of the time. However more accurate predictions would require order of magnitude more computing resources. But pattern matching is still, pattern matching. There is no reasoning behind it. 1+1 will never equals to 11 but it may have skewed towards that results because of Javascript. When fundamental logic isn't behind any of these progress and process. The very bottom layer of any conversation / information / results are fragile. So I have been skeptical of AI progress or LLM. That was until LRM or as the title said Reasoning LLMs. I thought we somehow manage to programme critical thinking into it, or some sort of reflection / fact checking / rationale / basic logic as fundamental principle. And while I can tell LRM isn't and wont be perfect, and possibly never quite reach AGI, the layer will improve over time until we find different ways to progress. And we will have something I called Assisted Intelligence. Which is what a lot of people uses as AI programming today. Instead what this shows is that LRM isn't reasoning at all. It is LLM conjured up excuses to make it look like it is reasoning. It is another set of pattern matching specially made up for reasoning to look like it is reasoning. It is basically a kid making things up on why he got the results without thinking because he just want to get away from class or homework that looks very clever. May be the title gave it away, and made be we got tricked. It was always a LLM specifically trained for showcasing "reasoning". The actual reasoning behind the scene is never done. Hence the title "The Illusion of Thinking".
• piskov 1y ago
  All "reasoning" models hit a complexity wall where they completely collapse to 0% accuracy. No matter how much computing power you give them, they can't solve harder problems. This research suggests we're not as close to AGI as the hype suggests. Current "reasoning" breakthroughs may be hitting fundamental walls that can't be solved by just adding more data or compute. Apple's researchers used controllable puzzle environments specifically because: • They avoid data contamination • They require pure logical reasoning • They can scale complexity precisely • They reveal where models actually break Models could handle 100+ moves in Tower of Hanoi puzzles but failed after just 4 moves in River Crossing puzzles. This suggests they memorized Tower of Hanoi solutions during training but can't actually reason. https://x.com/RubenHssd/status/1931389580105925115 https://x.com/RubenHssd/status/1931389580105925115
  • jmogly 1y ago
    I think this might be part of the reason Apple is “behind” on generative AI … LLMs have not really proven to be useful outside of relatively niche areas such as coding assistants, legal boiler plate and research, and maybe some data science/analysis which I’m less familiar with Other “end user” facing use cases have so far been comically bad or possibly harmful, and they just don’t meet the quality bar for inclusion in Apple products, which as much as some people like to doo doo on them and say they have gotten worse, still have a very high expectations of quality and UX from customers.
    • lostmsu 1y ago
      You are being too charitable to Apple. They are just being the fox from the Sour Grapes fable.
    • roxolotl 1y ago
      Apple prides itself on building high quality user experiences. One can argue over whether that’s true anymore or ever was but it’s very clear they pride themselves on that. This is why Apple tends to “be late” on many features. I think like you’re saying it’s becoming even more clear they aren’t seeing a UX they are willing to ship to customers. We’ll see with WWDC if they found something in the last year to make it better but this paper seems to indicate they haven’t.
• bilsbie 1y ago
  Interestingly I just hit an example of this. Highly specific but I was asking about pickleball strategy and grok and Claude both couldn’t seem to understand you can’t aim at the opponent’s feet when you’re hitting up. Just kept regurgitating internet advice and I couldn’t get it to understand the reasoning on why it was wrong.
  • jqpabc123 1y ago
    Hey --- if the internet says it, it can't be wrong.
    • bilsbie 1y ago
      In this case it found generic advice and was confusing itself.
      • jqpabc123 1y ago
        That's one explanation. Another could be that it simply has no real *understanding* of anything. It simply did a statistical comparison of the question to the available advice and picked the best match --- kinda what a search engine might do. Expecting *understanding* from a synthetic, statistical process will often end in disappointment.
        • naijaboiler 1y ago
          Yup. It’s time for us to just accept that LLMs are “similar in meaning” machines not “thinking/ understanding” machines
          • jqpabc123 1y ago
            If you think about it --- an LLM that could really *grasp* "pickleball" from a text description without ever seeing, playing or "experiencing" the game is not just human level intelligence --- it's superhuman. And the same applies to a lot of real world situations.
• bilsbie 1y ago
  I wonders if there’s past symbolic reasoning research we could integrate into LLMs. They’re really good at parsing text and understanding the relationships between objects ie getting the “symbols” correct. Maybe we plug into something like prolog (or other such strategies?)
  • sroussey 1y ago
    That’s called nuero-symbolic AI and there are people working on it.
• mcswell 1y ago
  Discussion of this article by Gary Marcus: https://garymarcus.substack.com/p/a-knockout-blow-for-llms https://garymarcus.substack.com/p/a-knockout-blow-for-llms
• yalogin 1y ago
  Isn't it a matter of training? This is the way I think about LLMs. They just "learned" so much context that they spit out tokens one after the other based on that context. So if it hallucinates it's because it understood the context wrong or doesn't grasp the nuance in the context. The more complicated the task the higher chance of hallucinations. Now I don't know if this can be improved with more training but that is the only tool we have.
• bigEnotation 1y ago
  Reasoning models are just wrappers over the base model. It was pretty obvious it wasn’t actually reasoning but rather just refining the results using some kind of reasoning like heuristic. At least that’s what I assumed when they were released and you couldn’t modify the system prompt.
  • hliyan 1y ago
    I don't understand why this comes as a surprise to a lot of people. Underlying all this is sequences of text tokens converted to semantic vectors, with some positional encoding, and then run through matrix multiplications to compute the probability of the next token. The probability is a function of all the text corpuses the model has previously consumed. You can run this process multiple times, chaining one output into the next one's input, but reason as humans know is unlikely to emerge.
  • rahimnathwani 1y ago
    Not just wrappers. Some models are fine-tuned with reasoning traces.
• MaxPock 1y ago
  Is it a case of being left by the train and now trying to rain on the parade ? Attention span has shifted from the next iPhone release to the newest LLM
• avsteele 1y ago
  People are drawing erroneous conclusions from this. My read of this is that the paper demonstrates that given a particular model (and the problems examined with it) that giving more thought tokens does not help on problems above a certain complexity. It does not say anything about the capabilities of future, larger, models to handle more complex tasks. (NB: humans trend similarly) My concern is that people are extrapolating from this to conclusions about LLM's generally, and this is not warranted The only part about this i find even surprising is he abstract's conclusion (1): that 'thinking' can lead to worse outcomes for certain simple problem. (again though, maybe you can say humans are the same here. You can overthink things)
  • lukev 1y ago
    You can absolutely extrapolate the results, because what this shows is that even when "reasoning" these models are still fundamentally repeating in-sample patterns, and that they collapse when faced with novel reasoning tasks above a small complexity threshold. That is not a model-specific claim, it's a claim on the nature of LLMs. For your argument to be true would need to mean that there is a qualitative difference, in which some models possess "true reasoning" capability and some don't, and this test only happened to look at the latter.
    • avsteele 1y ago
      The authors don't say anything like this that I can see. Their conclusion specifically identifies these as weaknesses of current frontier models. Furthermore we have clearly seen increases in reasoning from previous frontier models to current frontier models. If the authors could /did show that both previous-generation and current-generation frontier models hit a wall at similar complexity that would be something, AFAIK they do not.
    • lowbloodsugar 1y ago
      Human brains work the same way. Some of us are just better at analogy. I’ve worked with plenty of people who were unable to transfer knowledge of one area to another, identical area with different terms.
    • 1y ago
• rikafurude21 1y ago
  Apple, with all the compute and talent anyone could ask for, did not manage to create a frontier LLM model and was forced to bite the bullet and include GPT in their coveted walled garden in order to keep up and not be left behind in our current AI-Hype-(bubble) mania. I suspect this hurts their ego greatly and thats why they feel compelled to "talk trash". Saltiness wont get them anywhere though. I hope they manage to pull through. It would be a shame if they also went where Microsoft currently is.
• Yenrabbit 1y ago
  A few observations: 1) the performance vs complexity curve looks very similar to that for most humans (having seen groups attempt Towers of Hanoi with 5 car tires) haha 2) models can trivially solve some of these tasks when given tools 3) this is an internship paper with some quirks that many mostly dismissed, but is being quoted everywhere as "Apple proves LLMs can't ever reason" Anyway, fun experiment to test your understanding of these things but don't take any conclusions as gospel :)
• a11r 1y ago
  The system prompt in this experiment limits the solution to always spell out the concrete moves verbally. A human solving the Tower of Hanoi gives up around N=4 and goes off to invent a recursive solution instead. Prompted differently, the LLM would solve these puzzles just fine. Here is my complete review/analysis of the paper: https://www.linkedin.com/pulse/art-abstraction-human-advantage-age-artificial-abhishek-kumar-qhwwc https://www.linkedin.com/pulse/art-abstraction-human-advanta... edit: fixed typo
• bgnn 1y ago
  Who is thinking that LLMs are thinking to begin with. It would have been amazing if they could. All the reasoning etc are just taking a stab at it but far from thinking.
• jawiggins 1y ago
  Figure 5 is really quite remarkable. It seems to show that normal LLMs are better at tasks where the correct answer is likely to be the next token. For tasks that require a small number of intermediate steps, current reasoning models do much better, but break down as the number of intermediate steps grow. This seems to indicate that the next generation of models should focus on recursively solving small parts of the problem before function-calling another model to solve another small part of the problem and working it's answer into the reasoning loop. Many seem to be citing this paper as an indication that LLMs are over - I think this indicates a clear path towards the next step function change in their abilities.
• nrjpoddar 1y ago
  One of the best analysis I found is this blog by Vishal Misra https://medium.com/@vishalmisra/the-illusion-of-thinking-why-language-models-cant-improve-themselves-0c71a13811e2 https://medium.com/@vishalmisra/the-illusion-of-thinking-why...
• jackson12t 1y ago
  This feels a bit like a weird way to test 'thinking' in models, and reminds me of the old story of Gauss[1] and his classmates being assigned the task of adding up the numbers from 1-100. I think the way the paper lays out the performance regimes is pretty interesting, but I don't think they achieved their goal of demonstrating that LRMs can't use reasoning to solve complex puzzles organically (without contamination/memorization): IMO testing the model's ability to define an algorithm to solve the puzzle would have been a better evaluation of that (rather than having the model walk through all of the steps manually). I don't know that I'd use an LRM for this sort of long-tail reasoning where it has to follow one single process for a long time over just one prompt; if I needed a really long chain of reasoning I'd use an agent or workflow. It sounds more like the tests measure a model's ability to reason coherently and consistently over many steps rather than a model's ability to understand and solve a complex puzzle. For example, for the Tower of Hanoi, a prompt like "Define an algorithm that will find the sequence of moves to transform the initial configuration into the goal configuration" (e.g. "find an arithmetic series formula, young Gauss") seems like it would have been a better approach than "Find the sequence of moves to transform the initial configuration into the goal configuration" (e.g. "add up all these numbers"). This is kind of seen in how the study included a step where the LRMs were given the algorithm and then asked to solve the problem, the focus was on an LRM's ability to follow the steps, not their ability to come up with an algorithm/solution on their own. In a job interview, for example, who among us would accept inability to hold all of the `(2^n) - 1` steps of the Tower of Hanoi in our brain as evidence of poor reasoning ability? Again, I think it's a really interesting study covering a model's ability to consistently follow a simple process over time in pursuit of a static objective (and perhaps a useful benchmark moving forward), but I'm not confident that it successfully demonstrates a meaninful deficiency in overall reasoning capability. [1]: https://www.americanscientist.org/article/gausss-day-of-reckoning https://www.americanscientist.org/article/gausss-day-of-reck...
• giardini 1y ago
  Now that y'all have had time to digest this, could you please post an ELI5 for the great unwashed masses?
• whiplash451 1y ago
  The worse piece of news is that the performance of thinking and non-thinking models is inverted depending on whether the problem is simple or medium, which implies that you need to know the problem's complexity to choose the best model. Ouch.