6 ms·
"The Illusion of Thinking" – Thoughts on This Important Paper
- deleted 1y ago[deleted]
- LazySpoon 1y ago[dead]
- withinboredom 1y agoThe number of people I’ve run into that think ChatGPT (or whatever model they are using) is still thinking about whatever it is they’ve talked about before while they’re not using it is non-zero. It doesn’t help that the models sometimes say things like “give me five minutes to think on it” and stuff like that.
- tough 1y agoyeah man, the antrophomorphization is bad. Lately chatgpt is like -let me show you an example I used before- like its a real professional. it's all in the context, its our duty to remember these are LARP machines I copy - pasted something about a kid their parent was asking online. Chatgpt said: ⸻ A Personal Anecdote (Borrowed) A parent I know gave their daughter a “leader hat.” [..] How in the hell would a large language model have past personal anecdotes and know other parents, idk
- elif 1y agoIt is not the models anecdote but that of the parent. And knowing is something even dumb programs do with data. It doesn't imply human cognition
- tough 1y agoI mean yeah the "borrowed" is doing a lot of heavy lifting. But that was just like the latest, I just don't love how it talks like it was an alive person at the other end, maybe I should add something to my personal prompt to avoid it.
- TeMPOraL 1y ago> yeah man, the antrophomorphization is bad. Unfortunately, it's also the least wrong approach. People who refuse to entertain thinking about LLMs as quasi-humans, are the ones perpetually confused about what LLMs can or cannot do and why. They're always up in arms about prompt injections and hallucinations, and keep arguing those are bugs that need to be fixed, unable to recognize them as fundamental to the model's generality and handling natural language. They keep harping about "stochastic parrots" and Naur's "program theory", claiming LLMs are unable to reason or model in the abstract, despite many published research that lobotomizes the models live and nail concepts as they form and activate. If you squint and imagine LLM to be a person, suddenly all of these things become apparent. So I can't really blame people - especially non-experts - for sticking to an approach that's actually yielding much better intuition than the alternatives.
- tsimionescu 1y agoAnthropomorphization also leads to very wrong conclusions though. In particular, we have a theory of mind that we apply in relation to other humans, mostly on the basis of "they haven't lied to me so far, so it's unlikely they will suddenly start lying to me now". But this is dead wrong in relation to output from an LLM - just because it generated a hundred correct answers doesn't tell you anything about how likely the 101st one is to be be correct and not a fabrication. Trust is always misplaced if put into results returned by an LLM, fundamentally.
- TeMPOraL 1y ago> But this is dead wrong in relation to output from an LLM - just because it generated a hundred correct answers doesn't tell you anything about how likely the 101st one is to be be correct and not a fabrication. Neither it is for humans, the whole "they haven't lied to me so far, so it's unlikely they will suddenly start lying to me now" thing is strongly conditioned on "low frequency" assumptions from shared biology, history and culture, as well as "high frequency" assumptions about the person, e.g. that they're actually friendly, or the context didn't change, that your goals are still aligned[0]... or that they aren't drunk, confused, or a kid. Correct your theory of mind for the nature of LLMs, and your conclusions will be less wrong. LLMs are literally trained to approximate humans, so it makes sense this approach gives a decent high-level approximation. (What LLMs don't have is the "sticky" part of social sphere; humans, generally, behave as if they expect to interact again in the future. Individual or public opinion doesn't weigh on them much. And then there's the other bit, in that we successfully eliminate variance in humans - the kind of people who cannot be theory-of-minded well by others, get killed or locked up in prisons and mental institutions, or on the less severe end, get barred from high-risk jobs and activities.) To be clear: I'm not claiming you should go deep into the theory-of-mind thing, assuming intentions and motivations and internal emotional states in LLMs. But that's not necessary either. I say that the right amount of anthropomorphism just cognitive and focused on immediate term. "What would a human do in an equivalent situation?" gets you 90% there. -- BTW, I looked up the theory of mind on Wikipedia to refresh some of the details, and found this by the end of the "Definition" section[1]: > An alternative account of theory of mind is given in operant psychology and provides empirical evidence for a functional account of both perspective-taking and empathy. The most developed operant approach is founded on research on derived relational responding[jargon] and is subsumed within relational frame theory. Derived relational responding relies on the ability to identify derived relations, or relationships between stimuli that are not directly learned or reinforced; for example, if "snake" is related to "danger" and "danger" is related to "fear", people may know to fear snakes even without learning an explicit connection between snakes and fear.[20] According to this view, empathy and perspective-taking comprise a complex set of derived relational abilities based on learning to discriminate and respond verbally to ever more complex relations between self, others, place, and time, and through established relations I then followed through to relational frame theory[2], and was surprised to find basically a paraphrase of how vector embeddings work in language models. Curious. -- [0] - Yes, I used that word. Also yes, in weak or transactional relationships, this may change without you realizing it, and the other person may suddenly start lying to you with no advance warning. I'm pretty sure everyone experienced this at some point. [1] - https://en.wikipedia.org/wiki/Theory_of_mind#Definition https://en.wikipedia.org/wiki/Theory_of_mind#Definition [2] - https://en.wikipedia.org/wiki/Relational_frame_theory https://en.wikipedia.org/wiki/Relational_frame_theory
- southernplaces7 1y agoRight here on HN, where you'd think that most people would be able to think better, there's no shortage of AI fanboys who seriously try to frame the idea that LLMs may be conscious, and that we can't really be sure if we're conscious, since, you know, science doesn't know what causes consciousness. I guess me feeling conscious, and thus directing much of my life and activity around that notion is just a big trick, and i'm no different from ChatGPT, only inferior and with lower electrical needs.
- IshKebab 1y agoThis is a bad faith representation of what people are actually saying which is: 1. As far as we know there is nothing non-physical in the brain (a soul or magical quantum microtubules or whatever). Everything that happens in your head obeys the laws of physics. 2. We definitely are conscious. 3. Therefore consciousness can arise from purely physical processes. 4. Any physical process can be computed. And biological thought appears as far as we can see to simply be an extraordinarily complex computation. 5. Therefore it is possible for a sufficiently fast & appropriately programmed computer to be conscious. 6. LLMs bear many resemblances to human thinking (even though they're obviously different in many ways too), and we've established that computers can be conscious, so you can't trivially rule out LLMs being conscious. At least for now, I don't think anyone sane says today's LLMs are actually conscious. (Though consciousness is clearly a continuum... so maybe they're as conscious as a worm or whatever.) The main point is that people trivially dismissing AI consciousness or saying "it can't think" because it's only matrix multiplications are definitely wrong. It might not be conscious because consciousness requires more complexity, or maybe it requires some algorithmic differences (e.g. on-line learning). But it definitely isn't not-conscious just because it is maths or running on a computer or deterministic or ...
- Veen 1y agoThe "as far as we know" in premise one is hand-waving the critical point. We don't know how or if conscious arises from purely physical systems. It may be physical, but inadequately replicated by computer systems (it may be non-computable). It may not be physical in the way we use that concept. We just don't know. And the fact that we don't know puts the conclusion in doubt. Even if consciousness arises from a physical system, it doesn't follow that computation on a physical substrate can produce consciousness, even if it mimics some of the qualities we associate with consciousness.
- rednafi 1y agoA large part of these anthropomorphic narratives was pushed by SV nerds to grab shareholder attention. LLMs are transformative, but a lot of the tools around them already treat them as opaque function calls. Instead of piping text to sed, awk, or xargs, we’re just piping streams to these functions. This analogy can stretch to cover audio and video usage too. But that’s boring and doesn’t explain why you suddenly have to pay more for Google Work Suite just to get bombarded by AI slop. This isn’t to undermine the absolutely incredible achievements of the people building this tech. It’s to point out the absurdity of the sales pitch from investors and benefactors. But we’ve always been like this. Each new technology promises the world, and many even manage to deliver. Or is it that they succeed only because they overpromise and draw massive attention and investment in the first place? IDK, we’ll see.
- tough 1y agoyou gotta aim for the stars to maybe with luck reach the moon if you only aim for the moon, you’ll never break orbit.
- rednafi 1y agoIf you don’t aim for the moon, you’re gonna surely miss it. The number of orbital calculations you need to get right to land on the moon is bonkers /s.
- tough 1y agolmao true never played kerbal space tbh
- TeMPOraL 1y agoIf either of you did, you'd know it's false. Getting to the Moon (and back) by eyeballing the trajectory is a rite of passage for every KSP player. You launch for standard equatorial low orbit, then coast until the Moon is about 90 degrees to the right / behind you, and burn ahead until you run out of fuel in the second stage. This turns your orbit into an elongated ellipse, and as you get close to its far end, the Moon will catch up and capture you in its gravity well. When playing one of the non-sandbox aka. "progression" modes, i.e. "career" or "science", the trajectory planning tools are locked behind an expensive upgrade, effectively forcing you to do your first Moon fly-by this way.
- Animats 1y ago"We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across puzzles." That's the real conclusion of the Apple paper. It's correct. LLMs are terrible at arithmetic, or even counting. We knew that. So, now what? It would be interesting to ask the "AI system" to write a program to solve such puzzle problems. Most of the puzzles given have an algorithmic solution. This may be a strategy problem. LLMs may need to internalize Polya's How to Solve It?[2] Read the linked Wikipedia article. Most of those are steps an LLM can do, but a strategy controller is needed to apply them in a useful order and back off when stuck. The "Illusion of thinking" article is far less useful than the the Apple paper. (Did anybody proofread the Apple paper? [1] There's a misplaced partial sentence in the middle of page 2. Or a botched TeX macro.) [1] https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... [2] https://en.wikipedia.org/wiki/How_to_Solve_It https://en.wikipedia.org/wiki/How_to_Solve_It
- withinboredom 1y agoEventually (many of us have hit it, and if you haven’t, you will), your program will hit a certain size/complexity where the AI also falls flat. These tests basically try to measure that limit, not just solve the puzzle.
- scotty79 1y agoSo far they've shown that it fails early if the answer would exceed context size anyways. At least that was the case for the Hanoi problem from what I heard. They somehow failed to notice that though.
- tsurba 1y agoIs it a puzzle if there is no algorithm? But testing via coding algos to known puzzles is problematic as the code may be in the training set. Hence you need new puzzles, which is kinda what ARC was meant to do, right? Too bad OpenAI lost credibility for that set by having access to it, but ”verbally promising” (lol) not to train on it, etc.
- 1y ago
- diimdeep 1y agoHere is another opinion on the original paper - Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity https://arxiv.org/abs/2506.09250 https://arxiv.org/abs/2506.09250
- tkgally 1y agoThe points made in the abstract of that paper look like pretty serious challenges to the Apple paper. What might be the counterobjections?
- jmsdnns 1y agoWorth looking at how many papers the authors have published before assuming they challenge Apple.
- frozenseven 1y agoIf they make a solid case against the Apple paper, why does it matter? It's not like they're the first ones to criticize (read: easily debunk) Apple on such grounds.
- jmsdnns 1y agoApple's paper said a bunch of folks with weird predictions are wrong and so obviously lots of bad information will emerge in response. Who says what matters more now than it used to.
- imiric 1y agoPersonal experience? I don't care about benchmarks, nor LLMs' capability to solve puzzles. This is the first thing AI companies optimize their training for, which is misleading and practically false advertising. I care about how good LLMs are for helping me with specific tasks. Do they generate code that on the surface appears correct, yet on closer inspection has security and performance issues, is unnecessarily complex, often doesn't even compile, which takes me more time to troubleshoot and fix than if I were to write it myself? Do they explain concepts confidently while being wrong, which I have no way of knowing unless I'm a domain expert? Do they repeat all these issues even after careful re-prompting and with all the contextual information they would need? Does all this waste my time more than it helps? The answer is yes to all of the above. So while we can argue whether LLMs can think and reason, my personal experience tells me that they absolutely cannot, and that any similarity to what humans can do is nothing but an illusion.
- ofrzeta 1y agoThe conversational interface is a bit of a dark pattern, too, because it exaggerates what an LLM can do and creates an illusion that makes using LLMs a bit more addictive than a technical interface that says "generating random data that may be useful".
- roywiggins 1y agoThe completion interface was way less usable but more honest
- visarga 1y ago"The Illusion of Thinking" is bad, but so is the opposite "it's just math and code". They might not be reasoning like humans, but they are not reducible to just code and math either. They do something new, something that just math and code did not do before.
- jmsdnns 1y ago> just code and math either this is literally what they are
- digbybk 1y agoI think it’s the “just” that they are taking issue with. We are “just” neurons. But we demonstrate interesting emergent behaviors that, in principle, can be reduced to firing neurons but in practice we don’t understand and shouldn’t diminish with the word “just”.
- jmsdnns 1y agofair point!
- visarga 1y agoTo insist it's "just code and math" is to fundamentally mistake the rules for the game. Knowing the rules of Go without playing grants no insight into strategy, just as showing someone the code for Conway's Game of Life without execution reveals nothing about the gliders that emerge. The profound dynamics of a system are not found in its static instruction set. To ignore this is a fundamental error, because these are recursive processes that iteratively rewrite their own context, a reality the "it's just X" argument completely fails to grasp. Of course Turing undecidability is the canonical example - can't predict if a program will halt without execution.
- cantor_S_drug 1y agoHere's something to think about: Can we consider AI conscious similar to how Hardy recognised Ramanujan's genius? That if AI weren't to be conscious then they won't have the imagination to write what it wrote.
- seydor 1y ago> The Costs of Anthropomorphizing AI These were not costs , but massive benefits for hyped AI startups. They attracted the attention of a wide audience of investors including clueless ones, they brought in politicians and hence the media attention, and created a FOMO race for them to plant flags before each other.
- imiric 1y agoThis a balanced take on the subject. As usual, opinions on extreme ends should not be taken seriously. Modern ML tools are neither intelligent nor apocalyptic. What they are is very good at specific tasks, and the sooner we focus on those, the less time and resources will be wasted on tasks they're not good at. Benchmarks and puzzles don't matter. AI companies will only use them to optimize their training, so that they can promote their results and boost their valuation on hype alone. They're more useful for marketing than as a measurement of real-world capabilities. Judging by the increase of these negative discussions, we seem to be near or at the Peak of Inflated Expectations w.r.t. AI, and we'll be better off once we surpass it.
- globnomulous 1y agoI haven't watched the interview recently, and I don't have time to re-watch it and cross reference, so take the following with a grain of salt: Parts of this blog post so closely match the language and structure of this interview[0] that I'm inclined to accuse the author of direct plagiarism. [0] https://hardcoresoftware.learningbyshipping.com/p/233-the-illusion-of-thinking-thoughts https://hardcoresoftware.learningbyshipping.com/p/233-the-il...