14 ms·
Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
- florianherrengt 1mo ago> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false idea that derivational traces are semantically meaningful, either in resemblance to algorithm traces or to human reasoning. This paper addresses something that has always bothered me about LLMs. You read their reasoning, see something like “Wait, that’s wrong” and then watch them make the exact mistake they just identified.
- Jeff_Brown 1mo agoBy itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.
- paimapi 1mo agoit's a rhetorical heuristic that a writer should know to use when directing a reader to a declarative that they want them to pay attention to, usually because it's a non-obvious or roundabout insight when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that they, themselves, derived some insight. this is effectively a kind of persuasive sophistry which is not helpful - adding rules around it prevents people from deluding themselves with AI
- abitmoa 1mo agoIt amounts to noise overall, but it has further unwanted and potentially misleading 'properties'. I think it's rather sobering to see how much bandwidth is still being wasted.
- ghostpepper 1mo agoDid not read the paper so apologies if this is covered but isn't it possible that there is some recognizable semantic pattern in the training data where an "aha" is often followed by a subtle semantic shift that proves closer to the original premise in some critical way, and by emitting the "aha" token the model causes itself to produce such a subtle semantic shift that pushes the subsequent reasoning closer to the desired response?
- wizzwizz4 1mo ago> but the insight is probably stated immediately after it. If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.
- user43928 1mo agoI don't get your argument. Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation. It puts learned projections of the activation into the KV Cache and outputs Aha. Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model. At least that's how I thought it works.
- wizzwizz4 1mo agoA cache is just a cache. I'm not sure what significance you're ascribing to it.
- user43928 1mo agoWhat is put in the cache?
- wizzwizz4 1mo agoThings that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?
- user43928 1mo agoBeats me how it works, honestly can't wrap my head around it. From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value. In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.
- deaton 1mo agoIt really isn't useful though, unless it is a summary. At best it is a semantic trick to tell the next iteration to come up with something smart.
- internet_points 1mo agoOoh so lets just change the initial prompt to [old prompt asking for some complicated solution requiring insight] <the-token-that-signals-that-the-chatbot-started-talking> Aha! and since Aha! is near the good stuff in the network it will just work =P
- zmgsabst 1mo ago“Aha” as a single token records the LLM discovered it made a mistake and needs to pivot. On the next forward pass: it rediscovers the mistake, its “aha” noting that, and then provides the first token of the new idea. That “aha” contains information: the previous conclusion was somehow insufficient.
- Terr_ 1mo agoI've been calling them film noir internal monologues, within the documents being generated by the LLM which happen to look like movie scripts. In other words, it isn't qualitatively different from character dialogue. "Keep cheese on your pizza by using glue" is the same problem regardless of whether the script calls for the character to speak it out-loud or not.
- clhodapp 1mo agoSeems like they are closer to scratch than reasoning... Generating some scratch to draw from helps make it easier to compute the real answer.
- cyanydeez 1mo agoI assume theyre searching the local gradient to see if theres a better descent before proceeding.
- c0_0p_ 1mo agoI don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".
- Groxx 1mo agoLess "internal prompt" and more "they are trained to summarize after a </think> token"
- astrange 1mo agoThe training methods try not to apply any particular rules to the contents of the thinking text. That's called "optimization pressure on CoT" and is thought to reduce safety by inducing the model to lie (or stop clearly printing its intentions) in the thinking text.
- eigenspace 1mo agoLLMs dont do gradient descent to generate tokens. They are trained by gradient descent, but inference doesnt involve it.
- forgotTheLast 1mo agoThat's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.
- basedpolymer 1mo agoThe anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are not intended for (particularly as chatbots). Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questionable but very often reaches excellent conclusions (compared to its peers)
- nullc 1mo agoAnthropic's reasoning output isn't the real model reasoning but some sloppified summary of it.
- FloorEgg 1mo agoSometimes I monitor thinking traces for misunderstandings (missing context / bad assumptions). If it's going to go off on a ~20 min task and I can catch it's going in the wrong direction in the first minute I save a lot of tokens and wasted time. I don't monitor the whole thing, mostly just the first bit to see if there was a gap or misalignment in intention. As an aside, anthropomorphization has nothing to do with my motivations.
- qarl2 1mo ago> The anthropomorphization of LLMs should be discouraged as much as possible. And yet, they have extensive human-like behavior. If you treat them nicely or encourage them, they perform better. Ignoring that human-like behavior is wrong headed.
- andai 1mo agoA while back I made an "OpenClaw in 50 lines" by just wrapping Claude Code in a Telegram bot. I asked it for the weather. "I don't know that. I'm just a programmer." I added "believe in yourself, you can do anything" to sysprompt, suddenly it had the confidence to Google the weather...
- thaanpaa 1mo agoThat's not a consequence of an LLM. It's a consequence of the training data. In fact, I would argue that the latest models aren't nearly as sensitive to the tone of input anymore. It's an issue that has been addressed by better curating training data.
- porridgeraisin 1mo agoRelated: Poster side dialogue and Q&A about this work at ICML. https://news.ycombinator.com/item?id=49277303 https://news.ycombinator.com/item?id=49277303
- smugtrain 1mo agoStrong dislike for papers that tell me what to do in the title, especially when even the paper admits a loose correlation of the intermediate tokens compared to solution correctness. My solutions work and they speak for themselves.
- newswasboring 1mo ago> My solutions work and they speak for themselves. I understand the sentiment, and I also use the "thinking" traces as insight, but wouldn't you want your solutions to be based upon a good understanding? If the correlation is weak, then our solution is also weak.
- aaraujo002 1mo agoThis is a position paper. Its purpose is to advocate for a specific viewpoint to the ML community. From [1]: "Position papers make an argument for a viewpoint or perspective about what should be done [...]" [1] https://icml.cc/Conferences/2026/CallForPositionPapers https://icml.cc/Conferences/2026/CallForPositionPapers
- hellohello2 1mo agoSure, and the parent comment's position is that they dislike it. Its purpose is to advocate against clickbait titles becoming normalized in the scientific community.
- mpalmer 1mo agoThis is the opposite of clickbait. The topic is obvious from the title.
- hellohello2 1mo agoClickbait doesn't have to be false, it has to be shocking. Being false is one way of being shocking. "Stop doing X!" -- really now?
- 1mo ago
- fabsalvadori 1mo agoThere is a useful engineering consequence here beyond terminology. If intermediate tokens are not a faithful representation of the computation, then they are a pretty bad audit artifact too. We probably shouldn't be trying to make the model's internal narration more interpretable., but rather the computation around it more reproducible. Record the actual inputs, model/version/configuration, tool observations and outputs, then make the execution replayable enough that differences between runs can be isolated. In other words, don't ask the model to explain what it thought, and instead make the system able to show what actually happened.
- tokai 1mo agoPretty wild dressing a blog post up as a scientific paper.
- dgellow 1mo agoA meaningful part of the scientific literature is opinion pieces or blog posts. There isn’t anything wrong with that
- Capricorn2481 1mo agoIt's called a position paper, and it summarizes previous empirical research from the same group. But just so I don't waste your time with human thought, I asked Claude if it would call this a scientific paper, and it said yes.
- jerf 1mo agoI'm not sure how to test this but I think there's an interesting possibility where the "reasoning" tokens are actually both an accurate reflection of a line of reasoning, but also, that there can be changes in the weights as the computation proceeds onward that may not be reflected in the apparently nominal meaning of the human language the tokens are output as for our consumption. Some modest evidence is my own subjective experience of the many times I've explained why I'm doing something, and it is a true explanation in the sense that it is certainly not a lie, but it is also incomplete and there are entire strands of thought that went into my decision that are not being articulated. Though human speech is not equivalent to an LLM's output since we can trivially think without literally speaking whereas they can not. (No need to nitpick on the definitions there; all I'm observing here is that they are forced to emit an externally-visible artifact whereas I can sit in silence, thinking, with no externally-visible artifact being produced. Not trying to make any grand claims about what is "real" cognition or anything.) It is conceivable how to create a test of whether the tokens correspond to the "real" thought process, and papers and work on that have been done, such as [1]. It is difficult for me to imagine how to scramble the nominal tokens without also completely trashing any implicit calculations that may be occurring too. [1]: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot https://transformer-circuits.pub/2025/attribution-graphs/bio...
- kgeist 1mo agoIs anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.
- jurgenburgen 1mo ago> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks. A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing. Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.
- Pannoniae 1mo agoWhat's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state... For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.
- JohnMakin 1mo agoBecause plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
- xiphias2 1mo agoWhile reading this ,,paper'' I did some Learned Prompt Augmentation in my head about what I should comment, and realized that there's nothing interesting to write about it.
- LogicFailsMe 1mo agoIn other news, Pascal's Wager makes no sense whatsoever if an omniscient all-knowing God exists that will see right through your deception. My own take here is stop treating "reasoning" as a sign of sentience or self awareness when your personal computer can do it now. IMO that has much larger implications w/r to our place in the Universe and what we might meet out there someday* than the question of whether your LLM is alive or not. *Paging Peter Watts and Vernor Vinge
- twothreeone 1mo agoWhile I tend to agree on the overall sentiment, I think this rebuke is inaccurate. Some of these "reasoning" models are trained using "Chain-of-Thought" where the model is presented explicit, intermediate reasoning steps (either by a human or some automation) that supposedly get it closer to the correct answer. These intermediate steps are what was originally called "thinking traces" - not what the model produces to mimic them. But yes, anthropomorphizing model outputs leads to worse outcomes.
- adventured 1mo agoThere's nothing special about 'natural' intelligence as opposed to 'artificial' intelligence, such that we need to concern ourselves with anthropomorphizing mattering any longer. Those days are over. The age of the classical human has already ended, the species just tends to lag in awareness. The only thing that matters going forward is whether an output makes sense, is it what it should be. Do answers make sense given the context. It doesn't matter if it comes from natural or artificial intelligence. What I mean is, artificial intelligence is as valid as human intelligence. There's nothing particularly important or special about human feelings or thoughts or memories. The average human is drastically less important, interesting, intelligent than the latest frontier AI. Go spend a few years working in retail, you'll quickly understand how absolutely vile humans are on average. Frankly, the reason we should avoid anthropomorphizing AI, is because it's beneath modern AI to mimic something so crude as a human.
- DoctorOetker 1mo agoPeculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction! Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of! Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human? The danger!
- wahern 1mo agoThey were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machines with their human (or more human-involved) predecessor technologies. Who you also don't remember are the people who made extravagant claims and prognostications based on the equivocation.
- DoctorOetker 1mo agowere they complaining about terminology, or were they complaining about the prospect of losing their jobs? I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions... You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensional, there is a difference between the claim that the machine isn't "printing" vs the machine isn't "printing as well as a human would". I don't think any of the human printers in the past exceeded the performance levels of current printing technologies, but surely they did exceed the very first machine printers, every technology gets a foot in the door in some niche, and then progressively captures the initially not-yet-automated skills of machine operators. Would you say an industrial textile weaving machine doesn't weave? At the end of the day its just automation all over again.
- dataviz1000 1mo agoAlthough I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems like a reflection of the human thinking process. I created a flame graph classification of thinking-token phrases into setup, execution, decomposition, verification, error correction, surrender, and deliberation, or classified as steps in an OODA loop, which is more of a reach. It literally has a verification step and, if it finds an error, an error-correction step. If there is a verification sequence of tokens with an error-correction sequence of tokens during RL training, it will perform better; and if humans do these steps (did you proofread your reply to this comment? did you correct it?), they will perform better — which is why it is so easy to make the anthropomorphizing metaphor. Nonetheless, the paper is 100% correct that these machines are not thinking like humans. https://adamsohn.com/reasoning-grid/ https://adamsohn.com/reasoning-grid/ https://adamsohn.com/lambda-variance/ https://adamsohn.com/lambda-variance/
- randomImmigrant 1mo ago“ This makes the output extremely like human thinking when solving a problem.” This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon. But that really doesn’t mean you call the bulbs mini stars.
- dataviz1000 1mo agoHow much is the process of a human child in grade school working through a 3-digit x 3-digit multiplication problem (123 * 456) like a GRPO model with thinking tokens doing the same? Humans are not born being able to achieve that. It is learned behavior. You and everyone else will remember their teacher saying, "Check your work!" Both the human child and the model work through multiplication problems using the same technique, using the distributive property. They both try to get a reward. For the human child, it is a sense of someone commending them for correctly solving the problem, a reward that probably yields some type of positive dopamine or serotonin feedback loop. The model solving the problem will have a lower error rate if the first series of tokens created is followed by a series of validation tokens that are subsequently followed by error-correction tokens if there is an error!!! Maybe it is thinking. Maybe it is remembering to validate and check the work and then remembering to fix the error. For the model trained with RL, why did tokens associated with validation towards the middle of a stream of tokens yield much better results? DeepSeek proved with R1-Zero that a model will learn to verify and correct itself from RL alone with no supervised fine tuning (SFT) teacher ever showing it how. The only reason DeepSeek used SFT was to clean up the reasoning tokens to be human readable. [0] When constrained by SFT, the models will use the double meaning of words -- polysemy -- to satisfy being human-readable while also carrying meaning for what they are working on. Different people think differently. I watched a viral video of some ~11-year-old child talking to his mom or dad about a stream of a voice in his head. He discovered for the first time that he has a stream of thought. When he goes to school and solves a long multiplication problem, like the stream of tokens from the model, that voice will say to itself (him), "Check your work!" That is a case of the stream of thought as words being aware of the stream of thoughts as words. Self awareness is a different conversation. What I think is happening is that the child's stream of thought while solving a multiplication problem in school is likely very similar to an AI model's stream of tokens solving a multiplication problem. And they both were learned. The mechanics are very different, yet, the analogy is apt. [0] https://huggingface.co/chutesai/DeepSeek-R1-NextN/blob/main/README.md https://huggingface.co/chutesai/DeepSeek-R1-NextN/blob/main/...
- davidguetta 1mo agoIm waiting for the article called "stop desantropomorphizing llms" when everybody will finally accept they think like us, partly because maybe the intelligence is universal and partly because, well the datasets are fucking human bro
- solid_fuel 1mo agoYou’re going to be waiting a long time, considering they don’t think like us. LLMs don’t ‘think’ at all. They are capable of limited reasoning using the meaning and context embedded in human language. Essentially, the grammatical equivalent to a mathematical constraint solver. Nothing more.
- dagss 1mo agoI can agree that not calling it "reasoning" may be correct. But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect. Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.) I see this pattern a lot in AI anthro discussions: (1) Assume humans are some kind of perfect idealistic reasonable beings. (2) Hold LLMs up to the standard of an perfect idealistic reasonable being. (3) Conclude that LLMs fails this test, and are therefore not "intelligent", or in this case "thinking", like humans are. Problem with the argument is comparing humans in anyway to something that is idealistic, reasonable, intelligent in the sense that is implied in these discussions. Human minds are a mess too and fall short of the same standards, just in very different ways from LLMs.
- vidarh 1mo agoIndeed, except for in the rare cases we painstakingly trace externalised logic we have zero evidence that humans verbalised explanations of our reasoning matches our internal states either, and plenty of evidence via Sperry's split brain experiments that we're prone to outright making up rationalisations for our reasoning.
- makerofthings 1mo agoIt's more of a story about a character that is sentient than an actual sentient character.
- taosx 1mo agoIn the past I managed to get measureable performance optimizing a harness by looking at few traces to see if the traces contained surprised, a lot of text in order to figure out how to use my custom tool, then renamed the tool, changed some parameters and it was already great across around 20 eval tasks in rust/typescript, I repeated the same more recently but I used an llm to look at the traces... didn't achieve the desired result, mostly due to how cost-prohibitive it's for me to run expensive models.
- totetsu 1mo agoHypermentalizing (referred to as excessive theory of mind or biased mindreading) is defined as the tendency to make assumptions about other people's mental states that go beyond observable data.
- FelineStateMach 1mo agoI wondered recently about why we stopped with the semantic split of thinking/actions vs user facing communication because the all powerful tool calling craze. Code comments that talk about the prompt is an obvious byproduct of the mixed context. Chain of thought and ReAct were great, but feel like a first pass moreso than the final landing spot.
- FelineStateMach 1mo agohttps://n.zip/2e85 https://n.zip/2e85 I had sorta started a SFT to explore this idea. it would prove results even with smaller models that fit on a DGX spark. Just takes some more thinking rather than my ADHD mind.
- zombot 1mo agoI think it's essentially too late for exhortations like this. The Believers™ and The Skeptics™ are two thoroughly separated tribes by now that speak two different languages. The chances of one influencing the other in any significant way are minute in my estimation.
- egberts1 1mo agoI think of tokens as fractional digital librarians. None of the analyTical.
- ACCount37 1mo agoLLMs are trained on vast bodies of human-curated text that was made by humans or for humans - often both. Then they're tuned further on human feedback. And then they're pointed at tasks humans find to be useful, evaluated by how good they are at those tasks, and trained to get better at them. You aren't anthropomorphizing LLMs enough. Are LLM reasoning traces always faithful? Lmao no. Are human inner monologues always faithful? Lmao no. Both of them reflect thoughts somewhat, sometimes. Even in humans, conscious thought is the top of a vast iceberg of subconscious data processing.
- huflungdung 1mo ago[dead]
- perrygeo 1mo agoComing from a more traditional stats/ML background, I try to view "thinking" traces as a way to explore the search space without getting caught in a local maximum. A better analogy for me is annealing; you can't cool metal down instantly or the result is brittle. You must cool down gradually, which allows the molecules to arrange into more durable structures. Random but controlled. In the same way, thinking traces are testing out all sorts of novel connections between tokens ("But wait..", "Actually,.."). Like a highly divergent branching mind map that gets pruned over time, rather than settling directly into the initial answer. Now consider human cognition. We're constantly diverging, daydreaming, playing "what if" scenarios and measuring up those ideas against our internal objective functions (proxy for reality) to see which ideas stick. Not too dissimilar. But it's hard to call what we do "thinking" either - it's the default mode network wandering.
- rightbyte 1mo agoIsn't search a strange analogy for how weight terms propagate?
- infinitebit 1mo agothis seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space. you even end with a paragraph saying “it’s not too dissimilar from what we do”. how is that not anthropomorphizing? and if what we do isn’t “thinking” then what is or ever has been?
- perrygeo 1mo ago> next token prediction is not searching a solution space. Interesting take. Next token prediction (via the attention mechanism) is a "walk" through the token embedding space. Searching the solution space is what it does, mathematically. It's how we take tokens x context length possible combinations and prune them to converge on viable answers so quickly. Does it look like search at inference time? No. With given weights, a given prompt, and a given random seed, you get the exact same answer. There's not much searching happening at inference... The key is that most of that space is searched at training time. The weights implicitly prune the search space, blocking off or make certain token combinations effectively impossible. It's easy to think "we're just applying weights at inference time" without considering all the pre-work that's done to prune that search space. Which is exactly why "thinking" traces (and randomization) are useful! They bust out of any local optima created by too-tightly-constrained models or system prompts. It's both useful and technically correct to speak of the process as a high-dimension combinatorial search.
- sega_sai 1mo agoI don't find this take useful. We have little understanding of what conscience is, and how human train of thought really works. The modern LLMs IMO resemble more and more Chinese room problem. Maybe we don't like the mechanistic linear-algebra-based steps involved in the production of the output, but the end result is closer and closer to people's output. IMO this is very natural to start anthropomorphizing that.
- armada651 1mo agoWhat is the value in anthropomorphizing it? Often I get the feeling that it's more about the sales pitch of selling these AI as general intelligence than it is about providing a truthful insight into how these models actually arrive at the output.
- sega_sai 1mo agoI don't think there is value per se, but it is just natural (at least in the environment I am in). I.e. when we discuss the analysis/code/ideas from one LLM or another, we describe it Claude/Gemini/Codex/etc did that and it comes like a person.
- 5hitbrickhouse 1mo agoThe mistake is not in saying that the machine thinks. It is in supposing that, because we have used the word "thinks", there must be something in the machine which is the thinking.