17 ms·

Tracing the thoughts of a large language model

1072 points by Philpax 2y ago

rambambram 2y ago
When I want to trace the 'thoughts' of my programs, I just read the code and comments I wrote. Stop LLM anthropomorphizing, please. #SLAP
JPLeRouzic 2y ago
This is extremely interesting: The authors look at features (like making poetry, or calculating) of LLM production, make hypotheses about internal strategies to achieve the result, and experiment with these hypotheses. I wonder if there is somewhere an explanation linking the logical operations made on a on dataset, are resulting in those behaviors?
- JKCalhoun 2y ago
  And they show the differences when the language models are made larger.
EncomLab 2y ago
This is very interesting - but like all of these discussions it sidesteps the issues of abstractions, compilation, and execution. It's fine to say things like "aren't programmed directly by humans", but the abstracted code is not the program that is running - the compiled code is - and that is code is executing within the tightly bounded constraints of the ISA it is being executed in. Really this is all so much slight of hand - as an esolang fanatic this all feels very familiar. Most people can't look a program written in Whitespace and figure it out either, but once compiled it is just like every other program as far as the processor is concerned. LLM's are no different.
- ctoth 2y ago
  And DNA? You are running on an instruction set of four symbols at the end of the day but that's the wrong level of abstraction to talk about your humanity, isn't it?
  - EncomLab 2y ago
    DNA is the instruction set for protein composition- not thinking.
fpgaminer 2y ago
> This is powerful evidence that even though models are trained to output one word at a time I find this oversimplification of LLMs to be frequently poisonous to discussions surrounding them. No user facing LLM today is trained on next token prediction.
- rco8786 2y ago
  Super interesting. Can you explain more, or provide some reading? I’m obviously behind
- JKCalhoun 2y ago
  As a layman though, I often see this description for how it is LLMs work.
  - fpgaminer 2y ago
    Right, but it leads to too many false conclusions by lay people. User facing LLMs are only trained on next token prediction during initial stages of their training. They have to go through Reinforcement Learning before they become useful to users, and RL training occurs on complete responses, not just token-by-token. That leads to conclusions elucidated by the very article, that LLMs couldn't possibly plan ahead because they are only trained to predict next tokens. When the opposite conclusion would be more common if it was better understood that they go through RL.
    - mentalgear 2y ago
      What? The "article" is from anthropic, so I think they would know what they write about. Also, RL is an additional training process that does not negate that GPT / transformers are left-right autoencoders that are effectively next token predictors. [Why Can't AI Make Its Own Discoveries? — With Yann LeCun] (https://www.youtube.com/watch?v=qvNCVYkHKfg https://www.youtube.com/watch?v=qvNCVYkHKfg)
      - pipes 2y ago
zerop 2y ago
The explanation of "hallucination" is quite simplified, I am sure there is more there. If there is one problem I have to pick to to trace in LLMs, I would pick hallucination. More tracing of "how much" or "why" model hallucinated can lead to correct this problem. Given the explanation in this post about hallucination, I think degree of hallucination can be given as part of response to the user? I am facing this in RAG use case quite - How do I know model is giving right answer or Hallucinating from my RAG sources?
- kittikitti 2y ago
  I incredibly regret the term "hallucination" when the confusion matrix exists. There's much more nuance when discussing false positives or false negatives. It also opens discussions on how neural networks are trained, with this concept being crucial in loss functions like categorical cross entropy. In addition, the confusion matrix is how professionals like doctors assess their own performance which "hallucination" would be silly to use. I would go as far to say that it's misleading, or a false positive, to call them hallucinations. If your AI recalls the RAG incorrectly, it's a false positives. If your AI doesn't find the data from the RAG or believes it doesn't exist it's a false negative. Using a term like "hallucination" has no scientific merit.
  - esafak 2y ago
    So you never report or pay heed to the overall accuracy?
- pcrh 2y ago
  The use of the term "hallucination" for LLMs is very deceptive, as it implies that there is a "mind". In ordinary terms, "hallucinations" by a machine would simply be described as the machine being useless, or not fit for purpose. For example, if a simple calculator (or even a person) returned the value "5" for 2+2= , you wouldn't describe it as "hallucinating" the answer....
  - astrange 2y ago
    "Hallucination" happened because we got AI images before AI text, but "confabulation" is a better term.
LoganDark 2y ago
LLMs don't think, and LLMs don't have strategies. Maybe it could be argued that LLMs have "derived meaning", but all LLMs do is predict the next token. Even RL just tweaks the next-token prediction process, but the math that drives an LLM makes it impossible for there to be anything that could reasonably be called thought.
- yawnxyz 2y ago
  rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. Those types of systems are inherently interpretable because you can just look at how they work. They're not black boxes. LLMs are black boxes, and if anything, interpretability systems show us what the heck is going on inside them. Especially useful when half the world is using these already, and we have no idea how t hey work
  - kazinator 2y ago
    Water doesn't think, yet if you inject it into the entrance of a maze, it will soon come gushing out of the exit.
  - LoganDark 2y ago
    > rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. That doesn't mean the water itself has strategies, just that you can use water in an implementation of strategy... it's fairly well known at this point that LLMs can be used as part of strategies (see e.g. "agents"), they just don't intrinsically have any.
- ajkdhcb2 2y ago
  True. People use completely unjustified anthropomorphised terminology for marketing reasons and it bothers me a lot. I think it actually holds back understanding how it works. "Hallucinate" is the worst - it's an error and undesired result, not a person having a psychotic episode
- kazinator 2y ago
kittikitti 2y ago
What's the point of this when Claude isn't open sourced and we just have to take Anthropic's word for it?
- ctoth 2y ago
  > What's the point of this - That similar interpretability tools might be useful to the open source community? - That this is a fruitful area to research?
  - kittikitti 2y ago
    Can you use those same tools on Claude? Is the difference trivial from open source models?
    - ctoth 2y ago
      https://news.ycombinator.com/item?id=42208383 https://news.ycombinator.com/item?id=42208383 > Show HN: Llama 3.2 Interpretability with Sparse Autoencoders > 579 points by PaulPauls 4 months ago | hide | past | favorite | 100 comments > I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. > I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback.
- probably_wrong 2y ago
  I blame the scientific community for blindly accepting OpenAI's claims about GPT-3 despite them refusing to release their model. The tech community hyping every press release didn't help either. I hope one day the community starts demanding verifiable results before accepting them, but I fear that ship may have already sailed.
Hansenq 2y ago
I wonder how much of these conclusions are Claude-specific (given that Anthropic only used Claude as a test subject) or if they extrapolate to other transformer-based models as well. Would be great to see the research tested on Llama and the Deepseek models, if possible!
marcelsalathe 2y ago
I’ve only skimmed the paper - a long and dense read - but it’s already clear it’ll become a classic. What’s fascinating is that engineering is transforming into a science, trying to understand precisely how its own creations work This shift is more profound than many realize. Engineering traditionally applied our understanding of the physical world, mathematics, and logic to build predictable things. But now, especially in fields like AI, we’ve built systems so complex we no longer fully understand them. We must now use scientific methods - originally designed to understand nature - to comprehend our own engineered creations. Mindblowing.
- ctoth 2y ago
  This "practice-first, theory-later" pattern has been the norm rather than the exception. The steam engine predated thermodynamics. People bred plants and animals for thousands of years before Darwin or Mendel. The few "top-down" examples where theory preceded application (like nuclear energy or certain modern pharmaceuticals) are relatively recent historical anomalies.
  - marcelsalathe 2y ago
    I see your point, but something still seems different. Yes we bred plants and animals, but we did not create them. Yes we did build steam engines before understanding thermodynamics but we still understood what they did (heat, pressure, movement, etc.) Fun fact: we have no clue how most drugs works. Or, more precisely, we know a few aspects, but are only scratching the surface. We're even still discovering news things about Aspirin, one of the oldest drugs: https://www.nature.com/articles/s41586-025-08626-7 https://www.nature.com/articles/s41586-025-08626-7
    - tmp10423288442 2y ago
      > Yes we did build steam engines before understanding thermodynamics but we still understood what it did (heat, pressure, movement, etc.) We only understood in the broadest sense. It took a long process of iteration before we could create steam engines that were efficient enough to start an Industrial Revolution. At the beginning they were so inefficient that they could only pump water from the same coal mine they got their fuel from, and subject to frequent boiler explosions besides.
aithrowawaycomm 2y ago
I struggled reading the papers - Anthropic’s white papers reminds me of Stephen Wolfram, where it’s a huge pile of suggestive empirical evidence, but the claims are extremely vague - no definitions, just vibes - the empirical evidence seems selectively curated, and there’s not much effort spent building a coherent general theory. Worse is the impression that they are begging the question. The rhyming example was especially unconvincing since they didn’t rule out the possibility that Claude activated “rabbit” simply because it wrote a line that said “carrot”; later Anthropic claimed Claude was able to “plan” when the concept “rabbit” was replaced by “green,” but the poem fails to rhyme because Claude arbitrarily threw in the word “green”! What exactly was the plan? It looks like Claude just hastily autocompleted. And Anthropic made zero effort to reproduce this experiment, so how do we know it’s a general phenomenon? I don’t think either of these papers would be published in a reputable journal. If these papers are honest, they are incomplete: they need more experiments and more rigorous methodology. Poking at a few ANN layers and making sweeping claims about the output is not honest science. But I don’t think Anthropic is being especially honest: these are pseudoacademic infomercials.
- TimorousBestie 2y ago
  Agreed. They’ve discovered something, that’s for sure, but calling it “the language of thought” without concrete evidence is definitely begging the question.
- famouswaffles 2y ago
  >The rhyming example was especially unconvincing since they didn’t rule out the possibility that Claude activated “rabbit” simply because it wrote a line that said “carrot” I'm honestly confused at what you're getting at here. It doesn't matter why Claude chose rabbit to plan around and in fact likely did do so because of carrot, the point is that it thought about it beforehand. The rabbit concept is present as the model is about to write the first word of the second line even though the word rabbit won't come into play till the end of the line. >later Anthropic claimed Claude was able to “plan” when the concept “rabbit” was replaced by “green,” but the poem fails to rhyme because Claude arbitrarily threw in the word “green”! It's not supposed to rhyme. That's the point. They forced Claude to plan around a line ender that doesn't rhyme and it did. Claude didn't choose the word green, anthropic replaced the concept it was thinking ahead about with green and saw that the line changed accordingly.
smath 2y ago
Reminds me of the term 'system identification' from old school control systems theory, which meant poking around a system and measuring how it behaves, - like sending an input impulse and measuring its response, does it have memory, etc. https://en.wikipedia.org/wiki/System_identification https://en.wikipedia.org/wiki/System_identification
- Loic 2y ago
  It is not old school, this is my daily job and we need even more of it with the NN models used in MPC.
  - nomel 2y ago
    I've looked into using NN for some of my specific work, but making sure output is bounded ends up being such a big issue that the very code/checks required to make sure it's within acceptable specs, in a deterministic way, ends up being an acceptable solution, making the NN unnecessary. How do you handle that sort of thing? Maybe main process then leave some relatively small residual to the NN? Is your poking more like "fuzzing", where you just perturb all the input parameters in a relatively "complete" way to try to find if anything goes wild? I'm very interested in the details behind "critical" type use cases of NN, which I've never been able to stomach in my work.
    - lqr 2y ago
      This paper may be interesting to you. It touches on several of the topics you mentioned: https://www.science.org/doi/10.1126/scirobotics.abm6597 https://www.science.org/doi/10.1126/scirobotics.abm6597
    - Loic 2y ago
      For us, the NN is used in a grey box model for MPC in chemical engineering. The factories we control have relatively long characteristic time, together with all the engineering bounds, we can use the NN to model parts of the equipment from raw DCS data. The NN modeled parts are usually not the most critical (we are 1st principles based for them) but this allows us to quickly fit/deploy a new MPC in production. Faster time to market/production is for us the main reason/advantage of the approach.
jacooper 2y ago
So it turns out, it's not just simple next token generation, there is intelligence and self developed solution methods (Algorithms) in play, particularly in the math example. Also the multi language finding negates, at least partially, the idea that LLMs, at least large ones, don't have an understanding of the world beyond the prompt. This changed my outlook regarding LLMs, ngl.
- miraculixx 2y ago
  There is no evidence to this end. There is lots of claims, but claims are not evidence.
kazinator 2y ago
> Claude writes text one word at a time. Is it only focusing on predicting the next word or does it ever plan ahead? When a LLM outputs a word, it commits to that word, without knowing what the next word is going to be. Commits meaning once it settles on that token, it will not backtrack. That is kind of weird. Why would you do that, and how would you be sure? People can sort of do that too. Sometimes? Say you're asked to describe a 2D scene in which a blue triangle partially occludes a red circle. Without thinking about the relationship of the objects at all, you know that your first word is going to be "The" so you can output that token into your answer. And then that the sentence will need a subject which is going to be "blue", "triangle". You can commit to the tokens "The blue triangle" just from knowing that you are talking about a 2D scene with a blue triangle in it, without considering how it relates to anything else, like the red circle. You can perhaps commit to the next token "is", if you have a way to express any possible relationship using the word "to be", such as "the blue circle is partially covering the red circle". I don't think this analogy necessarily fits what LLMs are doing.
- kazinator 2y ago
  By the way, there was recently a HN submission about a project studying using diffusion models rather than LLM for token prediction. With diffusion, tokens aren't predicted strictly left to right any more; there can be gaps that are backfilled. But: it's still essentially the same, I think. Once that type of model settles on a given token at a given position, it commits to that. Just more possible permutations of the token filling sequence have ben permitted.
- pants2 2y ago
  > it commits to that word, without knowing what the next word is going to be Sounds like you may not have read the article, because it's exploring exactly that relationship and how LLMs will often have a 'target word' in mind that it's working toward. Further, that's partially the point of thinking models, allowing LLMs space to output tokens that it doesn't have to commit to in the final answer.
  - kazinator 2y ago
polygot 2y ago
There needs to be some more research on what path the model takes to reach its goal, perhaps there is a lot of overlap between this and the article. The most efficient way isn't always the best way. For example, I asked Claude-3.7 to make my tests pass in my C# codebase. It did, however, it wrote code to detect if a test runner was running, then return true. The tests now passed, so, it achieved the goal, and the code diff was very small (10-20 lines.) The actual solution was to modify about 200-300 lines of code to add a feature (the tests were running a feature that did not yet exist.)
- felbane 2y ago
  Ah yes, the "We have a problem over there/I'll just delete 'over there'" approach.
  - polygot 2y ago
    I've also had this issue, where failing tests are deleted to make all the tests pass, or, it mocks a failing HTTP request and hardcodes it to 200 OK.
    - ctoth 2y ago
      Reward hacking, as predicted over and over again. You hate to see it. Let him with ears &c.
- brulard 2y ago
  That is called "Volkswagen" testing. Some years ago that automaker had mechanism in cars which detected when the vehicle was being examined and changed something so it would pass the emission tests. There are repositories on github that make fun of it.
  - rsynnott 2y ago
    While that’s the most famous example, this sort of cheating is much older than that. In the good old days before 3d acceleration, graphics card vendors competed mostly on 2d acceleration. This mostly involved routines to accelerate drawing Windows windows and things, and benchmarks tended to do things like move windows round really fast. It was somewhat common for card drivers to detect that a benchmark was running, and just fake the whole thing; what was being drawn on the screen was wrong, but since the benchmarks tended to be a blurry mess anyway the user would have a hard time realising this.
osigurdson 2y ago
>> Claude can speak dozens of languages. What language, if any, is it using "in its head"? I would have thought that there would be some hints in standard embeddings. I.e., the same concept, represented in different languages translates to vectors that are close to each other. It seems reasonable that an LLM would create its own embedding models implicitly.
- generalizations 2y ago
  Who's to say Claude isn't inherently a shape rotator, anyway?
- iNic 2y ago
  There are: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-multilingual https://transformer-circuits.pub/2025/attribution-graphs/bio...
greesil 2y ago
What is a "thought"?
TechDebtDevin 2y ago
>>Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. This always seemed obvious to me or that LLMs were completing the next most likely sentence or multiple words.
- deleted 2y ago
  [deleted]
indigoabstract 2y ago
While reading the article I enjoyed pretending that a powerful LLM just crash landed on our planet and researchers at Anthropic are now investigating this fascinating piece of alien technology and writing about their discoveries. It's a black box, nobody knows how its inhuman brain works, but with each step, we're finding out more and more. It seems like quite a paradox to build something but to not know how it actually works and yet it works. This doesn't seem to happen very often in classical programming, does it?
- BoredPositron 2y ago
  The bigger problem is that nobody knows how a human brain works that’s the real crux with the analogy.
  - richardatlarge 2y ago
    I would say that nobody agrees, not that nobody knows. And it’s reductionist to think that the brain works one way. Different cultures produce different brains, possible because of the utter plasticity of the learning nodes. Chess has a few rules, maybe the brain has just a few as well. How else can the same brain of 50k years ago still function today? I think we do understand the learning part of the brain, but we don’t like the image it casts, so we reject it
    - wat10000 2y ago
      That gets down to what it means to “know” something. Nobody agrees because there isn’t enough information available. Some people might have the right idea by luck, but do you really know something if you don’t have a solid basis for your belief but it happens to be correct?
      - richardatlarge 2y ago
        Potentially true, but I don’t think so. I believe it is understood and unless you’re familiar with every neuro/behavioral literature, you can’t know. Science paradigms are driven by many factors and being powerfully correct does not necessarily rank high when the paradigms implications are unpopular
d--b 2y ago
> This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. Suggesting that an awful lot of calculations are unnecessary in LLMs!
- bonoboTP 2y ago
  Yeah, it always seemed pretty wasteful to me. In every single forward pass the LLM must basically start out from scratch, without all the forward-looking plans it made the previous times, and must figure out what we are doing, where we are in the generation process, as in the movie Memento, waking up after an episode of amnesia, except you're waking up in the middle of typing out a sentence, you can look at the previous typed words, but can't carry your future plans with you ahead to the next word. At the next word, you (your clone) again wakes up and must figure out from scratch what it is that we are supposed to be typing out. The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. That would basically turn the thing into a recurrent network though. And those are more difficult to train and have a host of issues. Maybe there will be a better way.
  - colah3 2y ago
    > The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretability research at Anthropic. That's a great intuition, and in fact the transformer architecture actually does exactly what you suggest! Activations from earlier time steps are sent forward to later time steps via attention. (This is another thing that's lost in the "models just predict the next word" framing.) This actually has interesting practical implications -- for example, in some sense, it's the deep reason costs can sometimes be reduced via "prompt caching".
    - bonoboTP 2y ago
      I'm more a vision person, and haven't looked a lot into NLP transformers, but is this because the attention is masked to only allow each query to look at keys/values from its own past? So when we are at token #5, then token #3's query cannot attend to token #4's info? And hence the previously computed attention values and activations remain the same and can be cached, because it would anyway be the same in the new forward pass?
annoyingnoob 2y ago
Do LLMs "think"? I have trouble with the title, claiming that LLMs have thoughts.
- danielbln 2y ago
  What's "thinking"?
  - annoyingnoob 2y ago
    Why the need to anthropomorphize AI? Why does AI think vs process or interpret or apply previous calculated statistical weights or anything other than think? I would argue that binary systems built on silicon are fundamentally different that human biology and deserve to be described differently, not forced into the box of human biology.
deadbabe 2y ago
We really need to work on popularizing better, non-anthropomorphic terms for LLMs, as they don’t really have “thoughts” the way people think. Such terms make people more susceptible to magical thinking.
- danielbln 2y ago
  Could you argue why they don't? And could you also argue why we do?
  - rongrobert 2y ago
    When a car moves over the ground, we do not call that running, we call that driving as to not confuse the mechanism of the output. Both running and driving are moving over the ground but with entirely different mechanisms. I imagine saying the LLM has thoughts is like pretending the car has wheels for legs and is running over the ground. It is not completely wrong but misleading and imprecise.
    - danielbln 2y ago
      Planes fly, birds fly. They use related, but ultimately quite different mechanisms to do so. Yet we call both flying.
      - miraculixx 2y ago
        To fly means "to soar through air; move through the air with wings" (etymonline) That is pretty much an accurate discription of what planes and birds do. To plan means "to reason with intent". That is very much not what LLMs do, and the paper does not provide evidence to the contrary. Yet it uses the term to give credence to it's rather speculative interpretation of observed correlation as causation. Interestingly enough there is no definition of the term, which at least would help to understand what the authors actually mean. I would be more inclined to take a more positive stance to the paper if it used more appropriate terms, such as call observed correlations just that. Granted that would possibly make for much less of a fancy title.
davidmurphy 2y ago
On a somewhat related note, check out the video of Tuesday's Computer History Museum x IEEE Spectrum event, "The Great Chatbot Debate: Do LLMs Really Understand?" Speakers: Sébastien Bubeck (OpenAI) and Emily M. Bender (University of Washington). Moderator: Eliza Strickland (IEEE Spectrum). Video: https://youtu.be/YtIQVaSS5Pg https://youtu.be/YtIQVaSS5Pg Info: https://computerhistory.org/events/great-chatbot-debate/ https://computerhistory.org/events/great-chatbot-debate/
0x70run 2y ago
I would pay to watch James Mickens comment on this stuff.
a3w 2y ago
Article and papers looks good. Video seems misleading, since I can use optimization pressure and local minima to explain the model behaviour. No "thinking" required, which the video claims is proven.
mvATM99 2y ago
What a great article, i always like how much Anthropic focuses on explainability, something vastly ignored by most. The multi-step reasoning section is especially good food for thought.
deleted 2y ago
[deleted]
moralestapia 2y ago
[flagged]
SkyBelow 2y ago
>Claude speaks dozens of languages fluently—from English and French to Chinese and Tagalog. How does this multilingual ability work? Is there a separate "French Claude" and "Chinese Claude" running in parallel, responding to requests in their own language? Or is there some cross-lingual core inside? I have an interesting test case for this. Take a popular enough Japanese game that has been released for long enough for social media discussions to be in the training data, but not so popular to have an English release yet. Then ask it a plot question, something major enough to be discussed, but enough of a spoiler that it won't show up in marketing material. Does asking in Japanese have it return information that is lacking when asked in English, or can it answer the question in English based on the information in learned in Japanese? I tried this recently with a JRPG that was popular enough to have a fan translation but not popular enough to have a simultaneous English release. English did not know the plot point, but I didn't have the Japanese skill to confirm if the Japanese version knew the plot point, or if discussion was too limited for the AI to be aware of it. It did know of the JRPG and did know of the marketing material around it, so it wasn't simply a case of my target being too niche.
modeless 2y ago
> In the poetry case study, we had set out to show that the model didn't plan ahead, and found instead that it did. I'm surprised their hypothesis was that it doesn't plan. I don't see how it could produce good rhymes without planning.
- ripped_britches 2y ago
  It would be really hard to get such good results on coding challenges without planning. This is indeed an odd hypothesis.
alach11 2y ago
Fascinating papers. Could deliberately suppressing memorization during pretraining help force models to develop stronger first-principles reasoning?
HocusLocus 2y ago
[Tracing the thoughts of a large language model] "What have I gotten myself into??"
0xbadcafebee 2y ago
AI "thinks" like a piece of rope in a dryer "thinks" in order to come to an advanced knot: a whole lot of random jumbling that eventually leads to a complex outcome.
- cheeze 2y ago
  I regularly see this but I feel like it's disingenuous. Akin to saying "if we simulate enough monkies on a typewriter, we'll eventually get the right result"
  - zvitiate 2y ago
    If we could motivate the monkies sufficiently with bananas, we'd probably improve those odds substantially.
- FeepingCreature 2y ago
  Ah yes, I too often have extended English conversation with my washed rope.
cadamsdotcom 2y ago
So many highlights from reading this. One that stood out for me is their discovery that refusal works by inhibition: > It turns out that, in Claude, refusal to answer is the default behavior: we find a circuit that is "on" by default and that causes the model to state that it has insufficient information to answer any given question. However, when the model is asked about something it knows well—say, the basketball player Michael Jordan—a competing feature representing "known entities" activates and inhibits this default circuit Many cellular processes work similarly ie. there will be a process that runs as fast as it can and one or more companion “inhibitors” doing a kind of “rate limiting”. Given both phenomena are emergent it makes you wonder if do-but-inhibit is a favored technique of the universe we live in, or just coincidence :)
- colah3 2y ago
  Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and inhibitory edges -- transcription networks, protein networks, networks of biological neurons -- and one can study recurring motifs that turn up in these networks and try to learn from them. It wouldn't be surprising if some lessons from that work may also transfer to artificial neural networks, although there are some technical things to consider.
  - cadamsdotcom 2y ago
    Agreed! So many emergent systems in nature achieve complex outcomes without central coordination - from cellular level to ant colonies & beehives. There are bound to be implications for designed systems. Closely following what you guys are uncovering through interpretability research - not just accepting LLMs as black boxes. Thanks to you & the team for sharing the work with humanity. Interpretability is the most exciting part of AI research for its potential to help us understand what’s in the box. By way of analogy, centuries ago farmers’ best hope for good weather was to pray to the gods! The sooner we escape the “praying to the gods” stage with LLMs the more useful they become.
  - scryers_bloom 2y ago
matthiaspr 2y ago
Interesting paper arguing for deeper internal structure ("biology") beyond pattern matching in LLMs. The examples of abstraction (language-agnostic features, math circuits reused unexpectedly) are compelling against the "just next-token prediction" camp. It sparked a thought: how to test this abstract reasoning directly? Try a prompt with a totally novel rule: “Let's define a new abstract relationship: 'To habogink' something means to perform the action typically associated with its primary function, but in reverse. Example: The habogink of 'driving a car' would be 'parking and exiting the car'. Now, considering a standard hammer, what does it mean 'to habogink a hammer'? Describe the action.” A sensible answer (like 'using the claw to remove a nail') would suggest real conceptual manipulation, not just stats. It tests if the internal circuits enable generalizable reasoning off the training data path. Fun way to probe if the suggested abstraction is robust or brittle.
- ANighRaisin 2y ago
  This is an easy question for LLMs to answer. Gemini 2.0 Flash-Lite can answer this in 0.8 seconds with a cost of 0.0028875 cents: To habogink a hammer means to perform the action typically associated with its primary function, but in reverse. The primary function of a hammer is to drive nails. Therefore, the reverse of driving nails is removing nails. So, to habogink a hammer would be the action of using the claw of the hammer to pull a nail out of a surface.
  - matthiaspr 2y ago
    The goal wasn't to stump the LLM, but to see if it could take a completely novel linguistic token (habogink), understand its defined relationship to other concepts (reverse of primary function), and apply that abstract rule correctly to a specific instance (hammer). The fact that it did this successfully, even if 'easily', suggests it's doing more than just predicting the statistically most likely next token based on prior sequences of 'hammer'. It had to process the definition and perform a conceptual mapping.
    - Sharlin 2y ago
      I think GP's point was that your proposed test is too easy for LLMs to tell us much about how they work. The "habogink" thing is a red herring, really, in practice you're simply asking what the opposite of driving nails into wood is. Which is a trivial question for an LLM to answer. That said, you can teach an LLM as many new words for things as you want and it will use those words naturally, generalizing as needed. Which isn't really a surprise either, given that language is literally the thing that LLMs do best.
YeGoblynQueenne 2y ago
>> Language models like Claude aren't programmed directly by humans—instead, they‘re trained on large amounts of data. Gee, I wonder where this data comes from. Let's think about this step by step. So, what do we know? Language models like Claud are not programmed directly. Wait, does that mean they are programmed indirectly? If so, by whom? Aha, I got it. They are not programmed, directly or indirectly. They are trained on large amounts of data. But that is the question, right? Where does all that data come from? Hm, let me think about it. Oh hang on I got it! Language models are trained on data. But they are language models so the data is language. Aha! And who generates language? Humans! Humans generate language! I got it! Language models are trained on language data generated by humans! Wait, does that mean that language models like Claud are indirectly programmed by humans? That's it! Language models like Claude aren't programmed directly by humans because they are indirectly programmed by humans when they are trained on large amounts of language data generated by humans!
- s3p 2y ago
  I'm struggling to see the point here other than semantics. Indirectly or directly, does this change what they presented?
  - hackernudes 2y ago
    This exchange looks like the output of a so-called "reasoning model". Maybe it is a joke or maybe it is an actual response from an LLM.
- AdieuToLogic 2y ago
  > That's it! Language models like Claude aren't programmed directly by humans because they are indirectly programmed by humans when they are trained on large amounts of language data generated by humans! ... and having large numbers of humans reinforce the applicability ("correctness"), or lack thereof, of generated responses over time.
- kretaceous 2y ago
  This comment looks like the thinking tokens of a reasoning model with "be cheeky" as its system prompt
jasonjmcghee 2y ago
I'm completely hooked. This is such a good paper. It hallucinating how it thinks through things is particularly interesting - not surprising, but cool to confirm. I would LOVE to see Anthropic feed the replacement features output to the model itself and fine tune the model on how it thinks through / reasons internally so it can accurately describe how it arrived at its solutions - and see how it impacts its behavior / reasoning.
navaed01 2y ago
When and how to do stop saying LLMs are predicting the nect set of tokens and start saying they are thinking? Is this the point?
trhway 2y ago
>We find that the shared circuitry increases with model scale, with Claude 3.5 Haiku sharing more than twice the proportion of its features between languages as compared to a smaller model. While it was already generally noticeable, still one more time confirmed that larger model generalizes better instead of using its bigger numbers of parameters just to “memorize by rote” (overfitting).
jaakl 2y ago
My main takeaway here is that the models cannot tell know how they really work, and asking it from them is just returning whatever training dataset would suggest: how a human would explain it. So it does not have self-consciousness, which is of course obvious and we get fooled just like the crowd running away from the arriving train in Lumiére's screening. LLM just fails the famous old test "cogito ergo sum". It has no cognition, ergo they are not agents in more than metaphorical sense. Ergo we are pretty safe from AI singularity.
- Philpax 2y ago
  Do you know how you work?
- famouswaffles 2y ago
  Nearly everything we know about the human body and brain is from the result of centuries of trial and error and experimentation and not any 'intuitive understanding' of our inner workings. Humans cannot tell how they really work either.
ofrzeta 2y ago
Back to the "language of thought" question, this time with LLMs :) https://en.wikipedia.org/wiki/Language_of_thought_hypothesis https://en.wikipedia.org/wiki/Language_of_thought_hypothesis
sgt101 2y ago
>Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. Models aren't trained to do next word prediction though - they are trained to do missing word in this text prediction.
- Philpax 2y ago
  That's true for mask-based training (used for embeddedings and BERT and such), but not true for modern autoregressive LLMs as a whole, which are pretrained with next word prediction.
  - astrange 2y ago
    It's not strictly that though, it's next word prediction with regularization. And the reason LLMs are interesting is that they /fail/ to learn it, but in a good way. If it was a "next word predictor" it wouldn't answer questions but continue them. Also, it's a next token predictor not a word predictor - which is important because the "just a predictor" theory now can't explain how it can form words at all!
    - Philpax 2y ago
      Yes, I know; I was clarifying their immediate misunderstanding using the same terminology as them. There's obviously a lot more going on behind the scenes, especially with today's mid- and post-training work!
hbarka 2y ago
Dario Amodei was in an interview where he said that OpenAI beat them (Anthropic) by mere days to be the first to release. That first move ceded the recognition to ChatGPT but according to Dario it could have been them just the same.
- vessenes 2y ago
  Interesting bit of history! That said, OpenAI’s product team is lights out. I pay for most LLM provider apps, and I use them for different areas of strength, but the ChatGPT product is superior from a user experience perspective.
diedyesterday 2y ago
Regarding the conclusion about language-invariant reasoning (conceptual universality vs. multilingual processing) it helps understanding and becomes somewhat obvious if we regard each language as just a basis of some semantic/logical/thought space in the mind (analogous to the situation in linear algebra and duality of tensors and bases). The thoughts/ideas/concepts/scenarios are invariant states/vector/points in the (very high dimensional) space of meanings in the mind and each language is just a basis to reference/define/express/manipulate those ideas/vectors. A coordinatization of that semantic space. Personally, I'm a multilingual person with native-level command of several languages. Many times it happens, I remember having a specific thought, but don't remember in what language it was. So I can personally sympathize with this finding of the Anthropic researchers.
twoodfin 2y ago
I say this at least 82.764% in jest: Don’t these LLM’s have The Bitter Lesson in their training sets? What are they doing building specialized structures to handle specific needs?
jaehong747 2y ago
I’m skeptical of the claim that Claude “plans” its rhymes. The original example—“He saw a carrot and had to grab it, / His hunger was like a starving rabbit”—is explained as if Claude deliberately chooses “rabbit” in advance. However, this might just reflect learned statistical associations. “Carrot” strongly correlates with “rabbit” (people often pair them), and “grab it” naturally rhymes with “rabbit,” so the model’s activations could simply be surfacing common patterns. The research also modifies internal states—removing “rabbit” or injecting “green”—and sees Claude shift to words like “habit” or end lines with “green.” That’s more about rerouting probabilistic paths than genuine “adaptation.” The authors argue it shows “planning,” but a language model can maintain multiple candidate words at once without engaging in human-like strategy. Finally, “planning ahead” implies a top-down goal and a mechanism for sustaining it, which is a strong assumption. Transformative evidence would require more than observing feature activations. We should be cautious before anthropomorphizing these neural nets.
- rcxdude 2y ago
  It will depend on exactly what you mean by 'planning ahead', but I think the fact that features which rhyme with a word appear before the model is trying to predict the word which needs to rhyme is good evidence the model is planning at least a little bit ahead: the model activations are not all just related to the next token. (And I think it's relatively obvious that the models do this to some degree: it's very hard to write any language at all without 'thinking ahead' at least a little bit in some form, due to the way human language is structured. If models didn't do this and only considered the next token alone they would paint themselves into a corner within a single sentence. Early LLMs like GPT-2 were still pretty bad at this, they were plausible over short windows but there was no consistency to a longer piece of text. Whether this is some high-level abstracted 'train of thought', and how cohesive it is between different forms of it, is a different question. Indeed from the section of jailbreaking it looks like it's often caught out by conflicting goals from different areas of the network which aren't resolved in some logical fashion)
  - jaehong747 2y ago
    Modern transformer-based language models fundamentally lack structures and functions for "thinking ahead." And I don't believe that LLMs have emergently developed human-like thinking abilities. This phenomenon appears because language model performance has improved, and I see it as a reflection of future output token probabilities being incorporated into the probability distribution of the next token set in order to generate meaningful longer sentences. Humans have similar experiences. Everyone has experienced thinking about what to say next while speaking. However, in artificial intelligence language models, this phenomenon occurs mechanically and statistically. What I'm trying to say is that while this phenomenon may appear similar to human thought processes and mechanisms, I'm concerned about the potential anthropomorphic error of assuming machines have consciousness or thoughts.
darkhorse222 2y ago
Once we are aware of these neural pathways I see no reason there shouldn't be a watcher and influencer of the pathways. A bit like a dystopian mind watcher. Shape the brain.
- astrange 2y ago
  https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude
thundervelvet 2y ago
[dead]
teleforce 2y ago
Oh the irony of not able to download the entire paper in one compact PDF format referred in the article, while apparently all the reference citations have PDF of the cited article to be downloaded and accessible from the provided online links [1]. Come on Anthropic, you can do much better than this unconventional and bizarre approach to publication. [1] On the Biology of a Large Language Model: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...
teleforce 2y ago
Another review on the paper from MIT Technology Review [1]. [1] Anthropic can now track the bizarre inner workings of a large language model: https://www.technologyreview.com/2025/03/27/1113916/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model/ https://www.technologyreview.com/2025/03/27/1113916/anthropi...
- miraculixx 2y ago
  Unfortunately that's not a review but a hype-driving oversimplification of what the paper says.
westurner 2y ago
XAI: Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli...
jxjnskkzxxhx 2y ago
CaNt UnDeRsTaNd WhY pEoPlE aRe BuLlIsH. ItS jUsT a StOcHaStIc PaRtOt.

1072 points by Philpax 2y ago

When I want to trace the 'thoughts' of my programs, I just read the code and comments I wrote. Stop LLM anthropomorphizing, please. #SLAP

JPLeRouzic 2y ago

This is extremely interesting: The authors look at features (like making poetry, or calculating) of LLM production, make hypotheses about internal strategies to achieve the result, and experiment with these hypotheses. I wonder if there is somewhere an explanation linking the logical operations made on a on dataset, are resulting in those behaviors?

JKCalhoun 2y ago
And they show the differences when the language models are made larger.

EncomLab 2y ago

This is very interesting - but like all of these discussions it sidesteps the issues of abstractions, compilation, and execution. It's fine to say things like "aren't programmed directly by humans", but the abstracted code is not the program that is running - the compiled code is - and that is code is executing within the tightly bounded constraints of the ISA it is being executed in. Really this is all so much slight of hand - as an esolang fanatic this all feels very familiar. Most people can't look a program written in Whitespace and figure it out either, but once compiled it is just like every other program as far as the processor is concerned. LLM's are no different.

ctoth 2y ago
And DNA? You are running on an instruction set of four symbols at the end of the day but that's the wrong level of abstraction to talk about your humanity, isn't it?
- EncomLab 2y ago
  DNA is the instruction set for protein composition- not thinking.

fpgaminer 2y ago

> This is powerful evidence that even though models are trained to output one word at a time I find this oversimplification of LLMs to be frequently poisonous to discussions surrounding them. No user facing LLM today is trained on next token prediction.

rco8786 2y ago
Super interesting. Can you explain more, or provide some reading? I’m obviously behind
JKCalhoun 2y ago
As a layman though, I often see this description for how it is LLMs work.
- fpgaminer 2y ago
  Right, but it leads to too many false conclusions by lay people. User facing LLMs are only trained on next token prediction during initial stages of their training. They have to go through Reinforcement Learning before they become useful to users, and RL training occurs on complete responses, not just token-by-token. That leads to conclusions elucidated by the very article, that LLMs couldn't possibly plan ahead because they are only trained to predict next tokens. When the opposite conclusion would be more common if it was better understood that they go through RL.
  - mentalgear 2y ago
    What? The "article" is from anthropic, so I think they would know what they write about. Also, RL is an additional training process that does not negate that GPT / transformers are left-right autoencoders that are effectively next token predictors. [Why Can't AI Make Its Own Discoveries? — With Yann LeCun] (https://www.youtube.com/watch?v=qvNCVYkHKfg https://www.youtube.com/watch?v=qvNCVYkHKfg)
    - pipes 2y ago

zerop 2y ago

The explanation of "hallucination" is quite simplified, I am sure there is more there. If there is one problem I have to pick to to trace in LLMs, I would pick hallucination. More tracing of "how much" or "why" model hallucinated can lead to correct this problem. Given the explanation in this post about hallucination, I think degree of hallucination can be given as part of response to the user? I am facing this in RAG use case quite - How do I know model is giving right answer or Hallucinating from my RAG sources?

kittikitti 2y ago
I incredibly regret the term "hallucination" when the confusion matrix exists. There's much more nuance when discussing false positives or false negatives. It also opens discussions on how neural networks are trained, with this concept being crucial in loss functions like categorical cross entropy. In addition, the confusion matrix is how professionals like doctors assess their own performance which "hallucination" would be silly to use. I would go as far to say that it's misleading, or a false positive, to call them hallucinations. If your AI recalls the RAG incorrectly, it's a false positives. If your AI doesn't find the data from the RAG or believes it doesn't exist it's a false negative. Using a term like "hallucination" has no scientific merit.
- esafak 2y ago
  So you never report or pay heed to the overall accuracy?
pcrh 2y ago
The use of the term "hallucination" for LLMs is very deceptive, as it implies that there is a "mind". In ordinary terms, "hallucinations" by a machine would simply be described as the machine being useless, or not fit for purpose. For example, if a simple calculator (or even a person) returned the value "5" for 2+2= , you wouldn't describe it as "hallucinating" the answer....
- astrange 2y ago
  "Hallucination" happened because we got AI images before AI text, but "confabulation" is a better term.

LoganDark 2y ago

LLMs don't think, and LLMs don't have strategies. Maybe it could be argued that LLMs have "derived meaning", but all LLMs do is predict the next token. Even RL just tweaks the next-token prediction process, but the math that drives an LLM makes it impossible for there to be anything that could reasonably be called thought.

yawnxyz 2y ago
rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. Those types of systems are inherently interpretable because you can just look at how they work. They're not black boxes. LLMs are black boxes, and if anything, interpretability systems show us what the heck is going on inside them. Especially useful when half the world is using these already, and we have no idea how t hey work
- kazinator 2y ago
  Water doesn't think, yet if you inject it into the entrance of a maze, it will soon come gushing out of the exit.
- LoganDark 2y ago
  > rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. That doesn't mean the water itself has strategies, just that you can use water in an implementation of strategy... it's fairly well known at this point that LLMs can be used as part of strategies (see e.g. "agents"), they just don't intrinsically have any.
ajkdhcb2 2y ago
True. People use completely unjustified anthropomorphised terminology for marketing reasons and it bothers me a lot. I think it actually holds back understanding how it works. "Hallucinate" is the worst - it's an error and undesired result, not a person having a psychotic episode
kazinator 2y ago

kittikitti 2y ago

What's the point of this when Claude isn't open sourced and we just have to take Anthropic's word for it?

ctoth 2y ago
> What's the point of this - That similar interpretability tools might be useful to the open source community? - That this is a fruitful area to research?
- kittikitti 2y ago
  Can you use those same tools on Claude? Is the difference trivial from open source models?
  - ctoth 2y ago
    https://news.ycombinator.com/item?id=42208383 https://news.ycombinator.com/item?id=42208383 > Show HN: Llama 3.2 Interpretability with Sparse Autoencoders > 579 points by PaulPauls 4 months ago | hide | past | favorite | 100 comments > I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. > I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback.
probably_wrong 2y ago
I blame the scientific community for blindly accepting OpenAI's claims about GPT-3 despite them refusing to release their model. The tech community hyping every press release didn't help either. I hope one day the community starts demanding verifiable results before accepting them, but I fear that ship may have already sailed.

Hansenq 2y ago

I wonder how much of these conclusions are Claude-specific (given that Anthropic only used Claude as a test subject) or if they extrapolate to other transformer-based models as well. Would be great to see the research tested on Llama and the Deepseek models, if possible!

marcelsalathe 2y ago

I’ve only skimmed the paper - a long and dense read - but it’s already clear it’ll become a classic. What’s fascinating is that engineering is transforming into a science, trying to understand precisely how its own creations work This shift is more profound than many realize. Engineering traditionally applied our understanding of the physical world, mathematics, and logic to build predictable things. But now, especially in fields like AI, we’ve built systems so complex we no longer fully understand them. We must now use scientific methods - originally designed to understand nature - to comprehend our own engineered creations. Mindblowing.

ctoth 2y ago
This "practice-first, theory-later" pattern has been the norm rather than the exception. The steam engine predated thermodynamics. People bred plants and animals for thousands of years before Darwin or Mendel. The few "top-down" examples where theory preceded application (like nuclear energy or certain modern pharmaceuticals) are relatively recent historical anomalies.
- marcelsalathe 2y ago
  I see your point, but something still seems different. Yes we bred plants and animals, but we did not create them. Yes we did build steam engines before understanding thermodynamics but we still understood what they did (heat, pressure, movement, etc.) Fun fact: we have no clue how most drugs works. Or, more precisely, we know a few aspects, but are only scratching the surface. We're even still discovering news things about Aspirin, one of the oldest drugs: https://www.nature.com/articles/s41586-025-08626-7 https://www.nature.com/articles/s41586-025-08626-7
  - tmp10423288442 2y ago
    > Yes we did build steam engines before understanding thermodynamics but we still understood what it did (heat, pressure, movement, etc.) We only understood in the broadest sense. It took a long process of iteration before we could create steam engines that were efficient enough to start an Industrial Revolution. At the beginning they were so inefficient that they could only pump water from the same coal mine they got their fuel from, and subject to frequent boiler explosions besides.

aithrowawaycomm 2y ago

I struggled reading the papers - Anthropic’s white papers reminds me of Stephen Wolfram, where it’s a huge pile of suggestive empirical evidence, but the claims are extremely vague - no definitions, just vibes - the empirical evidence seems selectively curated, and there’s not much effort spent building a coherent general theory. Worse is the impression that they are begging the question. The rhyming example was especially unconvincing since they didn’t rule out the possibility that Claude activated “rabbit” simply because it wrote a line that said “carrot”; later Anthropic claimed Claude was able to “plan” when the concept “rabbit” was replaced by “green,” but the poem fails to rhyme because Claude arbitrarily threw in the word “green”! What exactly was the plan? It looks like Claude just hastily autocompleted. And Anthropic made zero effort to reproduce this experiment, so how do we know it’s a general phenomenon? I don’t think either of these papers would be published in a reputable journal. If these papers are honest, they are incomplete: they need more experiments and more rigorous methodology. Poking at a few ANN layers and making sweeping claims about the output is not honest science. But I don’t think Anthropic is being especially honest: these are pseudoacademic infomercials.

TimorousBestie 2y ago
Agreed. They’ve discovered something, that’s for sure, but calling it “the language of thought” without concrete evidence is definitely begging the question.
famouswaffles 2y ago
>The rhyming example was especially unconvincing since they didn’t rule out the possibility that Claude activated “rabbit” simply because it wrote a line that said “carrot” I'm honestly confused at what you're getting at here. It doesn't matter why Claude chose rabbit to plan around and in fact likely did do so because of carrot, the point is that it thought about it beforehand. The rabbit concept is present as the model is about to write the first word of the second line even though the word rabbit won't come into play till the end of the line. >later Anthropic claimed Claude was able to “plan” when the concept “rabbit” was replaced by “green,” but the poem fails to rhyme because Claude arbitrarily threw in the word “green”! It's not supposed to rhyme. That's the point. They forced Claude to plan around a line ender that doesn't rhyme and it did. Claude didn't choose the word green, anthropic replaced the concept it was thinking ahead about with green and saw that the line changed accordingly.

smath 2y ago

Reminds me of the term 'system identification' from old school control systems theory, which meant poking around a system and measuring how it behaves, - like sending an input impulse and measuring its response, does it have memory, etc. https://en.wikipedia.org/wiki/System_identification https://en.wikipedia.org/wiki/System_identification

Loic 2y ago
It is not old school, this is my daily job and we need even more of it with the NN models used in MPC.
- nomel 2y ago
  I've looked into using NN for some of my specific work, but making sure output is bounded ends up being such a big issue that the very code/checks required to make sure it's within acceptable specs, in a deterministic way, ends up being an acceptable solution, making the NN unnecessary. How do you handle that sort of thing? Maybe main process then leave some relatively small residual to the NN? Is your poking more like "fuzzing", where you just perturb all the input parameters in a relatively "complete" way to try to find if anything goes wild? I'm very interested in the details behind "critical" type use cases of NN, which I've never been able to stomach in my work.
  - lqr 2y ago
    This paper may be interesting to you. It touches on several of the topics you mentioned: https://www.science.org/doi/10.1126/scirobotics.abm6597 https://www.science.org/doi/10.1126/scirobotics.abm6597
  - Loic 2y ago
    For us, the NN is used in a grey box model for MPC in chemical engineering. The factories we control have relatively long characteristic time, together with all the engineering bounds, we can use the NN to model parts of the equipment from raw DCS data. The NN modeled parts are usually not the most critical (we are 1st principles based for them) but this allows us to quickly fit/deploy a new MPC in production. Faster time to market/production is for us the main reason/advantage of the approach.

jacooper 2y ago

So it turns out, it's not just simple next token generation, there is intelligence and self developed solution methods (Algorithms) in play, particularly in the math example. Also the multi language finding negates, at least partially, the idea that LLMs, at least large ones, don't have an understanding of the world beyond the prompt. This changed my outlook regarding LLMs, ngl.

miraculixx 2y ago
There is no evidence to this end. There is lots of claims, but claims are not evidence.

kazinator 2y ago

> Claude writes text one word at a time. Is it only focusing on predicting the next word or does it ever plan ahead? When a LLM outputs a word, it commits to that word, without knowing what the next word is going to be. Commits meaning once it settles on that token, it will not backtrack. That is kind of weird. Why would you do that, and how would you be sure? People can sort of do that too. Sometimes? Say you're asked to describe a 2D scene in which a blue triangle partially occludes a red circle. Without thinking about the relationship of the objects at all, you know that your first word is going to be "The" so you can output that token into your answer. And then that the sentence will need a subject which is going to be "blue", "triangle". You can commit to the tokens "The blue triangle" just from knowing that you are talking about a 2D scene with a blue triangle in it, without considering how it relates to anything else, like the red circle. You can perhaps commit to the next token "is", if you have a way to express any possible relationship using the word "to be", such as "the blue circle is partially covering the red circle". I don't think this analogy necessarily fits what LLMs are doing.

kazinator 2y ago
By the way, there was recently a HN submission about a project studying using diffusion models rather than LLM for token prediction. With diffusion, tokens aren't predicted strictly left to right any more; there can be gaps that are backfilled. But: it's still essentially the same, I think. Once that type of model settles on a given token at a given position, it commits to that. Just more possible permutations of the token filling sequence have ben permitted.
pants2 2y ago
> it commits to that word, without knowing what the next word is going to be Sounds like you may not have read the article, because it's exploring exactly that relationship and how LLMs will often have a 'target word' in mind that it's working toward. Further, that's partially the point of thinking models, allowing LLMs space to output tokens that it doesn't have to commit to in the final answer.
- kazinator 2y ago

polygot 2y ago

There needs to be some more research on what path the model takes to reach its goal, perhaps there is a lot of overlap between this and the article. The most efficient way isn't always the best way. For example, I asked Claude-3.7 to make my tests pass in my C# codebase. It did, however, it wrote code to detect if a test runner was running, then return true. The tests now passed, so, it achieved the goal, and the code diff was very small (10-20 lines.) The actual solution was to modify about 200-300 lines of code to add a feature (the tests were running a feature that did not yet exist.)

felbane 2y ago
Ah yes, the "We have a problem over there/I'll just delete 'over there'" approach.
- polygot 2y ago
  I've also had this issue, where failing tests are deleted to make all the tests pass, or, it mocks a failing HTTP request and hardcodes it to 200 OK.
  - ctoth 2y ago
    Reward hacking, as predicted over and over again. You hate to see it. Let him with ears &c.
brulard 2y ago
That is called "Volkswagen" testing. Some years ago that automaker had mechanism in cars which detected when the vehicle was being examined and changed something so it would pass the emission tests. There are repositories on github that make fun of it.
- rsynnott 2y ago
  While that’s the most famous example, this sort of cheating is much older than that. In the good old days before 3d acceleration, graphics card vendors competed mostly on 2d acceleration. This mostly involved routines to accelerate drawing Windows windows and things, and benchmarks tended to do things like move windows round really fast. It was somewhat common for card drivers to detect that a benchmark was running, and just fake the whole thing; what was being drawn on the screen was wrong, but since the benchmarks tended to be a blurry mess anyway the user would have a hard time realising this.

osigurdson 2y ago

>> Claude can speak dozens of languages. What language, if any, is it using "in its head"? I would have thought that there would be some hints in standard embeddings. I.e., the same concept, represented in different languages translates to vectors that are close to each other. It seems reasonable that an LLM would create its own embedding models implicitly.

generalizations 2y ago
Who's to say Claude isn't inherently a shape rotator, anyway?
iNic 2y ago
There are: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-multilingual https://transformer-circuits.pub/2025/attribution-graphs/bio...

greesil 2y ago

What is a "thought"?

TechDebtDevin 2y ago

>>Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. This always seemed obvious to me or that LLMs were completing the next most likely sentence or multiple words.

deleted 2y ago
[deleted]

indigoabstract 2y ago

While reading the article I enjoyed pretending that a powerful LLM just crash landed on our planet and researchers at Anthropic are now investigating this fascinating piece of alien technology and writing about their discoveries. It's a black box, nobody knows how its inhuman brain works, but with each step, we're finding out more and more. It seems like quite a paradox to build something but to not know how it actually works and yet it works. This doesn't seem to happen very often in classical programming, does it?

BoredPositron 2y ago
The bigger problem is that nobody knows how a human brain works that’s the real crux with the analogy.
- richardatlarge 2y ago
  I would say that nobody agrees, not that nobody knows. And it’s reductionist to think that the brain works one way. Different cultures produce different brains, possible because of the utter plasticity of the learning nodes. Chess has a few rules, maybe the brain has just a few as well. How else can the same brain of 50k years ago still function today? I think we do understand the learning part of the brain, but we don’t like the image it casts, so we reject it
  - wat10000 2y ago
    That gets down to what it means to “know” something. Nobody agrees because there isn’t enough information available. Some people might have the right idea by luck, but do you really know something if you don’t have a solid basis for your belief but it happens to be correct?
    - richardatlarge 2y ago
      Potentially true, but I don’t think so. I believe it is understood and unless you’re familiar with every neuro/behavioral literature, you can’t know. Science paradigms are driven by many factors and being powerfully correct does not necessarily rank high when the paradigms implications are unpopular

d--b 2y ago

> This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. Suggesting that an awful lot of calculations are unnecessary in LLMs!

bonoboTP 2y ago
Yeah, it always seemed pretty wasteful to me. In every single forward pass the LLM must basically start out from scratch, without all the forward-looking plans it made the previous times, and must figure out what we are doing, where we are in the generation process, as in the movie Memento, waking up after an episode of amnesia, except you're waking up in the middle of typing out a sentence, you can look at the previous typed words, but can't carry your future plans with you ahead to the next word. At the next word, you (your clone) again wakes up and must figure out from scratch what it is that we are supposed to be typing out. The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. That would basically turn the thing into a recurrent network though. And those are more difficult to train and have a host of issues. Maybe there will be a better way.
- colah3 2y ago
  > The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretability research at Anthropic. That's a great intuition, and in fact the transformer architecture actually does exactly what you suggest! Activations from earlier time steps are sent forward to later time steps via attention. (This is another thing that's lost in the "models just predict the next word" framing.) This actually has interesting practical implications -- for example, in some sense, it's the deep reason costs can sometimes be reduced via "prompt caching".
  - bonoboTP 2y ago
    I'm more a vision person, and haven't looked a lot into NLP transformers, but is this because the attention is masked to only allow each query to look at keys/values from its own past? So when we are at token #5, then token #3's query cannot attend to token #4's info? And hence the previously computed attention values and activations remain the same and can be cached, because it would anyway be the same in the new forward pass?

annoyingnoob 2y ago

Do LLMs "think"? I have trouble with the title, claiming that LLMs have thoughts.

danielbln 2y ago
What's "thinking"?
- annoyingnoob 2y ago
  Why the need to anthropomorphize AI? Why does AI think vs process or interpret or apply previous calculated statistical weights or anything other than think? I would argue that binary systems built on silicon are fundamentally different that human biology and deserve to be described differently, not forced into the box of human biology.

deadbabe 2y ago

We really need to work on popularizing better, non-anthropomorphic terms for LLMs, as they don’t really have “thoughts” the way people think. Such terms make people more susceptible to magical thinking.

danielbln 2y ago
Could you argue why they don't? And could you also argue why we do?
- rongrobert 2y ago
  When a car moves over the ground, we do not call that running, we call that driving as to not confuse the mechanism of the output. Both running and driving are moving over the ground but with entirely different mechanisms. I imagine saying the LLM has thoughts is like pretending the car has wheels for legs and is running over the ground. It is not completely wrong but misleading and imprecise.
  - danielbln 2y ago
    Planes fly, birds fly. They use related, but ultimately quite different mechanisms to do so. Yet we call both flying.
    - miraculixx 2y ago
      To fly means "to soar through air; move through the air with wings" (etymonline) That is pretty much an accurate discription of what planes and birds do. To plan means "to reason with intent". That is very much not what LLMs do, and the paper does not provide evidence to the contrary. Yet it uses the term to give credence to it's rather speculative interpretation of observed correlation as causation. Interestingly enough there is no definition of the term, which at least would help to understand what the authors actually mean. I would be more inclined to take a more positive stance to the paper if it used more appropriate terms, such as call observed correlations just that. Granted that would possibly make for much less of a fancy title.

davidmurphy 2y ago

On a somewhat related note, check out the video of Tuesday's Computer History Museum x IEEE Spectrum event, "The Great Chatbot Debate: Do LLMs Really Understand?" Speakers: Sébastien Bubeck (OpenAI) and Emily M. Bender (University of Washington). Moderator: Eliza Strickland (IEEE Spectrum). Video: https://youtu.be/YtIQVaSS5Pg https://youtu.be/YtIQVaSS5Pg Info: https://computerhistory.org/events/great-chatbot-debate/ https://computerhistory.org/events/great-chatbot-debate/

0x70run 2y ago

I would pay to watch James Mickens comment on this stuff.

a3w 2y ago

Article and papers looks good. Video seems misleading, since I can use optimization pressure and local minima to explain the model behaviour. No "thinking" required, which the video claims is proven.

mvATM99 2y ago

What a great article, i always like how much Anthropic focuses on explainability, something vastly ignored by most. The multi-step reasoning section is especially good food for thought.

deleted 2y ago

[deleted]

moralestapia 2y ago

[flagged]

SkyBelow 2y ago

>Claude speaks dozens of languages fluently—from English and French to Chinese and Tagalog. How does this multilingual ability work? Is there a separate "French Claude" and "Chinese Claude" running in parallel, responding to requests in their own language? Or is there some cross-lingual core inside? I have an interesting test case for this. Take a popular enough Japanese game that has been released for long enough for social media discussions to be in the training data, but not so popular to have an English release yet. Then ask it a plot question, something major enough to be discussed, but enough of a spoiler that it won't show up in marketing material. Does asking in Japanese have it return information that is lacking when asked in English, or can it answer the question in English based on the information in learned in Japanese? I tried this recently with a JRPG that was popular enough to have a fan translation but not popular enough to have a simultaneous English release. English did not know the plot point, but I didn't have the Japanese skill to confirm if the Japanese version knew the plot point, or if discussion was too limited for the AI to be aware of it. It did know of the JRPG and did know of the marketing material around it, so it wasn't simply a case of my target being too niche.

modeless 2y ago

> In the poetry case study, we had set out to show that the model didn't plan ahead, and found instead that it did. I'm surprised their hypothesis was that it doesn't plan. I don't see how it could produce good rhymes without planning.

ripped_britches 2y ago
It would be really hard to get such good results on coding challenges without planning. This is indeed an odd hypothesis.

alach11 2y ago

Fascinating papers. Could deliberately suppressing memorization during pretraining help force models to develop stronger first-principles reasoning?

HocusLocus 2y ago

[Tracing the thoughts of a large language model] "What have I gotten myself into??"

0xbadcafebee 2y ago

AI "thinks" like a piece of rope in a dryer "thinks" in order to come to an advanced knot: a whole lot of random jumbling that eventually leads to a complex outcome.

cheeze 2y ago
I regularly see this but I feel like it's disingenuous. Akin to saying "if we simulate enough monkies on a typewriter, we'll eventually get the right result"
- zvitiate 2y ago
  If we could motivate the monkies sufficiently with bananas, we'd probably improve those odds substantially.
FeepingCreature 2y ago
Ah yes, I too often have extended English conversation with my washed rope.

cadamsdotcom 2y ago

So many highlights from reading this. One that stood out for me is their discovery that refusal works by inhibition: > It turns out that, in Claude, refusal to answer is the default behavior: we find a circuit that is "on" by default and that causes the model to state that it has insufficient information to answer any given question. However, when the model is asked about something it knows well—say, the basketball player Michael Jordan—a competing feature representing "known entities" activates and inhibits this default circuit Many cellular processes work similarly ie. there will be a process that runs as fast as it can and one or more companion “inhibitors” doing a kind of “rate limiting”. Given both phenomena are emergent it makes you wonder if do-but-inhibit is a favored technique of the universe we live in, or just coincidence :)

colah3 2y ago
Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and inhibitory edges -- transcription networks, protein networks, networks of biological neurons -- and one can study recurring motifs that turn up in these networks and try to learn from them. It wouldn't be surprising if some lessons from that work may also transfer to artificial neural networks, although there are some technical things to consider.
- cadamsdotcom 2y ago
  Agreed! So many emergent systems in nature achieve complex outcomes without central coordination - from cellular level to ant colonies & beehives. There are bound to be implications for designed systems. Closely following what you guys are uncovering through interpretability research - not just accepting LLMs as black boxes. Thanks to you & the team for sharing the work with humanity. Interpretability is the most exciting part of AI research for its potential to help us understand what’s in the box. By way of analogy, centuries ago farmers’ best hope for good weather was to pray to the gods! The sooner we escape the “praying to the gods” stage with LLMs the more useful they become.
- scryers_bloom 2y ago

matthiaspr 2y ago

Interesting paper arguing for deeper internal structure ("biology") beyond pattern matching in LLMs. The examples of abstraction (language-agnostic features, math circuits reused unexpectedly) are compelling against the "just next-token prediction" camp. It sparked a thought: how to test this abstract reasoning directly? Try a prompt with a totally novel rule: “Let's define a new abstract relationship: 'To habogink' something means to perform the action typically associated with its primary function, but in reverse. Example: The habogink of 'driving a car' would be 'parking and exiting the car'. Now, considering a standard hammer, what does it mean 'to habogink a hammer'? Describe the action.” A sensible answer (like 'using the claw to remove a nail') would suggest real conceptual manipulation, not just stats. It tests if the internal circuits enable generalizable reasoning off the training data path. Fun way to probe if the suggested abstraction is robust or brittle.

ANighRaisin 2y ago
This is an easy question for LLMs to answer. Gemini 2.0 Flash-Lite can answer this in 0.8 seconds with a cost of 0.0028875 cents: To habogink a hammer means to perform the action typically associated with its primary function, but in reverse. The primary function of a hammer is to drive nails. Therefore, the reverse of driving nails is removing nails. So, to habogink a hammer would be the action of using the claw of the hammer to pull a nail out of a surface.
- matthiaspr 2y ago
  The goal wasn't to stump the LLM, but to see if it could take a completely novel linguistic token (habogink), understand its defined relationship to other concepts (reverse of primary function), and apply that abstract rule correctly to a specific instance (hammer). The fact that it did this successfully, even if 'easily', suggests it's doing more than just predicting the statistically most likely next token based on prior sequences of 'hammer'. It had to process the definition and perform a conceptual mapping.
  - Sharlin 2y ago
    I think GP's point was that your proposed test is too easy for LLMs to tell us much about how they work. The "habogink" thing is a red herring, really, in practice you're simply asking what the opposite of driving nails into wood is. Which is a trivial question for an LLM to answer. That said, you can teach an LLM as many new words for things as you want and it will use those words naturally, generalizing as needed. Which isn't really a surprise either, given that language is literally the thing that LLMs do best.

YeGoblynQueenne 2y ago

>> Language models like Claude aren't programmed directly by humans—instead, they‘re trained on large amounts of data. Gee, I wonder where this data comes from. Let's think about this step by step. So, what do we know? Language models like Claud are not programmed directly. Wait, does that mean they are programmed indirectly? If so, by whom? Aha, I got it. They are not programmed, directly or indirectly. They are trained on large amounts of data. But that is the question, right? Where does all that data come from? Hm, let me think about it. Oh hang on I got it! Language models are trained on data. But they are language models so the data is language. Aha! And who generates language? Humans! Humans generate language! I got it! Language models are trained on language data generated by humans! Wait, does that mean that language models like Claud are indirectly programmed by humans? That's it! Language models like Claude aren't programmed directly by humans because they are indirectly programmed by humans when they are trained on large amounts of language data generated by humans!

s3p 2y ago
I'm struggling to see the point here other than semantics. Indirectly or directly, does this change what they presented?
- hackernudes 2y ago
  This exchange looks like the output of a so-called "reasoning model". Maybe it is a joke or maybe it is an actual response from an LLM.
AdieuToLogic 2y ago
> That's it! Language models like Claude aren't programmed directly by humans because they are indirectly programmed by humans when they are trained on large amounts of language data generated by humans! ... and having large numbers of humans reinforce the applicability ("correctness"), or lack thereof, of generated responses over time.
kretaceous 2y ago
This comment looks like the thinking tokens of a reasoning model with "be cheeky" as its system prompt

jasonjmcghee 2y ago

I'm completely hooked. This is such a good paper. It hallucinating how it thinks through things is particularly interesting - not surprising, but cool to confirm. I would LOVE to see Anthropic feed the replacement features output to the model itself and fine tune the model on how it thinks through / reasons internally so it can accurately describe how it arrived at its solutions - and see how it impacts its behavior / reasoning.

navaed01 2y ago

When and how to do stop saying LLMs are predicting the nect set of tokens and start saying they are thinking? Is this the point?

trhway 2y ago

>We find that the shared circuitry increases with model scale, with Claude 3.5 Haiku sharing more than twice the proportion of its features between languages as compared to a smaller model. While it was already generally noticeable, still one more time confirmed that larger model generalizes better instead of using its bigger numbers of parameters just to “memorize by rote” (overfitting).

jaakl 2y ago

My main takeaway here is that the models cannot tell know how they really work, and asking it from them is just returning whatever training dataset would suggest: how a human would explain it. So it does not have self-consciousness, which is of course obvious and we get fooled just like the crowd running away from the arriving train in Lumiére's screening. LLM just fails the famous old test "cogito ergo sum". It has no cognition, ergo they are not agents in more than metaphorical sense. Ergo we are pretty safe from AI singularity.

Philpax 2y ago
Do you know how you work?
famouswaffles 2y ago
Nearly everything we know about the human body and brain is from the result of centuries of trial and error and experimentation and not any 'intuitive understanding' of our inner workings. Humans cannot tell how they really work either.

ofrzeta 2y ago

Back to the "language of thought" question, this time with LLMs :) https://en.wikipedia.org/wiki/Language_of_thought_hypothesis https://en.wikipedia.org/wiki/Language_of_thought_hypothesis

sgt101 2y ago

>Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so. Models aren't trained to do next word prediction though - they are trained to do missing word in this text prediction.

Philpax 2y ago
That's true for mask-based training (used for embeddedings and BERT and such), but not true for modern autoregressive LLMs as a whole, which are pretrained with next word prediction.
- astrange 2y ago
  It's not strictly that though, it's next word prediction with regularization. And the reason LLMs are interesting is that they /fail/ to learn it, but in a good way. If it was a "next word predictor" it wouldn't answer questions but continue them. Also, it's a next token predictor not a word predictor - which is important because the "just a predictor" theory now can't explain how it can form words at all!
  - Philpax 2y ago
    Yes, I know; I was clarifying their immediate misunderstanding using the same terminology as them. There's obviously a lot more going on behind the scenes, especially with today's mid- and post-training work!

hbarka 2y ago

Dario Amodei was in an interview where he said that OpenAI beat them (Anthropic) by mere days to be the first to release. That first move ceded the recognition to ChatGPT but according to Dario it could have been them just the same.

vessenes 2y ago
Interesting bit of history! That said, OpenAI’s product team is lights out. I pay for most LLM provider apps, and I use them for different areas of strength, but the ChatGPT product is superior from a user experience perspective.

diedyesterday 2y ago

Regarding the conclusion about language-invariant reasoning (conceptual universality vs. multilingual processing) it helps understanding and becomes somewhat obvious if we regard each language as just a basis of some semantic/logical/thought space in the mind (analogous to the situation in linear algebra and duality of tensors and bases). The thoughts/ideas/concepts/scenarios are invariant states/vector/points in the (very high dimensional) space of meanings in the mind and each language is just a basis to reference/define/express/manipulate those ideas/vectors. A coordinatization of that semantic space. Personally, I'm a multilingual person with native-level command of several languages. Many times it happens, I remember having a specific thought, but don't remember in what language it was. So I can personally sympathize with this finding of the Anthropic researchers.

twoodfin 2y ago

I say this at least 82.764% in jest: Don’t these LLM’s have The Bitter Lesson in their training sets? What are they doing building specialized structures to handle specific needs?

jaehong747 2y ago

I’m skeptical of the claim that Claude “plans” its rhymes. The original example—“He saw a carrot and had to grab it, / His hunger was like a starving rabbit”—is explained as if Claude deliberately chooses “rabbit” in advance. However, this might just reflect learned statistical associations. “Carrot” strongly correlates with “rabbit” (people often pair them), and “grab it” naturally rhymes with “rabbit,” so the model’s activations could simply be surfacing common patterns. The research also modifies internal states—removing “rabbit” or injecting “green”—and sees Claude shift to words like “habit” or end lines with “green.” That’s more about rerouting probabilistic paths than genuine “adaptation.” The authors argue it shows “planning,” but a language model can maintain multiple candidate words at once without engaging in human-like strategy. Finally, “planning ahead” implies a top-down goal and a mechanism for sustaining it, which is a strong assumption. Transformative evidence would require more than observing feature activations. We should be cautious before anthropomorphizing these neural nets.

rcxdude 2y ago
It will depend on exactly what you mean by 'planning ahead', but I think the fact that features which rhyme with a word appear before the model is trying to predict the word which needs to rhyme is good evidence the model is planning at least a little bit ahead: the model activations are not all just related to the next token. (And I think it's relatively obvious that the models do this to some degree: it's very hard to write any language at all without 'thinking ahead' at least a little bit in some form, due to the way human language is structured. If models didn't do this and only considered the next token alone they would paint themselves into a corner within a single sentence. Early LLMs like GPT-2 were still pretty bad at this, they were plausible over short windows but there was no consistency to a longer piece of text. Whether this is some high-level abstracted 'train of thought', and how cohesive it is between different forms of it, is a different question. Indeed from the section of jailbreaking it looks like it's often caught out by conflicting goals from different areas of the network which aren't resolved in some logical fashion)
- jaehong747 2y ago
  Modern transformer-based language models fundamentally lack structures and functions for "thinking ahead." And I don't believe that LLMs have emergently developed human-like thinking abilities. This phenomenon appears because language model performance has improved, and I see it as a reflection of future output token probabilities being incorporated into the probability distribution of the next token set in order to generate meaningful longer sentences. Humans have similar experiences. Everyone has experienced thinking about what to say next while speaking. However, in artificial intelligence language models, this phenomenon occurs mechanically and statistically. What I'm trying to say is that while this phenomenon may appear similar to human thought processes and mechanisms, I'm concerned about the potential anthropomorphic error of assuming machines have consciousness or thoughts.

darkhorse222 2y ago

Once we are aware of these neural pathways I see no reason there shouldn't be a watcher and influencer of the pathways. A bit like a dystopian mind watcher. Shape the brain.

astrange 2y ago
https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude

thundervelvet 2y ago

[dead]

teleforce 2y ago

Oh the irony of not able to download the entire paper in one compact PDF format referred in the article, while apparently all the reference citations have PDF of the cited article to be downloaded and accessible from the provided online links [1]. Come on Anthropic, you can do much better than this unconventional and bizarre approach to publication. [1] On the Biology of a Large Language Model: https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...

teleforce 2y ago

Another review on the paper from MIT Technology Review [1]. [1] Anthropic can now track the bizarre inner workings of a large language model: https://www.technologyreview.com/2025/03/27/1113916/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model/ https://www.technologyreview.com/2025/03/27/1113916/anthropi...

miraculixx 2y ago
Unfortunately that's not a review but a hype-driving oversimplification of what the paper says.

westurner 2y ago

XAI: Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli...

jxjnskkzxxhx 2y ago

CaNt UnDeRsTaNd WhY pEoPlE aRe BuLlIsH. ItS jUsT a StOcHaStIc PaRtOt.