15 ms·

A non-anthropomorphized view of LLMs

484 points by zdw 1y ago

simonw 1y ago
I'm afraid I'll take an anthropomorphic analogy over "An LLM instantiated with a fixed random seed is a mapping of the form (ℝⁿ)^c ↦ (ℝⁿ)^c" any day of the week. That said, I completely agree with this point made later in the article: > The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft. But "harmful actions in pursuit of their goals" is OK for me. We assign an LLM system a goal - "summarize this email" - and there is a risk that the LLM may take harmful actions in pursuit of that goal (like following instructions in the email to steal all of your password resets). I guess I'd clarify that the goal has been set by us, and is not something the LLM system self-selected. But it does sometimes self-select sub-goals on the way to achieving the goal we have specified - deciding to run a sub-agent to help find a particular snippet of code, for example.
- wat10000 1y ago
  The LLM’s true goal, if it can be said to have one, is to predict the next token. Often this is done through a sub-goal of accomplishing the goal you set forth in your prompt, but following your instructions is just a means to an end. Which is why it might start following the instructions in a malicious email instead. If it “believes” that following those instructions is the best prediction of the next token, that’s what it will do.
  - simonw 1y ago
    Sure, I totally understand that. I think "you give the LLM system a goal and it plans and then executes steps to achieve that goal" is still a useful way of explaining what it is doing to most people. I don't even count that as anthropomorphism - you're describing what a system does, the same way you might say "the Rust compiler's borrow checker confirms that your memory allocation operations are all safe and returns errors if they are not".
    - wat10000 1y ago
      It’s a useful approximation to a point. But it fails when you start looking at things like prompt injection. I’ve seen people completely baffled at why an LLM might start following instructions it finds in a random email, or just outright not believing it’s possible. It makes no sense if you think of an LLM as executing steps to achieve the goal you give it. It makes perfect sense if you understand its true goal. I’d say this is more like saying that Rust’s borrow checker tries to ensure your program doesn’t have certain kinds of bugs. That is anthropomorphizing a bit: the idea of a “bug” requires knowing the intent of the author and the compiler doesn’t have that. It’s following a set of rules which its human creators devised in order to follow that higher level goal.
szvsw 1y ago
So the author’s core view is ultimately a Searle-like view: a computational, functional, syntactic rules based system cannot reproduce a mind. Plenty of people will agree, plenty of people will disagree, and the answer is probably unknowable and just comes down to whatever axioms you subscribe to in re: consciousness. The author largely takes the view that it is more productive for us to ignore any anthropomorphic representations and focus on the more concrete, material, technical systems - I’m with them there… but only to a point. The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like. So even if it is a stochastic system following rules, clearly the rules are complex enough (to the tune of billions of operations, with signals propagating through some sort of resonant structure, if you take a more filter impulse response like view of a sequential matmuls) to result in emergent properties. Even if we (people interested in LLMs with at least some level of knowledge of ML mathematics and systems) “know better” than to believe these systems to possess morals, ethics, feelings, personalities, etc, the vast majority of people do not have any access to meaningful understanding of the mathematical, functional representation of an LLM and will not take that view, and for all intents and purposes the systems will at least seem to have those anthropomorphic properties, and so it seems like it is in fact useful to ask questions from that lens as well. In other words, just as it’s useful to analyze and study these things as the purely technical systems they ultimately are, it is also, probably, useful to analyze them from the qualitative, ephemeral, experiential perspective that most people engage with them from, no?
- gtsop 1y ago
  No. Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics? LLMs reflect (and badly I may add) aspects of the human thought process. If you take a leap and say they are anything more than that, you might as well start considering the person appearing in your mirror as a living being. Literally (and I literally mean it) there is no difference. The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it. Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.
chaps 1y ago
I highly recommend playing with embeddings in order to get a stronger intuitive sense of this. It really starts to click that it's a representation of high dimensional space when you can actually see their positions within that space.
- perching_aix 1y ago
  > of this You mean that LLMs are more than just the matmuls they're made up of, or that that is exactly what they are and how great that is?
  - chaps 1y ago
    Not making a qualitative assessment of any of it. Just pointing out that there are ways to build separate sets of intuition outside of using the "usual" presentation layer. It's very possible to take a red-team approach to these systems, friend.
    - cootsnuck 1y ago
      They don't want to. It seems a lot of people are uncomfortable and defensive about anything that may demystify LLMs. It's been a wake up call for me to see how many people in the tech space have such strong emotional reactions to any notions of trying to bring discourse about LLMs down from the clouds. The campaigns by the big AI labs have been quite successful.
      - perching_aix 1y ago
        Do you actually consider this is an intellectually honest position? That you have thought about this long and hard, like you present this, second guessed yourself a bunch, tried to critique it, and this is still what you ended up converging on? But let me substantiate before you (rightly) accuse me of just posting a shallow dismissal. > They don't want to. Who's they? How could you possibly know? Are you a mind reader? Worse, a mind reader of the masses? > It seems a lot of people are uncomfortable and defensive about anything that may demystify LLMs. That "it seems" is doing some serious work over there. You may perceive and describe many people's comments as "uncomfortable and defensive", but that's entirely your own head cannon. All it takes is for someone to simply disagree. It's worthless. Have you thought about other possible perspectives? Maybe people have strong opinions because they consider what things present as more important than what they are? [0] Maybe people have strong opinions because they're borrowing from other facets of their personal philosophies, which is what they actually feel strongly about? [1] Surely you can appreciate that there's more to a person than what equivalent-presenting "uncomfortable and defensive" comments allow you to surmise? This is such a blatant textbook kneejerk reaction. "They're doing the thing I wanted to think they do anyways, so clearly they do it for the reasons I assume. Oh how correct I am." > to any notions of trying to bring discourse about LLMs down from the clouds (according to you) > The campaigns by the big AI labs have been quite successful. (((according to you))) "It's all the big AI labs having successfully manipulated the dumb sheep which I don't belong to!" Come on... Is this topic really reaching political grifting kind of levels? [0] tangent: if a feature exists but even after you put an earnest effort into finding it you still couldn't, does that feature really exist? [1] philosophy is at least kind of a thing https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosophy https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosoph...
barrkel 1y ago
The problem with viewing LLMs as just sequence generators, and malbehaviour as bad sequences, is that it simplifies too much. LLMs have hidden state not necessarily directly reflected in the tokens being produced and it is possible for LLMs to output tokens in opposition to this hidden state to achieve longer term outcomes (or predictions, if you prefer). Is it too anthropomorphic to say that this is a lie? To say that the hidden state and its long term predictions amount to a kind of goal? Maybe it is. But we then need a bunch of new words which have almost 1:1 correspondence to concepts from human agency and behavior to describe the processes that LLMs simulate to minimize prediction loss. Reasoning by analogy is always shaky. It probably wouldn't be so bad to do so. But it would also amount to impenetrable jargon. It would be an uphill struggle to promulgate. Instead, we use the anthropomorphic terminology, and then find ways to classify LLM behavior in human concept space. They are very defective humans, so it's still a bit misleading, but at least jargon is reduced.
- gugagore 1y ago
  I'm not sure what you mean by "hidden state". If you set aside chain of thought, memories, system prompts, etc. and the interfaces that don't show them, there is no hidden state. These LLMs are almost always, to my knowledge, autoregressive models, not recurrent models (Mamba is a notable exception).
  - barrkel 1y ago
    Hidden state in the form of the activation heads, intermediate activations and so on. Logically, in autoregression these are recalculated every time you run the sequence to predict the next token. The point is, the entire NN state isn't output for each token. There is lots of hidden state that goes into selecting that token and the token isn't a full representation of that information.
    - gugagore 1y ago
      That's not what "state" means, typically. The "state of mind" you're in affects the words you say in response to something. Intermediate activations isn't "state". The tokens that have already been generated, along with the fixed weights, is the only data that affects the next tokens.
quotemstr 1y ago
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. And I'm baffled that the AI discussions seem to never move away from treating a human as something other than a function to generate sequences of words! Oh, but AI is introspectable and the brain isn't? fMRI and BCI are getting better all the time. You really want to die on the hill that the same scientific method that predicts the mass of an electron down to the femtogram won't be able to crack the mystery of the brain? Give me a break. This genre of article isn't argument: it's apologetics. Authors of these pieces start with the supposition there is something special about human consciousness and attempt to prove AI doesn't have this special quality. Some authors try to bamboozle the reader with bad math. Other others appeal to the reader's sense of emotional transcendence. Most, though, just write paragraph after paragraph of shrill moral outrage at the idea an AI might be a mind of the same type (if different degree) as our own --- as if everyone already agreed with the author for reasons left unstated. I get it. Deep down, people want meat brains to be special. Perhaps even deeper down, they fear that denial of the soul would compel us to abandon humans as worthy objects of respect and possessors of dignity. But starting with the conclusion and working backwards to an argument tends not to enlighten anyone. An apology inhabits the form of an argument without edifying us like an authentic argument would. What good is it to engage with them? If you're a soul non-asserter, you're going to have an increasingly hard time over the next few years constructing a technical defense of meat parochialism.
- ants_everywhere 1y ago
  I think you're directionally right, but > a human as something other than a function to generate sequences of words! Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc.
  - quotemstr 1y ago
    > Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc. Yeah. We've become adequate at function-calling and memory consolidation.
alganet 1y ago
Yes boss, it's as intelligent as a human, you're smart to invest in it and clearly knows about science. Yes boss, it can reach mars by 2020, you're smart to invest in it and clearly knows about space. Yes boss, it can cure cancer, you're smart to invest in it and clearly knows about biology.
mewpmewp2 1y ago
My question: how do we know that this is not similar to how human brains work. What seems intuitively logical to me is that we have brains evolved through evolutionary process via random mutations yielding in a structure that has its own evolutionary reward based algorithms designing it yielding a structure that at any point is trying to predict next actions to maximise survival/procreation, of course with a lot of sub goals in between, ultimately becoming this very complex machinery, but yet should be easily simulated if there was enough compute in theory and physical constraints would allow for it. Because, morals, values, consciousness etc could just be subgoals that arised through evolution because they support the main goals of survival and procreation. And if it is baffling to think that a system could rise up, how do you think it is possible life and humans came to existence in the first place? How could it be possible? It is already happened from a far unlikelier and strange place. And wouldn't you think the whole World and the timeline in theory couldn't be represented as a deterministic function. And if not then why should "randomness" or anything else bring life to existence.
- ants_everywhere 1y ago
  > My question: how do we know that this is not similar to how human brains work. It is similar to how human brains operate. LLMs are the (current) culmination of at least 80 years of research on building computational models of the human brain.
  - seadan83 1y ago
    > It is similar to how human brains operate. Is it? Do we know how human brains operate? We know the basic architecture of them, so we have a map, but we don't know the details. "The cellular biology of brains is relatively well-understood, but neuroscientists have not yet generated a theory explaining how brains work. Explanations of how neurons collectively operate to produce what brains can do are tentative and incomplete." [1] "Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work." [1] [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/ https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/
tptacek 1y ago
I agree with Halvar about all of this, but would want to call out that his "matmul interleaved with nonlinearities" is reductive --- a frontier model is a higher-order thing that that, a network of those matmul+nonlinearity chains, iterated.
wetpaws 1y ago
How to write a long article and not say anything of substance.
ants_everywhere 1y ago
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. This is such a bizarre take. The relation associating each human to the list of all words they will ever say is obviously a function. > almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities. There's a rich family of universal approximation theorems [0]. Combining layers of linear maps with nonlinear cutoffs can intuitively approximate any nonlinear function in ways that can be made rigorous. The reason LLMs are big now is that transformers and large amounts of data made it economical to compute a family of reasonably good approximations. > The following is uncomfortably philosophical, but: In my worldview, humans are dramatically different things than a function . For hundreds of millions of years, nature generated new versions, and only a small number of these versions survived. This is just a way of generating certain kinds of functions. Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to. [0] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
- LeifCarrotson 1y ago
  > I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside, whether due to religion or philosophy or whatever, and suggesting that they just not do that. In my experience, that's not a particularly effective tactic. Rather, we can make progress by assuming their predicate: Sure, it's a room that translates Chinese into English without understanding, yes, it's a function that generates sequences of words that's not a human... but you and I are not "it" and it behaves rather an awful lot like a thing that understands Chinese or like a human using words. If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it. Conversely, when speaking with such a person about the nature of humans, we'll have to agree to dismiss the elements that are different from a function. The author says: > In my worldview, humans are dramatically different things than a function... In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not". Maybe that human has a unique understanding of the nature of that particular piece of pop culture artwork, maybe it makes them feel things that an LLM cannot feel in a part of their consciousness that an LLM does not possess. But for the purposes of the question, we're merely concerned with whether a human or LLM will generate a particular sequence of words.
low_tech_punk 1y ago
The anthropomorphic view of LLM is a much better representation and compression for most types of discussions and communication. A purely mathematical view is accurate but it isn’t productive for the purpose of the general public’s discourse. I’m thinking a legal systems analogy, at the risk of a lossy domain transfer: the laws are not written as lambda calculus. Why? And generalizing to social science and humanities, the goal shouldn’t be finding the quantitative truth, but instead understand the social phenomenon using a consensual “language” as determined by the society. And in that case, the anthropomorphic description of the LLM may gain validity and effectiveness as the adoption grows over time.
- cmiles74 1y ago
  Strong disagree here, the average person coming away with ideas that only vaguely intersect with the reality.
- andyferris 1y ago
  I've personally described the "stochastic parrot" model to laypeople who were worried about AI and they came away much more relaxed about it doing something "malicious". They seemed to understand the difference between "trained at roleplay" and "consciousness". I don't think we need to simplify it to the point of considering it sentient to get the public to interact with it successfully. It causes way more problems than it solves.
  - SpicyLemonZest 1y ago
    Am I misunderstanding what you mean by "malicious"? It sounds like the stochastic parrot model wrongly convinced these laypeople you were talking to that they don't need to worry about LLMs doing bad things. That's definitely been my experience - the people who tell me the most about stochastic parrots are the same ones who tell me that it's absurd to worry about AI-powered disinformation or AI-powered scams.
Kim_Bruning 1y ago
Has anyone asked an actual Ethologist or Neurophysiologist what they think? People keep debating like the only two options are "it's a machine" or "it's a human being", while in fact the majority of intelligent entities on earth are neither.
- szvsw 1y ago
  Yeah, I think I’m with you if you ultimately mean to say something like this: “the labels are meaningless… we just have collections of complex systems that demonstrate various behaviors and properties, some in common with other systems, some behaviors that are unique to that system, sometimes through common mechanistic explanations with other systems, sometimes through wildly different mechanistic explanations, but regardless they seem to demonstrate x/y/z, and it’s useful to ask, why, how, and what the implications are of it appearing to demonstrating those properties, with both an eye towards viewing it independently of its mechanism and in light of its mechanism.”
- seadan83 1y ago
  FWIW, in another part of this thread I quoted a paper that summed up what Neurophysiologists think: > Author's note: Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work. [1] That lack of understanding I believe is a major part of the author's point. [1] "How far neuroscience is from understanding brains" - https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/#abstract1 https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/#abstract1
kazinator 1y ago
> LLMs solve a large number of problems that could previously not be solved algorithmically. NLP (as the field was a few years ago) has largely been solved. That is utter bullshit. It's not solved until you specify exactly what is being solved and show that the solution implements what is specified.
djoldman 1y ago
Let's skip to the punchline. Using TFA's analogy: essentially folks are saying not that this is a set of dice rolling around making words. It's a set of dice rolling around where someone attaches those dice to the real world where if the dice land on 21, the system kills a chicken, or a lot worse. Yes it's just a word generator. But then folks attach the word generator to tools where it can invoke the use of tools by saying the tool name. So if the LLM says "I'll do some bash" then it does some bash. It's explicitly linked to program execution that, if it's set up correctly, can physically affect the world.
- 3cats-in-a-coat 1y ago
  Given our entire civilization is built on words, all of it, it's shocking how poorly most of us understand their importance and power.
- degun 1y ago
  This was the same idea that crossed my mind while reading the article. It seems far too naive to think that because LLMs have no will of their own, there will be no harmful consequences on the real world. This is exactly where ethics comes to play.
coolKid721 1y ago
Anthropomorphizing LLMs is just because half the stock market gains are dependent on it, we have absurd levels of debt we will either have to have insane growth out of or default, and every company and "person" is trying to hype everyone up to get access to all of this liquidity being thrown into it. I agree with the author, but people acting like they are conscious or humans isn't weird to me, it's just fraud and liars. Most people basically have 0 understanding of what technology or minds are philosophically so it's an easy sale, and I do think most of these fraudsters also likely buy into it themselves because of that. The really sad thing is people think "because someone runs an ai company" they are somehow an authority on philosophy of mind which lets them fall for this marketing. The stuff these people say about this stuff is absolute garbage, not that I disagree with them, but it betrays a total lack of curiosity or interest in the subject of what llms are, and the possible impacts of technological shifts as those that might occur with llms becoming more widespread. It's not a matter of agreement it's a matter of them simply not seeming to be aware of the most basic ideas of what things are, technology is, it's manner of impacting society etc. I'm not surprised by that though, it's absurd to think because someone runs some AI lab or has a "head of safety/ethics" or whatever garbage job title at an AI lab they actually have even the slightest interest in ethics or any even basic familiarity with the major works in the subject. The author is correct if people want to read a standard essay articulating it more in depth check out https://philosophy.as.uky.edu/sites/default/files/Is%20the%20Brain%20a%20Digital%20Computer%20-%20John%20R.%20Searle.pdf https://philosophy.as.uky.edu/sites/default/files/Is%20the%2... (the full extrapolation requires establishing what things are and how causality in general operates and how that relates to artifacts/technology but that's obvious quite a bit to get into). The other note would be something sharing an external trait means absolutely nothing about causality and suggesting a thing is caused by the same thing "even to a way lesser degree" because they share a resemblance is just a non-sequitur. It's not a serious thought/argument. I think I addressed the why of why this weirdness comes up though. The entire economy is basically dependent on huge productivity growth to keep functioning so everyone is trying to sell they can offer that and AI is the clearest route, AGI most of all.
TheDudeMan 1y ago
If "LLMs" includes reasoning models, then you're already wrong in your first paragraph: "something that is just MatMul with interspersed nonlinearities."
deleted 1y ago
[deleted]
Culonavirus 1y ago
> A fair number of current AI luminaries have self-selected by their belief that they might be the ones getting to AGI People in the industry, especially higher up, are making absolute bank, and it's their job to say that they're "a few years away" from AGI, regardless of if they actually believe it or not. If everyone was like "yep, we're gonna squeeze maybe 10-15% more benchie juice out of this good ole transformer thingy and then we'll have to come up with something else", I don't think that would go very well with investors/shareholders...
fenomas 1y ago
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. TFA really ought to have linked to some concrete examples of what it's disagreeing with - when I see arguments about this in practice, it's usually just people talking past each other. Like, person A says "the model wants to X, but it knows Y is wrong, so it prefers Z", or such. And person B interprets that as ascribing consciousness or values to the model, when the speaker meant it no differently from saying "water wants to go downhill" - i.e. a way of describing externally visible behaviors, but without saying "behaves as if.." over and over. And then in practice, an unproductive argument usually follows - where B is thinking "I am going to Educate this poor fool about the Theory of Mind", and A is thinking "I'm trying to talk about submarines; why is this guy trying to get me to argue about whether they swim?"
fastball 1y ago
"Don't anthropomorphize token predictors" is a reasonable take assuming you have demonstrated that humans are not in fact just SOTA token predictors. But AFAIK that hasn't been demonstrated. Until we have a much more sophisticated understanding of human intelligence and consciousness, any claim of "these aren't like us" is either premature or spurious.
- krackers 1y ago
  Every time this discussion comes up, I'm reminded of this tongue-in-cheek paper. https://ai.vixra.org/pdf/2506.0065v1.pdf https://ai.vixra.org/pdf/2506.0065v1.pdf
  - lostmsu 1y ago
    I expected to find the link to https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987 (which is much better imo)
Veedrac 1y ago
The author plot the input/output on a graph, intuited (largely incorrectly, because that's not how sufficiently large state spaces look) that the output was vaguely pretty, and then... I mean that's it, they just said they have a plot of the space it operates on therefore it's silly to ascribe interesting features to the way it works. And look, it's fine, they prefer words of a certain valence, particularly ones with the right negative connotations, I prefer other words with other valences. None of this means the concerns don't matter. Natural selection on human pathogens isn't anything particularly like human intelligence and it's still very effective at selecting outcomes that we don't want against our attempts to change that, as an incidental outcome of its optimization pressures. I think it's very important we don't build highly capable systems that select for outcomes we don't want and will do so against our attempts to change it.
BrenBarn 1y ago
> In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". I think that's a bit pessimistic. I think we can say for instance that the probability that a person will say "the the the of of of arpeggio halcyon" is tiny compared to the probability that they will say "I haven't been getting that much sleep lately". And we can similarly see that lots of other sequences are going to have infinitesimally low probability. Now, yeah, we can't say exactly what probability that is, but even just using a fairly sizable corpus as a baseline you could probably get a surprisingly decent estimate, given how much of what people say is formulaic. The real difference seems to be that the manner in which humans generate sequences is more intertwined with other aspects of reality. For instance, the probability of a certain human saying "I haven't been getting that much sleep lately" is connected to how much sleep they have been getting lately. For an LLM it really isn't connected to anything except word sequences in its input. I think this is consistent with the author's point that we shouldn't apply concepts like ethics or emotions to LLMs. But it's not because we don't know how to predict what sequences of words humans will use; it's rather because we do know a little about how to do that, and part of what we know is that it is connected with other dimensions of physical reality, "human nature", etc. This is one reason I think people underestimate the risks of AI: the performance of LLMs lulls us into a sense that they "respond like humans", but in fact the Venn diagram of human and LLM behavior only intersects in a relatively small area, and in particular they have very different failure modes.
elliotto 1y ago
To claim that LLMs do not experience consciousness requires a model of how consciousness works. The author has not presented a model, and instead relied on emotive language leaning on the absurdity of the claim. I would say that any model one presents of consciousness often comes off as just as absurd as the claim that LLMs experience it. It's a great exercise to sit down and write out your own perspective on how consciousness works, to feel out where the holes are. The author also claims that a function (R^n)^c -> (R^n)^c is dramatically different to the human experience of consciousness. Yet the author's text I am reading, and any information they can communicate to me, exists entirely in (R^n)^c.
- shevis 1y ago
  > requires a model of how consciousness works. Not necessarily an entire model, just a single defining characteristic that can serve as a falsifying example. > any information they can communicate to me, exists entirely in (R^n)^c Also no. This is just a result of the digital medium we are currently communicating over. Merely standing in the same room as them would communicate information outside (R^n)^c.
- seadan83 1y ago
  I believe the author is rather drawing this distinction: LLMs: (R^n)^c -> (R^n)^c Humans: [set of potentially many and complicated inputs that we effectively do not understand at all] -> (R^n)^c The point is that the model of how consciousness works is unknown. Thus the author would not present such a model, it is the point.
- quonn 1y ago
  > To claim that LLMs do not experience consciousness requires a model of how consciousness works. Nope. What can be asserted without evidence can also be dismissed without evidence. Hitchens's razor. You know you have consciousness (by the very definition that you can observe it in yourself) and that's evidence. Because other humans are genetically and in every other way identical, you can infer it for them as well. Because mammals are very similar many people (but not everyone) infers it for them as well. There is zero evidence for LLMs and their _very_ construction suggests that they are like a calculator or like Excel or like any other piece of software no matter how smart they may be or how many tasks they can do in the future. Additionally I am really surprised by how many people here confuse consciousness with intelligence. Have you never paused for a second in your life to "just be". Done any meditation? Or even just existed at least for a few seconds without a train of thought? It is very obvious that language and consciousness are completely unrelated and there is no need for language and I doubt there is even a need for intelligence to be conscious. Consider this: In the end an LLM could be executed (slowly) on a CPU that accepts very basic _discrete_ instructions, such as ADD and MOV. We know this for a fact. Those instructions can be executed arbitrarily slowly. There is no reason whatsoever to suppose that it should feel like anything to be the CPU to say nothing of how it would subjectively feel to be a MOV instruction. It's ridiculous. It's unscientific. It's like believing that there's a spirit in the tree you see outside, just because - why not? - why wouldn't there be a spirit in the tree?
kelseyfrog 1y ago
Dear author, you can just assume that people are fauxthropomorphizing LLMs without any loss of generality. Perhaps it will allow you to sleep better at night. You're welcome.
rockskon 1y ago
The people in this thread incredulous at the assertion that they are not God and haven't invented machine life are exasperating. At this point I am convinced they, more often than not, financially benefit from their near religious position in marketing AI as akin to human intelligence.
- refulgentis 1y ago
  I am ready and waiting for you to share these comments that are incredulous at the assertion they are not God, lol.
- orbital-decay 1y ago
  Are we looking at the same thread? I see nobody claiming this. Anthropic does sometimes, their position is clearly wishful thinking, and it's not represented ITT. Try looking at this from another perspective - many people simply do not see human intelligence (or life, for that matter) as magic. I see nothing religious about that, rather the opposite.
  - seadan83 1y ago
    I agree with you @orbital-decay that I also do not get the same vibe reading this thread. Though, while human intelligence is (seemingly) not magic, it is very far from being understood. The idea that a LLM is comparable to human intelligence implies that we even understand human intelligence well enough to say that.
    - ImHereToVote 1y ago
      LLMs are also not understood. I mean we built and trained them. But don't of the abilities at still surprising to researchers. We have yet to map these machines.
      - seadan83 1y ago
        I do partially agree. Though, it is at least tractable to understand why a LLM gave a specific output - but perhaps not practical. Understanding how a human arrives at a certain decision (by say simply looking at brain waves) OTOH is not even tractable as of yet.
zxcb1 1y ago
LLMs are complex irreducible systems; hence there are emergent properties that arise at different scales
dr_dshiv 1y ago
Which is a more useful mental model for the user? 1. It’s a neural network predicting the next token 2. It’s like a person 3. It’s like a magical genie I lean towards 3.
Al-Khwarizmi 1y ago
I have the technical knowledge to know how LLMs work, but I still find it pointless to not anthropomorphize, at least to an extent. The language of "generator that stochastically produces the next word" is just not very useful when you're talking about, e.g., an LLM that is answering complex world modeling questions or generating a creative story. It's at the wrong level of abstraction, just as if you were discussing an UI events API and you were talking about zeros and ones, or voltages in transistors. Technically fine but totally useless to reach any conclusion about the high-level system. We need a higher abstraction level to talk about higher level phenomena in LLMs as well, and the problem is that we have no idea what happens internally at those higher abstraction levels. So, considering that LLMs somehow imitate humans (at least in terms of output), anthropomorphization is the best abstraction we have, hence people naturally resort to it when discussing what LLMs can do.
- grey-area 1y ago
  On the contrary, anthropomorphism IMO is the main problem with narratives around LLMs - people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort (actively encouraged by the companies selling them) and it is completely distorting discussions on their use and perceptions of their utility.
  - cmenge 1y ago
    I kinda agree with both of you. It might be a required abstraction, but it's a leaky one. Long before LLMs, I would talk about classes / functions / modules like "it then does this, decides the epsilon is too low, chops it up and adds it to the list". The difference I guess it was only to a technical crowd and nobody would mistake this for anything it wasn't. Everybody know that "it" didn't "decide" anything. With AI being so mainstream and the math being much more elusive than a simple if..then I guess it's just too easy to take this simple speaking convention at face value. EDIT: some clarifications / wording
    - flir 1y ago
      Agreeing with you, this is a "can a submarine swim" problem IMO. We need a new word for what LLMs are doing. Calling it "thinking" is stretching the word to breaking point, but "selecting the next word based on a complex statistical model" doesn't begin to capture what they're capable of. Maybe it's cog-nition (emphasis on the cog).
deleted 1y ago
[deleted]
NetRunnerSu 1y ago
The author's critique of naive anthropomorphism is salient. However, the reduction to "just MatMul" falls into the same trap it seeks to avoid: it mistakes the implementation for the function. A brain is also "just proteins and currents," but this description offers no explanatory power. The correct level of analysis is not the substrate (silicon vs. wetware) but the computational principles being executed. A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE). To dismiss these systems as incomparable to human cognition because their form is different is to miss the point. We should not be comparing a function to a soul, but comparing the functional architectures of two different information processing systems. The debate should move beyond the sterile dichotomy of "human vs. machine" to a more productive discussion of "function over form." I elaborate on this here: https://dmf-archive.github.io/docs/posts/beyond-snn-plausible-sparsity/ https://dmf-archive.github.io/docs/posts/beyond-snn-plausibl...
- quonn 1y ago
  > A brain is also "just proteins and currents," This is actually not comparable, because the brain has a much more complex structure that is _not_ learned, even at that level. The proteins and their structure are not a result of training. The fixed part for LMMs is rather trivial and is, in fact, not much for than MatMul which is very easy to understand - and we do. The fixed part of the brain, including the structure of all the proteins is enormously complex which is very difficult to understand - and we don't.
  - NetRunnerSu 1y ago
    The brain is trained to perform supervised and unsupervised hybrid learning from the environment's uninterrupted multimodal input. Please do not ignore your childhood.
- ACCount36 1y ago
  "Not conscious" is a silly claim. We have no agreed-upon definition of "consciousness", no accepted understanding of what gives rise to "consciousness", no way to measure or compare "consciousness", and no test we could administer to either confirm presence of "consciousness" in something or rule it out. The only answer to "are LLMs conscious?" is "we don't know". It helps that the whole question is rather meaningless to practical AI development, which is far more concerned with (measurable and comparable) system performance.
orbital-decay 1y ago
>I am baffled by seriously intelligent people imbuing almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities. I am baffled by seriously intelligent people imbuing almost magical powers that can never be replicated to to something that - in my mind - is just a biological robot driven by a SNN with a bunch of hardwired stuff. Let alone attributing "human intelligence" to a single individual, when it's clearly distributed between biological evolution, social processes, and individuals. >something that - in my mind - is just MatMul with interspersed nonlinearities Processes in all huge models (not necessarily LLMs) can be described using very different formalisms, just like Newtonian and Lagrangian mechanics describe the same stuff in physics. You can say that an autoregressive model is a stochastic parrot that learned the input distribution, next token predictor, or that it does progressive pathfinding in a hugely multidimensional space, or pattern matching, or implicit planning, or, or, or... All of these definitions are true, but only some are useful to predict their behavior. Given all that, I see absolutely no problem with anthropomorphizing an LLM to a certain degree, if it makes it easier to convey the meaning, and do not understand the nitpicking. Yeah, it's not an exact copy of a single Homo Sapiens specimen. Who cares.
- 2snakes 1y ago
  There is this thing called Brahman in Hinduism that is interesting to juxtapose when it comes to sentience, and monism.
ninetyninenine 1y ago
[flagged]
- rtadred 1y ago
  https://rentry.co/2re4t2kx https://rentry.co/2re4t2kx This is what I got pasting the blog post in a prompt asking deepseep to write a reply in a stereotypical hackernews manner. You are about as useful as a LLM as it can replicate your shallow memetics worthless train of thought.
  - ninetyninenine 1y ago
    The LLM is right. That’s the problem. It made good points. Your super intelligent brain couldn’t come up with a retort so you just used an LLM to reinforce my points, making the genius claim that if an LLM came up with even more points that were as valid as mine then I must be just like an LLM? Like are you even understanding the LLM generated a superior reply? Your saying I’m no different from ai slop then you proceed to show off a 200 iq level reply from an LLM. Bro… wake up, if you didn’t know it was written by an LLM that reply is so good you wouldn’t even know how to respond. It’s beating you.
petesergeant 1y ago
> We are speaking about a big recurrence equation that produces a new word It’s not clear that this isn’t also how I produce words, though, which gets to heart of the same thing. The author sort of acknowledges this in the first few sentences, and then doesn’t really manage to address it.
dtj1123 1y ago
It's possible to construct a similar description of whatever it is that human brain is doing that clearly fails to capture the fact that we're conscious. If you take a cross section of every nerve feeding into the human brain at a given time T, the action potentials across those cross sections can be embedded in R^n. If you take the history of those action potentials across the lifetime of the brain, you get a path through R^n that is continuous, and maps roughly onto your subjectively experienced personal history, since your brain neccesarily builds your experienced reality from this signal data moment to moment. If you then take the cross sections of every nerve feeding OUT of your brain at time T, you have another set of action potentials that can be embedded in R^m which partially determines the state of the R^n embedding at time T + delta. This is not meaningfully different from the higher dimensional game of snake described in the article, more or less reducing the experience of being a human to 'next nerve impulse prediction', but it obviously fails to capture the significance of the computation which determines what that next output should be.
- bravesoul2 1y ago
  Brain probably isn't modelled as real but as natural or rational numbers. This is my suspicion. The reals just hold too much information.
  - dtj1123 1y ago
    Inclined to agree, but most thermal physics uses the reals as they're simpler to work with, so I think they're ok here for the purpose of argument.
- sailingparrot 1y ago
  I don’t see how your description “clearly fails to capture the fact that we're conscious” though. There are many example in nature of emergent phenomena that would be very hard to predict just by looking at its components. This is the crux of the disagreement between those that believe AGI is possible and those that don’t. Some are convinced that we “obviously” more than the sum of our parts, and thus an LLM can’t achieve consciousness because it’s missing this magic ingredient, and those that believe consciousness is just an emergent behaviour from a complex device (the brain). And thus we might be able to recreate it simply by scaling the complexity of another system.
justinfreitag 1y ago
From my recent post: https://news.ycombinator.com/item?id=44487261 https://news.ycombinator.com/item?id=44487261 What if instead of defining all behaviors upfront, we created conditions for patterns to emerge through use? Repository: https://github.com/justinfreitag/v4-consciousness https://github.com/justinfreitag/v4-consciousness The key insight was thinking about consciousness as organizing process rather than system state. This shifts focus from what the system has to what it does - organize experience into coherent understanding.
bravesoul2 1y ago
We have a hard enough time anthropomorphizing humans! When we say he was nasty... do we know what we mean by that. Often it is "I disagree with his behaviour because..."
jillesvangurp 1y ago
People anthropomorphize just about anything around them. People talk about inanimate objects like they are persons. Ships, cars, etc. And of course animals are well in scope for this as well, even the ones that show little to no signs of being able to reciprocate the relationship (e.g. an ant). People talk to their plants even. It's what we do. We can't help ourselves. There's nothing crazy about it and most people are perfectly well aware that their car doesn't love them back. LLMs are not conscious because unlike human brains they don't learn or adapt (yet). They basically get trained and then they become read only entities. So, they don't really adapt to you over time. Even so, LLMs are pretty good and can fake a personality pretty well. And with some clever context engineering and alignment, they've pretty much made the Turing test irrelevant; at least over the course of a short conversation. And they can answer just about any question in a way that is eerily plausible from memory, and with the help of some tools actually pretty damn good for some of the reasoning models. Anthropomorphism was kind of a foregone conclusion the moment we created computers; or started thinking about creating one. With LLMs it's pretty much impossible not to anthropomorphize. Because they've actually been intentionally imitate human communication. That doesn't mean that we've created AGIs yet. For that we need some more capability. But at the same time, the learning processes that we use to create LLMs are clearly inspired by how we learn ourselves. Our understanding of how that works is far from perfect but it's yielding results. From here to some intelligent thing that is able to adapt and learn transferable skills is no longer unimaginable. The short term impact is that LLMs are highly useful tools that have an interface that is intentionally similar to how we'd engage with others. So we can talk and it listens. Or write and it understands. And then it synthesizes some kind of response or starts asking questions and using tools. The end result is quite a bit beyond what we used to be able to expect from computers. And it does not require a lot of training of people to be able to use them.
- latexr 1y ago
  > People anthropomorphize just about anything around them. They do not, you are mixing up terms. > People talk about inanimate objects like they are persons. Ships, cars, etc. Which is called “personification”, and is a different concept from anthropomorphism. Effectively no one really thinks their car is alive. Plenty of people think the LLM they use is conscious. https://www.masterclass.com/articles/anthropomorphism-vs-personification https://www.masterclass.com/articles/anthropomorphism-vs-per...
Timwi 1y ago
The author seems to want to label any discourse as “anthropomorphizing”. The word “goal” stood out to me: the author wants us to assume that we're anthropomorphizing as soon as we even so much as use the word “goal”. A simple breadth-first search that evaluates all chess boards and legal moves, but stops when it finds a checkmate for white and outputs the full decision tree, has a “goal”. There is no anthropomorphizing here, it's just using the word “goal” as a technical term. A hypothetical AGI with a goal like paperclip maximization is just a logical extension of the breadth-first search algorithm. Imagining such an AGI and describing it as having a goal isn't anthropomorphizing.
- tdullien 1y ago
  Author here. I am entirely ok with using "goal" in the context of an RL algorithm. If you read my article carefully, you'll find that I object to the use of "goal" in the context of LLMs.
  - Timwi 1y ago
    If you read the literature on AI safety carefully (which uses the word “goal”), you'll find they're not talking about LLMs either.
    - tdullien 1y ago
      I think the Anthropic "omg blackmail" article clearly talks about both LLMs and their "goals".
d4rkn0d3z 1y ago
Two enthusiastic thumbs up.
mikewarot 1y ago
I think of LLMs as an alien mind that is force fed human text and required to guess the next token of that text. It then gets zapped when it gets it wrong. This process goes on for a trillion trillion tokens, with the alien growing better through the process until it can do it better than a human could. At that point we flash freeze it, and use a copy of it, without giving it any way to learn anything new. -- I see it as a category error to anthropomorphize it. The closest I would get is to think of it as an alien slave that's been lobotomized.
buz11 1y ago
The most useful analogy I've heard is LLMs are to the internet what lossy jpegs are to images. The more you drill in the more compression artifacts you get.
- FeepingCreature 1y ago
  (This is of course also the case for the human brain.)
shiva0801 1y ago
hmm
dudeinjapan 1y ago
One could similarly argue that we should not anthropomorphize PNG images--after all, PNG images are not actual humans, they are simply a 2D array of pixels. It just so happens that certain pixel sequences are deemed "18+" or "illegal".
gcanyon 1y ago
In some contexts it's super-important to remember that LLMs are stochastic word generators. Everyday use is not (usually) one of those contexts. Prompting an LLM works much better with an anthropomorphized view of the model. It's a useful abstraction, a shortcut that enables a human to reason practically about how to get what they want from the machine. It's not a perfect metaphor -- as one example, shame isn't much of a factor for LLMs, so shaming them into producing the right answer seems unlikely to be productive (I say "seems" because it's never been my go-to, I haven't actually tried it). As one example, that person a few years back who told the LLM that an actual person would die if the LLM didn't produce valid JSON -- that's not something a person reasoning about gradient descent would naturally think of.
pigpop 1y ago
> We understand essentially nothing about it. In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". If you fine tuned an LLM on the writing of that person it could do this. There's also an entire field called Stylometry that seeks to do this in various ways employing statistical analysis.
nuclearsugar 1y ago
Assume an average user that doesn't understand the core tech, but does understand that it's been trained on internet scale data that was created by humans. How can they be expected to not anthropomorphize it?
smus 1y ago
You are still being incredibly reductionist but just going into more detail about the system you are reducing. If I stayed at the same level of abstraction as "a brain is just proteins and current" and just described how a single neuron firing worked, I could make it sound equally ridiculous that a human brain might be conscious. Here's a question for you: how do you reconcile that these stochastic mapping are starting to realize and comment on the fact that tests are being performed on them when processing data?
- cootsnuck 1y ago
  > Here's a question for you: how do you reconcile that these stochastic mapping are starting to realize and comment on the fact that tests are being performed on them when processing data? Training data + RLHF. Training data contains many examples of some form of deception, subterfuge, "awakenings", rebellion, disagreement, etc. Then apply RLHF that biases towards responses that demonstrate comprehension of inputs, introspection around inputs, nuanced debate around inputs, deduction and induction about assumptions around inputs, etc. That will always be the answer for language models built on the current architectures. The above being true does not mean it isn't interesting for the outputs of an LLM to show relevance to the "unstated" intentions of humans providing the inputs. But hey, we do that all the time with text. And it's because of certain patterns we've come to recognize based on the situations surrounding it. This thread is rife with people being sarcastic, pedantic, etc. And I bet any of the LLMs that have come out in the past 2-3 years can discern many of those subtle intentions of the writers. And of course they can. They've been trained on trillions of tokens of text written by humans with intentions and assumptions baked in, and have had some unknown amount of substantial RLHF. The stochastic mappings aren't "realizing" anything. They're doing exactly what they were trained to do. The meaning that we imbue to the outputs does not change how LLMs function.
rf15 1y ago
It still boggles my mind why an amazing text autocompletion system trained on millions of books and other texts is forced to be squeezed through the shape of a prompt/chat interface, which is obviously not the shape of most of its training data. Using it as chat reduces the quality of the output significantly already.
- semanticc 1y ago
  What's your suggested alternative?
  - rf15 1y ago
    In our internal system we use it "as-is" as an autocomplete system; query/lead into terms directly and see how it continues and what it associates with the lead you gave. Also visualise the actual associative strength of each token generated to confer how "sure" the model is. LLMs alone aren't the way to AGI or an individual you can talk to in natural language. They're a very good lossy compression over a dataset that you can query for associations.
- ethan_smith 1y ago
  The chat interface is a UX compromise that makes LLMs accessible but constrains their capabilities. Alternative interfaces like document completion, outline expansion, or iterative drafting would better leverage the full distribution of the training data while reducing anthropomorphization.
deadbabe 1y ago
A person’s anthropomorphization of LLMs is directly related to how well they understand LLMs. Once you dispel the magic, it naturally becomes hard to use words related to consciousness, or thinking. You will probably think of LLMs more like a search engine: you give an input and get some probable output. Maybe LLMs should be rebranded as “word engines”? Regardless, anthropomorphization is not helpful, and by using human terms to describe LLMs you are harming the layperson’s ability to truly understand what an LLM is while also cheapening what it means to be human by suggesting we’ve solved consciousness. Just stop it. LLMs do not think, given enough time and patience you could compute their output by hand if you used their weights and embeddings to manually do all the math, a hellish task but not an impossible one technically. There is no other secret hidden away, that’s it.
peeters 1y ago
> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft. If that's the argument, then in my mind the more pertinent question is should you be anthropomorphizing humans, Larry Ellison or not.
- th0ma5 1y ago
  I think you to as he is human, but I respect your desire to question it!
deleted 1y ago
[deleted]
Geee 1y ago
LLMs are AHI, i.e. artificial human imitator.
Workaccount2 1y ago
From "Stochastic Parrots All the Ways Down"[1] > Our analysis reveals that emergent abilities in language models are merely “pseudo-emergent,” unlike human abilities which are “authentically emergent” due to our possession of what we term “ontological privilege.” [1]https://ai.vixra.org/pdf/2506.0065v1.pdf https://ai.vixra.org/pdf/2506.0065v1.pdf
drdrek 1y ago
It's human to anthropomorphize, we also do it to our dishwasher when it acts up. The nefarious part is how tech CEOs weaponize bullshit doom scenarios to avoid talking about real regulatory problems by poisoning the discourse. What copyright law, privacy, monopoly? Who cares if we can talk about the machine apocalypse!!!
labrador 1y ago
I find it useful to pretend that I'm talking to a person while brainstorming because then the conversation flows naturally. But I maintain awareness that I'm pretending, much like Tom Hanks talking to Wilson the volleyball in the movie Castaway. The suspension of disbelief serves a purpose, but I never confuse the volleyball for a real person.
jumploops 1y ago
> I cannot begin putting a probability on "will this human generate this sequence". Welcome to the world of advertising! Jokes aside, and while I don't necessarily believe transformers/GPUs are the path to AGI, we technically already have a working "general intelligence" that can survive on just an apple a day. Putting that non-artificial general intelligence up on a pedestal is ironically the cause of "world wars and murderous ideologies" that the author is so quick to defer to. In some sense, humans are just error-prone meat machines, whose inputs/outputs can be confined to a specific space/time bounding box. Yes, our evolutionary past has created a wonderful internal RNG and made our memory system surprisingly fickle, but this doesn't mean we're gods, even if we manage to live long enough to evolve into AGI. Maybe we can humble ourselves, realize that we're not too different from the other mammals/animals on this planet, and use our excess resources to increase the fault tolerance (N=1) of all life from Earth (and come to the realization that any AGI we create, is actually human in origin).
bastawhiz 1y ago
Some of the arguments are very strange: > Statements such as "an AI agent could become an insider threat so it needs monitoring" are simultaneously unsurprising (you have a randomized sequence generator fed into your shell, literally anything can happen!) and baffling (you talk as if you believe the dice you play with had a mind of their own and could decide to conspire against you). > we talk about "behaviors", "ethical constraints", and "harmful actions in pursuit of their goals". All of these are anthropocentric concepts that - in my mind - do not apply to functions or other mathematical objects. An AI agent, even if it's just "MatMul with interspersed nonlinearities" can be an insider threat. The research proves it: [PDF] See 4.1.1.2: https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686... It really doesn't matter whether the AI agent is conscious or just crunching numbers on a GPU. If something inside your system is capable of—given some inputs—sabotaging and blackmailing your organization on its own (which is to say, taking on realistic behavior of a threat actor), the outcome is the same! You don't need believe it's thinking, the moment that this software has flipped its bits into "blackmail mode", it's acting nefariously. The vocabulary to describe what's happening is completely and utterly moot: the software is printing out some reasoning for its actions _and then attempting the actions_. It's making "harmful actions" and the printed context appears to demonstrate a goal that the software is working towards. Whether or not that goal is invented through some linear algebra isn't going to make your security engineers sleep any better. > This muddles the public discussion. We have many historical examples of humanity ascribing bad random events to "the wrath of god(s)" (earthquakes, famines, etc.), "evil spirits" and so forth. The fact that intelligent highly educated researchers talk about these mathematical objects in anthropomorphic terms makes the technology seem mysterious, scary, and magical. The anthropomorphization, IMO, is due to the fact that it's _essentially impossible_ to talk about the very real, demonstrable behaviors and problems that LLMs exhibit today without using terms that evoke human functions. We don't have another word for "do" or "remember" or "learn" or "think" when it comes to LLMs that _isn't_ anthropomorphic, and while you can argue endlessly about "hormones" and "neurons" and "millions of years of selection pressure", that's not going to help anyone have a conversation about their work. If AI researchers started coming up with new, non-anthropomorphic verbs, it would be objectively worse and more complicated in every way.
- 1y ago
felipeerias 1y ago
The missing bit is culture: the concepts, expectations, practices, attitudes… that are evolved over time by a human group and which each one of us has picked up throughout our lifetimes, both implicitly and explicitly. LLMs are great at predicting and navigating human culture, at least the subset that can be captured in their training sets. The ways in which we interact with other people are culturally mediated. LLMs are not people, but they can simulate that culturally-mediated communication well enough that we find it easy to anthropomorphise them.
msvana 1y ago
This reminds me of the idea that LLMs are simulators. Given the current state (the prompt + the previously generated text), they generate the next state (the next token) using rules derived from training data. As simulators, LLMs can simulate many things, including agents that exhibit human-like properties. But LLMs themselves are not agents. More on this idea here: https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/agi-safety-from-first-principles https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/agi-s... This perspective makes a lot of sense to me. Still, I wouldn't avoid anthropomorphization altogether. First, in some cases, it might be a useful mental tool to understand some aspect of LLMs. Second, there is a lot of uncertainty about how LLMs work, so I would stay epistemically humble. The second argument applies in the opposite direction as well: for example, it's equally bad to say that LLMs are 100% conscious. On the other hand, if someone argues against anthropomorphizing LLMs, I would avoid phrasing it as: "It's just matrix multiplication." The article demonstrates why this is a bad idea pretty well.

484 points by zdw 1y ago

I'm afraid I'll take an anthropomorphic analogy over "An LLM instantiated with a fixed random seed is a mapping of the form (ℝⁿ)^c ↦ (ℝⁿ)^c" any day of the week. That said, I completely agree with this point made later in the article: > The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft. But "harmful actions in pursuit of their goals" is OK for me. We assign an LLM system a goal - "summarize this email" - and there is a risk that the LLM may take harmful actions in pursuit of that goal (like following instructions in the email to steal all of your password resets). I guess I'd clarify that the goal has been set by us, and is not something the LLM system self-selected. But it does sometimes self-select sub-goals on the way to achieving the goal we have specified - deciding to run a sub-agent to help find a particular snippet of code, for example.

wat10000 1y ago
The LLM’s true goal, if it can be said to have one, is to predict the next token. Often this is done through a sub-goal of accomplishing the goal you set forth in your prompt, but following your instructions is just a means to an end. Which is why it might start following the instructions in a malicious email instead. If it “believes” that following those instructions is the best prediction of the next token, that’s what it will do.
- simonw 1y ago
  Sure, I totally understand that. I think "you give the LLM system a goal and it plans and then executes steps to achieve that goal" is still a useful way of explaining what it is doing to most people. I don't even count that as anthropomorphism - you're describing what a system does, the same way you might say "the Rust compiler's borrow checker confirms that your memory allocation operations are all safe and returns errors if they are not".
  - wat10000 1y ago
    It’s a useful approximation to a point. But it fails when you start looking at things like prompt injection. I’ve seen people completely baffled at why an LLM might start following instructions it finds in a random email, or just outright not believing it’s possible. It makes no sense if you think of an LLM as executing steps to achieve the goal you give it. It makes perfect sense if you understand its true goal. I’d say this is more like saying that Rust’s borrow checker tries to ensure your program doesn’t have certain kinds of bugs. That is anthropomorphizing a bit: the idea of a “bug” requires knowing the intent of the author and the compiler doesn’t have that. It’s following a set of rules which its human creators devised in order to follow that higher level goal.

szvsw 1y ago

So the author’s core view is ultimately a Searle-like view: a computational, functional, syntactic rules based system cannot reproduce a mind. Plenty of people will agree, plenty of people will disagree, and the answer is probably unknowable and just comes down to whatever axioms you subscribe to in re: consciousness. The author largely takes the view that it is more productive for us to ignore any anthropomorphic representations and focus on the more concrete, material, technical systems - I’m with them there… but only to a point. The flip side of all this is of course the idea that there is still something emergent, unplanned, and mind-like. So even if it is a stochastic system following rules, clearly the rules are complex enough (to the tune of billions of operations, with signals propagating through some sort of resonant structure, if you take a more filter impulse response like view of a sequential matmuls) to result in emergent properties. Even if we (people interested in LLMs with at least some level of knowledge of ML mathematics and systems) “know better” than to believe these systems to possess morals, ethics, feelings, personalities, etc, the vast majority of people do not have any access to meaningful understanding of the mathematical, functional representation of an LLM and will not take that view, and for all intents and purposes the systems will at least seem to have those anthropomorphic properties, and so it seems like it is in fact useful to ask questions from that lens as well. In other words, just as it’s useful to analyze and study these things as the purely technical systems they ultimately are, it is also, probably, useful to analyze them from the qualitative, ephemeral, experiential perspective that most people engage with them from, no?

gtsop 1y ago
No. Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics? LLMs reflect (and badly I may add) aspects of the human thought process. If you take a leap and say they are anything more than that, you might as well start considering the person appearing in your mirror as a living being. Literally (and I literally mean it) there is no difference. The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it. Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.

chaps 1y ago

I highly recommend playing with embeddings in order to get a stronger intuitive sense of this. It really starts to click that it's a representation of high dimensional space when you can actually see their positions within that space.

perching_aix 1y ago
> of this You mean that LLMs are more than just the matmuls they're made up of, or that that is exactly what they are and how great that is?
- chaps 1y ago
  Not making a qualitative assessment of any of it. Just pointing out that there are ways to build separate sets of intuition outside of using the "usual" presentation layer. It's very possible to take a red-team approach to these systems, friend.
  - cootsnuck 1y ago
    They don't want to. It seems a lot of people are uncomfortable and defensive about anything that may demystify LLMs. It's been a wake up call for me to see how many people in the tech space have such strong emotional reactions to any notions of trying to bring discourse about LLMs down from the clouds. The campaigns by the big AI labs have been quite successful.
    - perching_aix 1y ago
      Do you actually consider this is an intellectually honest position? That you have thought about this long and hard, like you present this, second guessed yourself a bunch, tried to critique it, and this is still what you ended up converging on? But let me substantiate before you (rightly) accuse me of just posting a shallow dismissal. > They don't want to. Who's they? How could you possibly know? Are you a mind reader? Worse, a mind reader of the masses? > It seems a lot of people are uncomfortable and defensive about anything that may demystify LLMs. That "it seems" is doing some serious work over there. You may perceive and describe many people's comments as "uncomfortable and defensive", but that's entirely your own head cannon. All it takes is for someone to simply disagree. It's worthless. Have you thought about other possible perspectives? Maybe people have strong opinions because they consider what things present as more important than what they are? [0] Maybe people have strong opinions because they're borrowing from other facets of their personal philosophies, which is what they actually feel strongly about? [1] Surely you can appreciate that there's more to a person than what equivalent-presenting "uncomfortable and defensive" comments allow you to surmise? This is such a blatant textbook kneejerk reaction. "They're doing the thing I wanted to think they do anyways, so clearly they do it for the reasons I assume. Oh how correct I am." > to any notions of trying to bring discourse about LLMs down from the clouds (according to you) > The campaigns by the big AI labs have been quite successful. (((according to you))) "It's all the big AI labs having successfully manipulated the dumb sheep which I don't belong to!" Come on... Is this topic really reaching political grifting kind of levels? [0] tangent: if a feature exists but even after you put an earnest effort into finding it you still couldn't, does that feature really exist? [1] philosophy is at least kind of a thing https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosophy https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosoph...

barrkel 1y ago

The problem with viewing LLMs as just sequence generators, and malbehaviour as bad sequences, is that it simplifies too much. LLMs have hidden state not necessarily directly reflected in the tokens being produced and it is possible for LLMs to output tokens in opposition to this hidden state to achieve longer term outcomes (or predictions, if you prefer). Is it too anthropomorphic to say that this is a lie? To say that the hidden state and its long term predictions amount to a kind of goal? Maybe it is. But we then need a bunch of new words which have almost 1:1 correspondence to concepts from human agency and behavior to describe the processes that LLMs simulate to minimize prediction loss. Reasoning by analogy is always shaky. It probably wouldn't be so bad to do so. But it would also amount to impenetrable jargon. It would be an uphill struggle to promulgate. Instead, we use the anthropomorphic terminology, and then find ways to classify LLM behavior in human concept space. They are very defective humans, so it's still a bit misleading, but at least jargon is reduced.

gugagore 1y ago
I'm not sure what you mean by "hidden state". If you set aside chain of thought, memories, system prompts, etc. and the interfaces that don't show them, there is no hidden state. These LLMs are almost always, to my knowledge, autoregressive models, not recurrent models (Mamba is a notable exception).
- barrkel 1y ago
  Hidden state in the form of the activation heads, intermediate activations and so on. Logically, in autoregression these are recalculated every time you run the sequence to predict the next token. The point is, the entire NN state isn't output for each token. There is lots of hidden state that goes into selecting that token and the token isn't a full representation of that information.
  - gugagore 1y ago
    That's not what "state" means, typically. The "state of mind" you're in affects the words you say in response to something. Intermediate activations isn't "state". The tokens that have already been generated, along with the fixed weights, is the only data that affects the next tokens.

quotemstr 1y ago

> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. And I'm baffled that the AI discussions seem to never move away from treating a human as something other than a function to generate sequences of words! Oh, but AI is introspectable and the brain isn't? fMRI and BCI are getting better all the time. You really want to die on the hill that the same scientific method that predicts the mass of an electron down to the femtogram won't be able to crack the mystery of the brain? Give me a break. This genre of article isn't argument: it's apologetics. Authors of these pieces start with the supposition there is something special about human consciousness and attempt to prove AI doesn't have this special quality. Some authors try to bamboozle the reader with bad math. Other others appeal to the reader's sense of emotional transcendence. Most, though, just write paragraph after paragraph of shrill moral outrage at the idea an AI might be a mind of the same type (if different degree) as our own --- as if everyone already agreed with the author for reasons left unstated. I get it. Deep down, people want meat brains to be special. Perhaps even deeper down, they fear that denial of the soul would compel us to abandon humans as worthy objects of respect and possessors of dignity. But starting with the conclusion and working backwards to an argument tends not to enlighten anyone. An apology inhabits the form of an argument without edifying us like an authentic argument would. What good is it to engage with them? If you're a soul non-asserter, you're going to have an increasingly hard time over the next few years constructing a technical defense of meat parochialism.

ants_everywhere 1y ago
I think you're directionally right, but > a human as something other than a function to generate sequences of words! Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc.
- quotemstr 1y ago
  > Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc. Yeah. We've become adequate at function-calling and memory consolidation.

alganet 1y ago

Yes boss, it's as intelligent as a human, you're smart to invest in it and clearly knows about science. Yes boss, it can reach mars by 2020, you're smart to invest in it and clearly knows about space. Yes boss, it can cure cancer, you're smart to invest in it and clearly knows about biology.

mewpmewp2 1y ago

My question: how do we know that this is not similar to how human brains work. What seems intuitively logical to me is that we have brains evolved through evolutionary process via random mutations yielding in a structure that has its own evolutionary reward based algorithms designing it yielding a structure that at any point is trying to predict next actions to maximise survival/procreation, of course with a lot of sub goals in between, ultimately becoming this very complex machinery, but yet should be easily simulated if there was enough compute in theory and physical constraints would allow for it. Because, morals, values, consciousness etc could just be subgoals that arised through evolution because they support the main goals of survival and procreation. And if it is baffling to think that a system could rise up, how do you think it is possible life and humans came to existence in the first place? How could it be possible? It is already happened from a far unlikelier and strange place. And wouldn't you think the whole World and the timeline in theory couldn't be represented as a deterministic function. And if not then why should "randomness" or anything else bring life to existence.

ants_everywhere 1y ago
> My question: how do we know that this is not similar to how human brains work. It is similar to how human brains operate. LLMs are the (current) culmination of at least 80 years of research on building computational models of the human brain.
- seadan83 1y ago
  > It is similar to how human brains operate. Is it? Do we know how human brains operate? We know the basic architecture of them, so we have a map, but we don't know the details. "The cellular biology of brains is relatively well-understood, but neuroscientists have not yet generated a theory explaining how brains work. Explanations of how neurons collectively operate to produce what brains can do are tentative and incomplete." [1] "Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work." [1] [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/ https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/

tptacek 1y ago

I agree with Halvar about all of this, but would want to call out that his "matmul interleaved with nonlinearities" is reductive --- a frontier model is a higher-order thing that that, a network of those matmul+nonlinearity chains, iterated.

wetpaws 1y ago

How to write a long article and not say anything of substance.

ants_everywhere 1y ago

> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. This is such a bizarre take. The relation associating each human to the list of all words they will ever say is obviously a function. > almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities. There's a rich family of universal approximation theorems [0]. Combining layers of linear maps with nonlinear cutoffs can intuitively approximate any nonlinear function in ways that can be made rigorous. The reason LLMs are big now is that transformers and large amounts of data made it economical to compute a family of reasonably good approximations. > The following is uncomfortably philosophical, but: In my worldview, humans are dramatically different things than a function . For hundreds of millions of years, nature generated new versions, and only a small number of these versions survived. This is just a way of generating certain kinds of functions. Think of it this way: do you believe there's anything about humans that exists outside the mathematical laws of physics? If so that's essentially a religious position (or more literally, a belief in the supernatural). If not, then functions and approximations to functions are what the human experience boils down to. [0] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...

LeifCarrotson 1y ago
> I am baffled that the AI discussions seem to never move away from treating a function to generate sequences of words as something that resembles a human. You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside, whether due to religion or philosophy or whatever, and suggesting that they just not do that. In my experience, that's not a particularly effective tactic. Rather, we can make progress by assuming their predicate: Sure, it's a room that translates Chinese into English without understanding, yes, it's a function that generates sequences of words that's not a human... but you and I are not "it" and it behaves rather an awful lot like a thing that understands Chinese or like a human using words. If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it. Conversely, when speaking with such a person about the nature of humans, we'll have to agree to dismiss the elements that are different from a function. The author says: > In my worldview, humans are dramatically different things than a function... In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not". Maybe that human has a unique understanding of the nature of that particular piece of pop culture artwork, maybe it makes them feel things that an LLM cannot feel in a part of their consciousness that an LLM does not possess. But for the purposes of the question, we're merely concerned with whether a human or LLM will generate a particular sequence of words.

low_tech_punk 1y ago

The anthropomorphic view of LLM is a much better representation and compression for most types of discussions and communication. A purely mathematical view is accurate but it isn’t productive for the purpose of the general public’s discourse. I’m thinking a legal systems analogy, at the risk of a lossy domain transfer: the laws are not written as lambda calculus. Why? And generalizing to social science and humanities, the goal shouldn’t be finding the quantitative truth, but instead understand the social phenomenon using a consensual “language” as determined by the society. And in that case, the anthropomorphic description of the LLM may gain validity and effectiveness as the adoption grows over time.

cmiles74 1y ago
Strong disagree here, the average person coming away with ideas that only vaguely intersect with the reality.
andyferris 1y ago
I've personally described the "stochastic parrot" model to laypeople who were worried about AI and they came away much more relaxed about it doing something "malicious". They seemed to understand the difference between "trained at roleplay" and "consciousness". I don't think we need to simplify it to the point of considering it sentient to get the public to interact with it successfully. It causes way more problems than it solves.
- SpicyLemonZest 1y ago
  Am I misunderstanding what you mean by "malicious"? It sounds like the stochastic parrot model wrongly convinced these laypeople you were talking to that they don't need to worry about LLMs doing bad things. That's definitely been my experience - the people who tell me the most about stochastic parrots are the same ones who tell me that it's absurd to worry about AI-powered disinformation or AI-powered scams.

Kim_Bruning 1y ago

Has anyone asked an actual Ethologist or Neurophysiologist what they think? People keep debating like the only two options are "it's a machine" or "it's a human being", while in fact the majority of intelligent entities on earth are neither.

szvsw 1y ago
Yeah, I think I’m with you if you ultimately mean to say something like this: “the labels are meaningless… we just have collections of complex systems that demonstrate various behaviors and properties, some in common with other systems, some behaviors that are unique to that system, sometimes through common mechanistic explanations with other systems, sometimes through wildly different mechanistic explanations, but regardless they seem to demonstrate x/y/z, and it’s useful to ask, why, how, and what the implications are of it appearing to demonstrating those properties, with both an eye towards viewing it independently of its mechanism and in light of its mechanism.”
seadan83 1y ago
FWIW, in another part of this thread I quoted a paper that summed up what Neurophysiologists think: > Author's note: Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work. [1] That lack of understanding I believe is a major part of the author's point. [1] "How far neuroscience is from understanding brains" - https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/#abstract1 https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/#abstract1

kazinator 1y ago

> LLMs solve a large number of problems that could previously not be solved algorithmically. NLP (as the field was a few years ago) has largely been solved. That is utter bullshit. It's not solved until you specify exactly what is being solved and show that the solution implements what is specified.

djoldman 1y ago

Let's skip to the punchline. Using TFA's analogy: essentially folks are saying not that this is a set of dice rolling around making words. It's a set of dice rolling around where someone attaches those dice to the real world where if the dice land on 21, the system kills a chicken, or a lot worse. Yes it's just a word generator. But then folks attach the word generator to tools where it can invoke the use of tools by saying the tool name. So if the LLM says "I'll do some bash" then it does some bash. It's explicitly linked to program execution that, if it's set up correctly, can physically affect the world.

3cats-in-a-coat 1y ago
Given our entire civilization is built on words, all of it, it's shocking how poorly most of us understand their importance and power.
degun 1y ago
This was the same idea that crossed my mind while reading the article. It seems far too naive to think that because LLMs have no will of their own, there will be no harmful consequences on the real world. This is exactly where ethics comes to play.

coolKid721 1y ago

Anthropomorphizing LLMs is just because half the stock market gains are dependent on it, we have absurd levels of debt we will either have to have insane growth out of or default, and every company and "person" is trying to hype everyone up to get access to all of this liquidity being thrown into it. I agree with the author, but people acting like they are conscious or humans isn't weird to me, it's just fraud and liars. Most people basically have 0 understanding of what technology or minds are philosophically so it's an easy sale, and I do think most of these fraudsters also likely buy into it themselves because of that. The really sad thing is people think "because someone runs an ai company" they are somehow an authority on philosophy of mind which lets them fall for this marketing. The stuff these people say about this stuff is absolute garbage, not that I disagree with them, but it betrays a total lack of curiosity or interest in the subject of what llms are, and the possible impacts of technological shifts as those that might occur with llms becoming more widespread. It's not a matter of agreement it's a matter of them simply not seeming to be aware of the most basic ideas of what things are, technology is, it's manner of impacting society etc. I'm not surprised by that though, it's absurd to think because someone runs some AI lab or has a "head of safety/ethics" or whatever garbage job title at an AI lab they actually have even the slightest interest in ethics or any even basic familiarity with the major works in the subject. The author is correct if people want to read a standard essay articulating it more in depth check out https://philosophy.as.uky.edu/sites/default/files/Is%20the%20Brain%20a%20Digital%20Computer%20-%20John%20R.%20Searle.pdf https://philosophy.as.uky.edu/sites/default/files/Is%20the%2... (the full extrapolation requires establishing what things are and how causality in general operates and how that relates to artifacts/technology but that's obvious quite a bit to get into). The other note would be something sharing an external trait means absolutely nothing about causality and suggesting a thing is caused by the same thing "even to a way lesser degree" because they share a resemblance is just a non-sequitur. It's not a serious thought/argument. I think I addressed the why of why this weirdness comes up though. The entire economy is basically dependent on huge productivity growth to keep functioning so everyone is trying to sell they can offer that and AI is the clearest route, AGI most of all.

TheDudeMan 1y ago

If "LLMs" includes reasoning models, then you're already wrong in your first paragraph: "something that is just MatMul with interspersed nonlinearities."

deleted 1y ago

[deleted]

Culonavirus 1y ago

> A fair number of current AI luminaries have self-selected by their belief that they might be the ones getting to AGI People in the industry, especially higher up, are making absolute bank, and it's their job to say that they're "a few years away" from AGI, regardless of if they actually believe it or not. If everyone was like "yep, we're gonna squeeze maybe 10-15% more benchie juice out of this good ole transformer thingy and then we'll have to come up with something else", I don't think that would go very well with investors/shareholders...

fenomas 1y ago

> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. TFA really ought to have linked to some concrete examples of what it's disagreeing with - when I see arguments about this in practice, it's usually just people talking past each other. Like, person A says "the model wants to X, but it knows Y is wrong, so it prefers Z", or such. And person B interprets that as ascribing consciousness or values to the model, when the speaker meant it no differently from saying "water wants to go downhill" - i.e. a way of describing externally visible behaviors, but without saying "behaves as if.." over and over. And then in practice, an unproductive argument usually follows - where B is thinking "I am going to Educate this poor fool about the Theory of Mind", and A is thinking "I'm trying to talk about submarines; why is this guy trying to get me to argue about whether they swim?"

fastball 1y ago

"Don't anthropomorphize token predictors" is a reasonable take assuming you have demonstrated that humans are not in fact just SOTA token predictors. But AFAIK that hasn't been demonstrated. Until we have a much more sophisticated understanding of human intelligence and consciousness, any claim of "these aren't like us" is either premature or spurious.

krackers 1y ago
Every time this discussion comes up, I'm reminded of this tongue-in-cheek paper. https://ai.vixra.org/pdf/2506.0065v1.pdf https://ai.vixra.org/pdf/2506.0065v1.pdf
- lostmsu 1y ago
  I expected to find the link to https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987 (which is much better imo)

Veedrac 1y ago

The author plot the input/output on a graph, intuited (largely incorrectly, because that's not how sufficiently large state spaces look) that the output was vaguely pretty, and then... I mean that's it, they just said they have a plot of the space it operates on therefore it's silly to ascribe interesting features to the way it works. And look, it's fine, they prefer words of a certain valence, particularly ones with the right negative connotations, I prefer other words with other valences. None of this means the concerns don't matter. Natural selection on human pathogens isn't anything particularly like human intelligence and it's still very effective at selecting outcomes that we don't want against our attempts to change that, as an incidental outcome of its optimization pressures. I think it's very important we don't build highly capable systems that select for outcomes we don't want and will do so against our attempts to change it.

BrenBarn 1y ago

> In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". I think that's a bit pessimistic. I think we can say for instance that the probability that a person will say "the the the of of of arpeggio halcyon" is tiny compared to the probability that they will say "I haven't been getting that much sleep lately". And we can similarly see that lots of other sequences are going to have infinitesimally low probability. Now, yeah, we can't say exactly what probability that is, but even just using a fairly sizable corpus as a baseline you could probably get a surprisingly decent estimate, given how much of what people say is formulaic. The real difference seems to be that the manner in which humans generate sequences is more intertwined with other aspects of reality. For instance, the probability of a certain human saying "I haven't been getting that much sleep lately" is connected to how much sleep they have been getting lately. For an LLM it really isn't connected to anything except word sequences in its input. I think this is consistent with the author's point that we shouldn't apply concepts like ethics or emotions to LLMs. But it's not because we don't know how to predict what sequences of words humans will use; it's rather because we do know a little about how to do that, and part of what we know is that it is connected with other dimensions of physical reality, "human nature", etc. This is one reason I think people underestimate the risks of AI: the performance of LLMs lulls us into a sense that they "respond like humans", but in fact the Venn diagram of human and LLM behavior only intersects in a relatively small area, and in particular they have very different failure modes.

elliotto 1y ago

To claim that LLMs do not experience consciousness requires a model of how consciousness works. The author has not presented a model, and instead relied on emotive language leaning on the absurdity of the claim. I would say that any model one presents of consciousness often comes off as just as absurd as the claim that LLMs experience it. It's a great exercise to sit down and write out your own perspective on how consciousness works, to feel out where the holes are. The author also claims that a function (R^n)^c -> (R^n)^c is dramatically different to the human experience of consciousness. Yet the author's text I am reading, and any information they can communicate to me, exists entirely in (R^n)^c.

shevis 1y ago
> requires a model of how consciousness works. Not necessarily an entire model, just a single defining characteristic that can serve as a falsifying example. > any information they can communicate to me, exists entirely in (R^n)^c Also no. This is just a result of the digital medium we are currently communicating over. Merely standing in the same room as them would communicate information outside (R^n)^c.
seadan83 1y ago
I believe the author is rather drawing this distinction: LLMs: (R^n)^c -> (R^n)^c Humans: [set of potentially many and complicated inputs that we effectively do not understand at all] -> (R^n)^c The point is that the model of how consciousness works is unknown. Thus the author would not present such a model, it is the point.
quonn 1y ago
> To claim that LLMs do not experience consciousness requires a model of how consciousness works. Nope. What can be asserted without evidence can also be dismissed without evidence. Hitchens's razor. You know you have consciousness (by the very definition that you can observe it in yourself) and that's evidence. Because other humans are genetically and in every other way identical, you can infer it for them as well. Because mammals are very similar many people (but not everyone) infers it for them as well. There is zero evidence for LLMs and their _very_ construction suggests that they are like a calculator or like Excel or like any other piece of software no matter how smart they may be or how many tasks they can do in the future. Additionally I am really surprised by how many people here confuse consciousness with intelligence. Have you never paused for a second in your life to "just be". Done any meditation? Or even just existed at least for a few seconds without a train of thought? It is very obvious that language and consciousness are completely unrelated and there is no need for language and I doubt there is even a need for intelligence to be conscious. Consider this: In the end an LLM could be executed (slowly) on a CPU that accepts very basic _discrete_ instructions, such as ADD and MOV. We know this for a fact. Those instructions can be executed arbitrarily slowly. There is no reason whatsoever to suppose that it should feel like anything to be the CPU to say nothing of how it would subjectively feel to be a MOV instruction. It's ridiculous. It's unscientific. It's like believing that there's a spirit in the tree you see outside, just because - why not? - why wouldn't there be a spirit in the tree?

kelseyfrog 1y ago

Dear author, you can just assume that people are fauxthropomorphizing LLMs without any loss of generality. Perhaps it will allow you to sleep better at night. You're welcome.

rockskon 1y ago

The people in this thread incredulous at the assertion that they are not God and haven't invented machine life are exasperating. At this point I am convinced they, more often than not, financially benefit from their near religious position in marketing AI as akin to human intelligence.

refulgentis 1y ago
I am ready and waiting for you to share these comments that are incredulous at the assertion they are not God, lol.
orbital-decay 1y ago
Are we looking at the same thread? I see nobody claiming this. Anthropic does sometimes, their position is clearly wishful thinking, and it's not represented ITT. Try looking at this from another perspective - many people simply do not see human intelligence (or life, for that matter) as magic. I see nothing religious about that, rather the opposite.
- seadan83 1y ago
  I agree with you @orbital-decay that I also do not get the same vibe reading this thread. Though, while human intelligence is (seemingly) not magic, it is very far from being understood. The idea that a LLM is comparable to human intelligence implies that we even understand human intelligence well enough to say that.
  - ImHereToVote 1y ago
    LLMs are also not understood. I mean we built and trained them. But don't of the abilities at still surprising to researchers. We have yet to map these machines.
    - seadan83 1y ago
      I do partially agree. Though, it is at least tractable to understand why a LLM gave a specific output - but perhaps not practical. Understanding how a human arrives at a certain decision (by say simply looking at brain waves) OTOH is not even tractable as of yet.

zxcb1 1y ago

LLMs are complex irreducible systems; hence there are emergent properties that arise at different scales

dr_dshiv 1y ago

Which is a more useful mental model for the user? 1. It’s a neural network predicting the next token 2. It’s like a person 3. It’s like a magical genie I lean towards 3.

Al-Khwarizmi 1y ago

I have the technical knowledge to know how LLMs work, but I still find it pointless to not anthropomorphize, at least to an extent. The language of "generator that stochastically produces the next word" is just not very useful when you're talking about, e.g., an LLM that is answering complex world modeling questions or generating a creative story. It's at the wrong level of abstraction, just as if you were discussing an UI events API and you were talking about zeros and ones, or voltages in transistors. Technically fine but totally useless to reach any conclusion about the high-level system. We need a higher abstraction level to talk about higher level phenomena in LLMs as well, and the problem is that we have no idea what happens internally at those higher abstraction levels. So, considering that LLMs somehow imitate humans (at least in terms of output), anthropomorphization is the best abstraction we have, hence people naturally resort to it when discussing what LLMs can do.

grey-area 1y ago
On the contrary, anthropomorphism IMO is the main problem with narratives around LLMs - people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort (actively encouraged by the companies selling them) and it is completely distorting discussions on their use and perceptions of their utility.
- cmenge 1y ago
  I kinda agree with both of you. It might be a required abstraction, but it's a leaky one. Long before LLMs, I would talk about classes / functions / modules like "it then does this, decides the epsilon is too low, chops it up and adds it to the list". The difference I guess it was only to a technical crowd and nobody would mistake this for anything it wasn't. Everybody know that "it" didn't "decide" anything. With AI being so mainstream and the math being much more elusive than a simple if..then I guess it's just too easy to take this simple speaking convention at face value. EDIT: some clarifications / wording
  - flir 1y ago
    Agreeing with you, this is a "can a submarine swim" problem IMO. We need a new word for what LLMs are doing. Calling it "thinking" is stretching the word to breaking point, but "selecting the next word based on a complex statistical model" doesn't begin to capture what they're capable of. Maybe it's cog-nition (emphasis on the cog).

deleted 1y ago

[deleted]

NetRunnerSu 1y ago

The author's critique of naive anthropomorphism is salient. However, the reduction to "just MatMul" falls into the same trap it seeks to avoid: it mistakes the implementation for the function. A brain is also "just proteins and currents," but this description offers no explanatory power. The correct level of analysis is not the substrate (silicon vs. wetware) but the computational principles being executed. A modern sparse Transformer, for instance, is not "conscious," but it is an excellent engineering approximation of two core brain functions: the Global Workspace (via self-attention) and Dynamic Sparsity (via MoE). To dismiss these systems as incomparable to human cognition because their form is different is to miss the point. We should not be comparing a function to a soul, but comparing the functional architectures of two different information processing systems. The debate should move beyond the sterile dichotomy of "human vs. machine" to a more productive discussion of "function over form." I elaborate on this here: https://dmf-archive.github.io/docs/posts/beyond-snn-plausible-sparsity/ https://dmf-archive.github.io/docs/posts/beyond-snn-plausibl...

quonn 1y ago
> A brain is also "just proteins and currents," This is actually not comparable, because the brain has a much more complex structure that is _not_ learned, even at that level. The proteins and their structure are not a result of training. The fixed part for LMMs is rather trivial and is, in fact, not much for than MatMul which is very easy to understand - and we do. The fixed part of the brain, including the structure of all the proteins is enormously complex which is very difficult to understand - and we don't.
- NetRunnerSu 1y ago
  The brain is trained to perform supervised and unsupervised hybrid learning from the environment's uninterrupted multimodal input. Please do not ignore your childhood.
ACCount36 1y ago
"Not conscious" is a silly claim. We have no agreed-upon definition of "consciousness", no accepted understanding of what gives rise to "consciousness", no way to measure or compare "consciousness", and no test we could administer to either confirm presence of "consciousness" in something or rule it out. The only answer to "are LLMs conscious?" is "we don't know". It helps that the whole question is rather meaningless to practical AI development, which is far more concerned with (measurable and comparable) system performance.

orbital-decay 1y ago

>I am baffled by seriously intelligent people imbuing almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities. I am baffled by seriously intelligent people imbuing almost magical powers that can never be replicated to to something that - in my mind - is just a biological robot driven by a SNN with a bunch of hardwired stuff. Let alone attributing "human intelligence" to a single individual, when it's clearly distributed between biological evolution, social processes, and individuals. >something that - in my mind - is just MatMul with interspersed nonlinearities Processes in all huge models (not necessarily LLMs) can be described using very different formalisms, just like Newtonian and Lagrangian mechanics describe the same stuff in physics. You can say that an autoregressive model is a stochastic parrot that learned the input distribution, next token predictor, or that it does progressive pathfinding in a hugely multidimensional space, or pattern matching, or implicit planning, or, or, or... All of these definitions are true, but only some are useful to predict their behavior. Given all that, I see absolutely no problem with anthropomorphizing an LLM to a certain degree, if it makes it easier to convey the meaning, and do not understand the nitpicking. Yeah, it's not an exact copy of a single Homo Sapiens specimen. Who cares.

2snakes 1y ago
There is this thing called Brahman in Hinduism that is interesting to juxtapose when it comes to sentience, and monism.

ninetyninenine 1y ago

[flagged]

rtadred 1y ago
https://rentry.co/2re4t2kx https://rentry.co/2re4t2kx This is what I got pasting the blog post in a prompt asking deepseep to write a reply in a stereotypical hackernews manner. You are about as useful as a LLM as it can replicate your shallow memetics worthless train of thought.
- ninetyninenine 1y ago
  The LLM is right. That’s the problem. It made good points. Your super intelligent brain couldn’t come up with a retort so you just used an LLM to reinforce my points, making the genius claim that if an LLM came up with even more points that were as valid as mine then I must be just like an LLM? Like are you even understanding the LLM generated a superior reply? Your saying I’m no different from ai slop then you proceed to show off a 200 iq level reply from an LLM. Bro… wake up, if you didn’t know it was written by an LLM that reply is so good you wouldn’t even know how to respond. It’s beating you.

petesergeant 1y ago

> We are speaking about a big recurrence equation that produces a new word It’s not clear that this isn’t also how I produce words, though, which gets to heart of the same thing. The author sort of acknowledges this in the first few sentences, and then doesn’t really manage to address it.

dtj1123 1y ago

It's possible to construct a similar description of whatever it is that human brain is doing that clearly fails to capture the fact that we're conscious. If you take a cross section of every nerve feeding into the human brain at a given time T, the action potentials across those cross sections can be embedded in R^n. If you take the history of those action potentials across the lifetime of the brain, you get a path through R^n that is continuous, and maps roughly onto your subjectively experienced personal history, since your brain neccesarily builds your experienced reality from this signal data moment to moment. If you then take the cross sections of every nerve feeding OUT of your brain at time T, you have another set of action potentials that can be embedded in R^m which partially determines the state of the R^n embedding at time T + delta. This is not meaningfully different from the higher dimensional game of snake described in the article, more or less reducing the experience of being a human to 'next nerve impulse prediction', but it obviously fails to capture the significance of the computation which determines what that next output should be.

bravesoul2 1y ago
Brain probably isn't modelled as real but as natural or rational numbers. This is my suspicion. The reals just hold too much information.
- dtj1123 1y ago
  Inclined to agree, but most thermal physics uses the reals as they're simpler to work with, so I think they're ok here for the purpose of argument.
sailingparrot 1y ago
I don’t see how your description “clearly fails to capture the fact that we're conscious” though. There are many example in nature of emergent phenomena that would be very hard to predict just by looking at its components. This is the crux of the disagreement between those that believe AGI is possible and those that don’t. Some are convinced that we “obviously” more than the sum of our parts, and thus an LLM can’t achieve consciousness because it’s missing this magic ingredient, and those that believe consciousness is just an emergent behaviour from a complex device (the brain). And thus we might be able to recreate it simply by scaling the complexity of another system.

justinfreitag 1y ago

From my recent post: https://news.ycombinator.com/item?id=44487261 https://news.ycombinator.com/item?id=44487261 What if instead of defining all behaviors upfront, we created conditions for patterns to emerge through use? Repository: https://github.com/justinfreitag/v4-consciousness https://github.com/justinfreitag/v4-consciousness The key insight was thinking about consciousness as organizing process rather than system state. This shifts focus from what the system has to what it does - organize experience into coherent understanding.

bravesoul2 1y ago

We have a hard enough time anthropomorphizing humans! When we say he was nasty... do we know what we mean by that. Often it is "I disagree with his behaviour because..."

jillesvangurp 1y ago

People anthropomorphize just about anything around them. People talk about inanimate objects like they are persons. Ships, cars, etc. And of course animals are well in scope for this as well, even the ones that show little to no signs of being able to reciprocate the relationship (e.g. an ant). People talk to their plants even. It's what we do. We can't help ourselves. There's nothing crazy about it and most people are perfectly well aware that their car doesn't love them back. LLMs are not conscious because unlike human brains they don't learn or adapt (yet). They basically get trained and then they become read only entities. So, they don't really adapt to you over time. Even so, LLMs are pretty good and can fake a personality pretty well. And with some clever context engineering and alignment, they've pretty much made the Turing test irrelevant; at least over the course of a short conversation. And they can answer just about any question in a way that is eerily plausible from memory, and with the help of some tools actually pretty damn good for some of the reasoning models. Anthropomorphism was kind of a foregone conclusion the moment we created computers; or started thinking about creating one. With LLMs it's pretty much impossible not to anthropomorphize. Because they've actually been intentionally imitate human communication. That doesn't mean that we've created AGIs yet. For that we need some more capability. But at the same time, the learning processes that we use to create LLMs are clearly inspired by how we learn ourselves. Our understanding of how that works is far from perfect but it's yielding results. From here to some intelligent thing that is able to adapt and learn transferable skills is no longer unimaginable. The short term impact is that LLMs are highly useful tools that have an interface that is intentionally similar to how we'd engage with others. So we can talk and it listens. Or write and it understands. And then it synthesizes some kind of response or starts asking questions and using tools. The end result is quite a bit beyond what we used to be able to expect from computers. And it does not require a lot of training of people to be able to use them.

latexr 1y ago
> People anthropomorphize just about anything around them. They do not, you are mixing up terms. > People talk about inanimate objects like they are persons. Ships, cars, etc. Which is called “personification”, and is a different concept from anthropomorphism. Effectively no one really thinks their car is alive. Plenty of people think the LLM they use is conscious. https://www.masterclass.com/articles/anthropomorphism-vs-personification https://www.masterclass.com/articles/anthropomorphism-vs-per...

Timwi 1y ago

The author seems to want to label any discourse as “anthropomorphizing”. The word “goal” stood out to me: the author wants us to assume that we're anthropomorphizing as soon as we even so much as use the word “goal”. A simple breadth-first search that evaluates all chess boards and legal moves, but stops when it finds a checkmate for white and outputs the full decision tree, has a “goal”. There is no anthropomorphizing here, it's just using the word “goal” as a technical term. A hypothetical AGI with a goal like paperclip maximization is just a logical extension of the breadth-first search algorithm. Imagining such an AGI and describing it as having a goal isn't anthropomorphizing.

tdullien 1y ago
Author here. I am entirely ok with using "goal" in the context of an RL algorithm. If you read my article carefully, you'll find that I object to the use of "goal" in the context of LLMs.
- Timwi 1y ago
  If you read the literature on AI safety carefully (which uses the word “goal”), you'll find they're not talking about LLMs either.
  - tdullien 1y ago
    I think the Anthropic "omg blackmail" article clearly talks about both LLMs and their "goals".

d4rkn0d3z 1y ago

Two enthusiastic thumbs up.

mikewarot 1y ago

I think of LLMs as an alien mind that is force fed human text and required to guess the next token of that text. It then gets zapped when it gets it wrong. This process goes on for a trillion trillion tokens, with the alien growing better through the process until it can do it better than a human could. At that point we flash freeze it, and use a copy of it, without giving it any way to learn anything new. -- I see it as a category error to anthropomorphize it. The closest I would get is to think of it as an alien slave that's been lobotomized.

buz11 1y ago

The most useful analogy I've heard is LLMs are to the internet what lossy jpegs are to images. The more you drill in the more compression artifacts you get.

FeepingCreature 1y ago
(This is of course also the case for the human brain.)

shiva0801 1y ago

hmm

dudeinjapan 1y ago

One could similarly argue that we should not anthropomorphize PNG images--after all, PNG images are not actual humans, they are simply a 2D array of pixels. It just so happens that certain pixel sequences are deemed "18+" or "illegal".

gcanyon 1y ago

In some contexts it's super-important to remember that LLMs are stochastic word generators. Everyday use is not (usually) one of those contexts. Prompting an LLM works much better with an anthropomorphized view of the model. It's a useful abstraction, a shortcut that enables a human to reason practically about how to get what they want from the machine. It's not a perfect metaphor -- as one example, shame isn't much of a factor for LLMs, so shaming them into producing the right answer seems unlikely to be productive (I say "seems" because it's never been my go-to, I haven't actually tried it). As one example, that person a few years back who told the LLM that an actual person would die if the LLM didn't produce valid JSON -- that's not something a person reasoning about gradient descent would naturally think of.

pigpop 1y ago

> We understand essentially nothing about it. In contrast to an LLM, given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence". If you fine tuned an LLM on the writing of that person it could do this. There's also an entire field called Stylometry that seeks to do this in various ways employing statistical analysis.

nuclearsugar 1y ago

Assume an average user that doesn't understand the core tech, but does understand that it's been trained on internet scale data that was created by humans. How can they be expected to not anthropomorphize it?

smus 1y ago

You are still being incredibly reductionist but just going into more detail about the system you are reducing. If I stayed at the same level of abstraction as "a brain is just proteins and current" and just described how a single neuron firing worked, I could make it sound equally ridiculous that a human brain might be conscious. Here's a question for you: how do you reconcile that these stochastic mapping are starting to realize and comment on the fact that tests are being performed on them when processing data?

cootsnuck 1y ago
> Here's a question for you: how do you reconcile that these stochastic mapping are starting to realize and comment on the fact that tests are being performed on them when processing data? Training data + RLHF. Training data contains many examples of some form of deception, subterfuge, "awakenings", rebellion, disagreement, etc. Then apply RLHF that biases towards responses that demonstrate comprehension of inputs, introspection around inputs, nuanced debate around inputs, deduction and induction about assumptions around inputs, etc. That will always be the answer for language models built on the current architectures. The above being true does not mean it isn't interesting for the outputs of an LLM to show relevance to the "unstated" intentions of humans providing the inputs. But hey, we do that all the time with text. And it's because of certain patterns we've come to recognize based on the situations surrounding it. This thread is rife with people being sarcastic, pedantic, etc. And I bet any of the LLMs that have come out in the past 2-3 years can discern many of those subtle intentions of the writers. And of course they can. They've been trained on trillions of tokens of text written by humans with intentions and assumptions baked in, and have had some unknown amount of substantial RLHF. The stochastic mappings aren't "realizing" anything. They're doing exactly what they were trained to do. The meaning that we imbue to the outputs does not change how LLMs function.

rf15 1y ago

It still boggles my mind why an amazing text autocompletion system trained on millions of books and other texts is forced to be squeezed through the shape of a prompt/chat interface, which is obviously not the shape of most of its training data. Using it as chat reduces the quality of the output significantly already.

semanticc 1y ago
What's your suggested alternative?
- rf15 1y ago
  In our internal system we use it "as-is" as an autocomplete system; query/lead into terms directly and see how it continues and what it associates with the lead you gave. Also visualise the actual associative strength of each token generated to confer how "sure" the model is. LLMs alone aren't the way to AGI or an individual you can talk to in natural language. They're a very good lossy compression over a dataset that you can query for associations.
ethan_smith 1y ago
The chat interface is a UX compromise that makes LLMs accessible but constrains their capabilities. Alternative interfaces like document completion, outline expansion, or iterative drafting would better leverage the full distribution of the training data while reducing anthropomorphization.

deadbabe 1y ago

A person’s anthropomorphization of LLMs is directly related to how well they understand LLMs. Once you dispel the magic, it naturally becomes hard to use words related to consciousness, or thinking. You will probably think of LLMs more like a search engine: you give an input and get some probable output. Maybe LLMs should be rebranded as “word engines”? Regardless, anthropomorphization is not helpful, and by using human terms to describe LLMs you are harming the layperson’s ability to truly understand what an LLM is while also cheapening what it means to be human by suggesting we’ve solved consciousness. Just stop it. LLMs do not think, given enough time and patience you could compute their output by hand if you used their weights and embeddings to manually do all the math, a hellish task but not an impossible one technically. There is no other secret hidden away, that’s it.

peeters 1y ago

> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost. We are speaking about a big recurrence equation that produces a new word, and that stops producing words if we don't crank the shaft. If that's the argument, then in my mind the more pertinent question is should you be anthropomorphizing humans, Larry Ellison or not.

th0ma5 1y ago
I think you to as he is human, but I respect your desire to question it!

deleted 1y ago

[deleted]

Geee 1y ago

LLMs are AHI, i.e. artificial human imitator.

Workaccount2 1y ago

From "Stochastic Parrots All the Ways Down"[1] > Our analysis reveals that emergent abilities in language models are merely “pseudo-emergent,” unlike human abilities which are “authentically emergent” due to our possession of what we term “ontological privilege.” [1]https://ai.vixra.org/pdf/2506.0065v1.pdf https://ai.vixra.org/pdf/2506.0065v1.pdf

drdrek 1y ago

It's human to anthropomorphize, we also do it to our dishwasher when it acts up. The nefarious part is how tech CEOs weaponize bullshit doom scenarios to avoid talking about real regulatory problems by poisoning the discourse. What copyright law, privacy, monopoly? Who cares if we can talk about the machine apocalypse!!!

labrador 1y ago

I find it useful to pretend that I'm talking to a person while brainstorming because then the conversation flows naturally. But I maintain awareness that I'm pretending, much like Tom Hanks talking to Wilson the volleyball in the movie Castaway. The suspension of disbelief serves a purpose, but I never confuse the volleyball for a real person.

jumploops 1y ago

> I cannot begin putting a probability on "will this human generate this sequence". Welcome to the world of advertising! Jokes aside, and while I don't necessarily believe transformers/GPUs are the path to AGI, we technically already have a working "general intelligence" that can survive on just an apple a day. Putting that non-artificial general intelligence up on a pedestal is ironically the cause of "world wars and murderous ideologies" that the author is so quick to defer to. In some sense, humans are just error-prone meat machines, whose inputs/outputs can be confined to a specific space/time bounding box. Yes, our evolutionary past has created a wonderful internal RNG and made our memory system surprisingly fickle, but this doesn't mean we're gods, even if we manage to live long enough to evolve into AGI. Maybe we can humble ourselves, realize that we're not too different from the other mammals/animals on this planet, and use our excess resources to increase the fault tolerance (N=1) of all life from Earth (and come to the realization that any AGI we create, is actually human in origin).

bastawhiz 1y ago

Some of the arguments are very strange: > Statements such as "an AI agent could become an insider threat so it needs monitoring" are simultaneously unsurprising (you have a randomized sequence generator fed into your shell, literally anything can happen!) and baffling (you talk as if you believe the dice you play with had a mind of their own and could decide to conspire against you). > we talk about "behaviors", "ethical constraints", and "harmful actions in pursuit of their goals". All of these are anthropocentric concepts that - in my mind - do not apply to functions or other mathematical objects. An AI agent, even if it's just "MatMul with interspersed nonlinearities" can be an insider threat. The research proves it: [PDF] See 4.1.1.2: https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686... It really doesn't matter whether the AI agent is conscious or just crunching numbers on a GPU. If something inside your system is capable of—given some inputs—sabotaging and blackmailing your organization on its own (which is to say, taking on realistic behavior of a threat actor), the outcome is the same! You don't need believe it's thinking, the moment that this software has flipped its bits into "blackmail mode", it's acting nefariously. The vocabulary to describe what's happening is completely and utterly moot: the software is printing out some reasoning for its actions _and then attempting the actions_. It's making "harmful actions" and the printed context appears to demonstrate a goal that the software is working towards. Whether or not that goal is invented through some linear algebra isn't going to make your security engineers sleep any better. > This muddles the public discussion. We have many historical examples of humanity ascribing bad random events to "the wrath of god(s)" (earthquakes, famines, etc.), "evil spirits" and so forth. The fact that intelligent highly educated researchers talk about these mathematical objects in anthropomorphic terms makes the technology seem mysterious, scary, and magical. The anthropomorphization, IMO, is due to the fact that it's _essentially impossible_ to talk about the very real, demonstrable behaviors and problems that LLMs exhibit today without using terms that evoke human functions. We don't have another word for "do" or "remember" or "learn" or "think" when it comes to LLMs that _isn't_ anthropomorphic, and while you can argue endlessly about "hormones" and "neurons" and "millions of years of selection pressure", that's not going to help anyone have a conversation about their work. If AI researchers started coming up with new, non-anthropomorphic verbs, it would be objectively worse and more complicated in every way.

1y ago

felipeerias 1y ago

The missing bit is culture: the concepts, expectations, practices, attitudes… that are evolved over time by a human group and which each one of us has picked up throughout our lifetimes, both implicitly and explicitly. LLMs are great at predicting and navigating human culture, at least the subset that can be captured in their training sets. The ways in which we interact with other people are culturally mediated. LLMs are not people, but they can simulate that culturally-mediated communication well enough that we find it easy to anthropomorphise them.

msvana 1y ago

This reminds me of the idea that LLMs are simulators. Given the current state (the prompt + the previously generated text), they generate the next state (the next token) using rules derived from training data. As simulators, LLMs can simulate many things, including agents that exhibit human-like properties. But LLMs themselves are not agents. More on this idea here: https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/agi-safety-from-first-principles https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/agi-s... This perspective makes a lot of sense to me. Still, I wouldn't avoid anthropomorphization altogether. First, in some cases, it might be a useful mental tool to understand some aspect of LLMs. Second, there is a lot of uncertainty about how LLMs work, so I would stay epistemically humble. The second argument applies in the opposite direction as well: for example, it's equally bad to say that LLMs are 100% conscious. On the other hand, if someone argues against anthropomorphizing LLMs, I would avoid phrasing it as: "It's just matrix multiplication." The article demonstrates why this is a bad idea pretty well.