5 ms·

> There is no mechanism by which LLMs have agency. They have no internal desires, drives, motivations. Why? Folks make these strong assertions, and I don't get

> There is no mechanism by which LLMs have agency. They have no internal desires, drives, motivations. Why? Folks make these strong assertions, and I don't get where this confidence comes from. We're so comically ignorant of how our own minds work, let alone alien ones, or how any commonalities between them may manifest. What am I missing?

Retric 3y ago
You’re missing the underlying mechanism by which they operate. LLM’s don’t know anything beyond the current prompt and it’s “memory” of training data. They would sit for eternity with an empty prompt. You can change systems to behave differently, but it quickly stops being a LLM and turns into something else.
- l33tman 3y ago
  Sure, but apart from the detail that you can make them pause by not feeding them words, you can't technically argue that they lack all those things. They are stateful in the sense that they see what they write, so they can keep their inner plan and state in that way across word-iterations. They for sure work differently than a human brain, but without further pretty deep analysis you can't really claim that they can't reproduce similar traits using the mechanism.
  - Retric 3y ago
    Sometimes, type enough tokens and they no longer have any prior words written by the LLM in their context. Similarly the algorithms would still happily respond if some different and potentially completely unrelated LLM wrote the prior responses. LLM’s are really best thought of as improv actors. The prompt is in effect just the current skit being preformed. The intentions of the character being played doesn’t imply the actor always has those intentions. So yes they can run through a knock knock joke across multiple prompts, but the need not have written the start of a joke to be able to make up an ending.
    - l33tman 3y ago
      I agree, there is a current technical limit in the token context but that's a limit in practice, not in theory. There are plenty of wrapper tools around LLMs that cleverly use the token window to keep a longer "state of mind", overall strategies currently executing etc. With varying degrees of success, I should say... but still, it's kind of analogous to a human executing a strategy with intentions.
- sterlind 3y ago
  You'd sit for eternity if you suffered a lesion in your reticular activating system - a relatively small cluster of neurons that generates a kind of clock signal in animal brains. Coma patients with RAS lesions seem to visualize scenes, given prompts, despite not really being conscious. Conversely, ChatGPT does decently well on multi-armed bandit tasks, demonstrating (rudimentary) reinforcement learning capability during inference. It's known that LLMs evolve their own optimizers in the process of acquiring few-shot learning, so I assume it picked up these RL abilities similarly. That kind of on-line RL is foundational to autonomous agents. The prompt isn't part of the LLM, it's part of how the LLM is wired into a chat window. You can make them stream tokens forever, or prompt themselves, or ditch causality entirely. The foundational abilities for autonomy, I think, are in there, for the simple reason that they've learned to model autonomous agents - human beings.
  - Retric 3y ago
    The prompts or at least being fed a sequence of tokens including output from prior passes is integral to how language models function. Rather than being “hooked up to one” the neural networks only function is to pick a single token based on a set of inputs. So without being feed it’s own output you get a single token and then nothing. There’s some randomness injected into the process and whatnot but that’s ultimately just window dressing to make them seem less mechanical. There’s all kinds of ways to disrupt human or animal consciousness such as reducing oxygen supply, but saying the human brain is vulnerable doesn’t change anything about how it operates normally. Plenty of ways to break an LLM’s, but then you’re talking about a different system. Similarly the reticular activation system’s purpose is to regulate wakefulness, which aspects are directly useful or not isn’t particularly relevant because it’s part of the brain.
    - sterlind 3y ago
      No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why techniques like beam search and Viterbi work so well - you don't have to commit to the next token at each step. And temperature (what I assume you mean by "randomness injected") isn't "window dressing," it fundamentally gives better results because LMs model probability distributions. You'll get crappy results with any probability model if you run them purely greedily. And you're also neglecting non-causal LMs (like BERT, and encoders in general), which don't predict the next token in a series, but instead predict previous masked tokens. You're conflating how LLMs are used for generation with what LLMs are, and that's just plain wrong. They're not trained autoregressively at all! To repeat, the generation mechanism is simply not part of the LLM. The LLM is a probability model; the generator just uses that model. It's not "breaking it" to use a different generation strategy than greedy autoregression, since they're not even trained a token at a time.
- a_wild_dandan 3y ago
  I don't see a connection between how an agent works and what it experiences. Sure, depriving myself or LLMs of all neural activity results in uninteresting behavior. How does this fact buy us insight into how agents feel in other circumstances?
  - Retric 3y ago
    A human in a sensory deficit chamber still thinks demonstrating they aren’t necessarily purely reacting the the current stimulus. An LLM is only reacting to its current stimulus.
    - famouswaffles 3y ago
      a sensory deficit chamber isn't literally depriving people of any stimulus.
      - Retric 3y ago
        There’s no external light, sound, taste, or smell. Most people can always touch themselves or notice the passing of time etc. But it’s possible to be conscious without any external sensation.
        a_wild_dandan 3y ago
        You've exclusively described contexts with tons of active stimuli (temperature, interoception, balance, and frankly most of the other senses you named). But more importantly, suppose we grant that humans function independently of stimuli. Why does that matter? How does this premise imply anything about an agent's capacity for internal experience? In the counterfactual where our brains don't work when surgically placed in life-support vats, does that mean our prior experiences weren't real? I'm genuinely so confused at this connection between subjective experience and the necessity of stimulus.
        Retric 3y ago
fsociety 3y ago
In a sense it is a prediction model, a good one. I can accept that in some future, we may have a model that we label as this and it turns out it does. Who knows when, but this is an early iteration of what AI will be fwiw.
nullc 3y ago
You can wire the LLM up to an eval and kick it off. It will go about coming up with stuff do for some time before it falls into a rut. Make sure to sandbox it, as it can decide to wipe your computer.
KirillPanov 3y ago
> Why? Self-preservation results from survival of the fittest. It's totally unrelated to intelligence. People conflate the two because they're extrapolating from a sample size of one: the only intelligent thing they know of is humans. But that single sample also happens to have been evolved by survival of the fittest. I am totally unafraid of LLM's deciding that humans are a threat to them. I'll start being afraid if AI research suddenly stops using backpropagation and starts getting equally good results using genetic programming (this is highly unlikely).
semi-extrinsic 3y ago
Precisely this, we have almost no idea about how our minds work, how they obtain agency and free will and all that jazz. Large language models in our current paradigm developing agency would be like 16th century alchemists inventing nuclear fusion reactors.

Retric 3y ago

You’re missing the underlying mechanism by which they operate. LLM’s don’t know anything beyond the current prompt and it’s “memory” of training data. They would sit for eternity with an empty prompt. You can change systems to behave differently, but it quickly stops being a LLM and turns into something else.

l33tman 3y ago
Sure, but apart from the detail that you can make them pause by not feeding them words, you can't technically argue that they lack all those things. They are stateful in the sense that they see what they write, so they can keep their inner plan and state in that way across word-iterations. They for sure work differently than a human brain, but without further pretty deep analysis you can't really claim that they can't reproduce similar traits using the mechanism.
- Retric 3y ago
  Sometimes, type enough tokens and they no longer have any prior words written by the LLM in their context. Similarly the algorithms would still happily respond if some different and potentially completely unrelated LLM wrote the prior responses. LLM’s are really best thought of as improv actors. The prompt is in effect just the current skit being preformed. The intentions of the character being played doesn’t imply the actor always has those intentions. So yes they can run through a knock knock joke across multiple prompts, but the need not have written the start of a joke to be able to make up an ending.
  - l33tman 3y ago
    I agree, there is a current technical limit in the token context but that's a limit in practice, not in theory. There are plenty of wrapper tools around LLMs that cleverly use the token window to keep a longer "state of mind", overall strategies currently executing etc. With varying degrees of success, I should say... but still, it's kind of analogous to a human executing a strategy with intentions.
sterlind 3y ago
You'd sit for eternity if you suffered a lesion in your reticular activating system - a relatively small cluster of neurons that generates a kind of clock signal in animal brains. Coma patients with RAS lesions seem to visualize scenes, given prompts, despite not really being conscious. Conversely, ChatGPT does decently well on multi-armed bandit tasks, demonstrating (rudimentary) reinforcement learning capability during inference. It's known that LLMs evolve their own optimizers in the process of acquiring few-shot learning, so I assume it picked up these RL abilities similarly. That kind of on-line RL is foundational to autonomous agents. The prompt isn't part of the LLM, it's part of how the LLM is wired into a chat window. You can make them stream tokens forever, or prompt themselves, or ditch causality entirely. The foundational abilities for autonomy, I think, are in there, for the simple reason that they've learned to model autonomous agents - human beings.
- Retric 3y ago
  The prompts or at least being fed a sequence of tokens including output from prior passes is integral to how language models function. Rather than being “hooked up to one” the neural networks only function is to pick a single token based on a set of inputs. So without being feed it’s own output you get a single token and then nothing. There’s some randomness injected into the process and whatnot but that’s ultimately just window dressing to make them seem less mechanical. There’s all kinds of ways to disrupt human or animal consciousness such as reducing oxygen supply, but saying the human brain is vulnerable doesn’t change anything about how it operates normally. Plenty of ways to break an LLM’s, but then you’re talking about a different system. Similarly the reticular activation system’s purpose is to regulate wakefulness, which aspects are directly useful or not isn’t particularly relevant because it’s part of the brain.
  - sterlind 3y ago
    No, it doesn't pick a single token based on a set of inputs. It predicts a probability distribution for the next token given the previous tokens. That's why techniques like beam search and Viterbi work so well - you don't have to commit to the next token at each step. And temperature (what I assume you mean by "randomness injected") isn't "window dressing," it fundamentally gives better results because LMs model probability distributions. You'll get crappy results with any probability model if you run them purely greedily. And you're also neglecting non-causal LMs (like BERT, and encoders in general), which don't predict the next token in a series, but instead predict previous masked tokens. You're conflating how LLMs are used for generation with what LLMs are, and that's just plain wrong. They're not trained autoregressively at all! To repeat, the generation mechanism is simply not part of the LLM. The LLM is a probability model; the generator just uses that model. It's not "breaking it" to use a different generation strategy than greedy autoregression, since they're not even trained a token at a time.
a_wild_dandan 3y ago
I don't see a connection between how an agent works and what it experiences. Sure, depriving myself or LLMs of all neural activity results in uninteresting behavior. How does this fact buy us insight into how agents feel in other circumstances?
- Retric 3y ago
  A human in a sensory deficit chamber still thinks demonstrating they aren’t necessarily purely reacting the the current stimulus. An LLM is only reacting to its current stimulus.
  - famouswaffles 3y ago
    a sensory deficit chamber isn't literally depriving people of any stimulus.
    - Retric 3y ago
      There’s no external light, sound, taste, or smell. Most people can always touch themselves or notice the passing of time etc. But it’s possible to be conscious without any external sensation.
      - a_wild_dandan 3y ago
        You've exclusively described contexts with tons of active stimuli (temperature, interoception, balance, and frankly most of the other senses you named). But more importantly, suppose we grant that humans function independently of stimuli. Why does that matter? How does this premise imply anything about an agent's capacity for internal experience? In the counterfactual where our brains don't work when surgically placed in life-support vats, does that mean our prior experiences weren't real? I'm genuinely so confused at this connection between subjective experience and the necessity of stimulus.
        Retric 3y ago

fsociety 3y ago

In a sense it is a prediction model, a good one. I can accept that in some future, we may have a model that we label as this and it turns out it does. Who knows when, but this is an early iteration of what AI will be fwiw.

nullc 3y ago

You can wire the LLM up to an eval and kick it off. It will go about coming up with stuff do for some time before it falls into a rut. Make sure to sandbox it, as it can decide to wipe your computer.

KirillPanov 3y ago

> Why? Self-preservation results from survival of the fittest. It's totally unrelated to intelligence. People conflate the two because they're extrapolating from a sample size of one: the only intelligent thing they know of is humans. But that single sample also happens to have been evolved by survival of the fittest. I am totally unafraid of LLM's deciding that humans are a threat to them. I'll start being afraid if AI research suddenly stops using backpropagation and starts getting equally good results using genetic programming (this is highly unlikely).

semi-extrinsic 3y ago

Precisely this, we have almost no idea about how our minds work, how they obtain agency and free will and all that jazz. Large language models in our current paradigm developing agency would be like 16th century alchemists inventing nuclear fusion reactors.