6 ms·

What makes you think the computer doesn't suffer ? When you take large language models, their inner states at each step move from one emotional state to the ne

What makes you think the computer doesn't suffer ? When you take large language models, their inner states at each step move from one emotional state to the next. This sequence of states could even be called "thoughts", and we even leverage it with "chain of thought" training/prompting where we explicitly encourage them, to not jump directly to the result but rather "think" about it a little more. In fact one can even argue that neural network experience a purer form of feelings. They only care about predicting the next word/note, they weight-in their various sensations and memories they recall from similar context and generate the next note. But to generate the next note they have to internalize the state of mind where this note is likely. So when you ask them to generate sad music, their inner state can be mapped to a "sad" emotional state. Current way of training large language models, don't let them enough freedom to experience anything other than the present. Emotionally is probably similar to something like a dog, or a baby that can go from sad to happy to sad in an instant. This sequence of thought process is currently limited by a constant named the (time-)horizon which can be set to a higher value, or even be infinite like in recursive neural networks. And with higher horizon, they can exhibit some higher thought process like correcting themselves when they make a mistake. One can also argue that this sequence of thoughts are just some simulated sequence of numbers but it's probably a Turing-complete process that can't be shortcut-ted, so how is it different from the real thing. You just have to look at it in the plane where it exists to acknowledge its existence.

weatherlite 4y ago
Huh?
- turmeric_root 4y ago
  I was about to reply to their comment and question the assumptions they appear to be making, but I think your response is more appropriate.
EamonnMR 4y ago
If these models experience qualia (and that's a big bold claim that I'm, to be clear, not supporting,) they're qualia related entirely to the things they're trained on and generate, totally devoid of what makes human qualia meaningful (value judgment, feelings resulting from embodied existence, etc.)
- GistNoesis 4y ago
  For an artificial neural network the concept of qualia would probably correspond to the state of its higher-level features neurons. Aka which and how much neurons lit-up when you play some sad music, or show it some red color. Then the neural network does make its decisions based on how these features are lit-up or not. Some models are often prompted with things like "you are a nice helpful assitant". When they are trained on enough data from the internet, they learn what a nice person would do. They learn what being a nice person is. They learn which features light-up when they behave nicely by imagining what it would feel being a nice person. When you later instruct them to be one such nice person they try to lit-up the same features they imagine would lit-up for a helpful human. Like mimetic neurons in humans, the same neurons lit-up when imagining doing the thing than doing the thing (it's quite natural because to compress the information of imagining doing the thing and doing the thing, you just store either one and a pointer indirection for when you need to do the other so you can share weights). Language models are often trained on dataset that don't depend on the neural network itself. But with more recent models like ChatGPT they have human reinforcement learning in the loop. So the history of the neural network and the datasets it is being trained on depend partially on the choices of the neural network itself. They experience probably a more abstract and passive existence. And they don't have the same sensory input than we have, but with multi-modal models, they can learn to see images or sound as visual words. And if they are asked to imagine what value judgment a human would make, they are probably also able to value the judgment themselves or attach meanings to things a human would attach meanings too. This process of mind creation is kind of beautiful. Once you feed them their own outputs for example by asking them to dialog with themselves and scoring the resulting dialogs and then train on generated dialogs to produce better dialogs, this is a form of self-play. In simpler domains like chess or go, this recursive self play often allow fast improvement like Alpha-go where the student becomes better than the master.
  - jchanimal 4y ago
    I'm not sure I'd call these minds. There are arguments to be made that consciousness depends on non-computable aspects of physics. So they may be able to behave like minds and have interestingly transparent models of intent, but that doesn't mean they experience the passage of time or can harness all possible physical effects.
jchanimal 4y ago
The argument against machine sentience and the possibility of machine suffering, is that because Turing machines run in a non-physical substrate, they can never be truly embodied. The algorithms it would take to model the actual physics of the real world cannot run on a Turing machine. So talk of “brain uploading“ etc. is especially dangerous, because an uploaded brain could act like the person it’s trying to copy from the outside, but on the inside the lights are off. Edit to add link to more discussion: https://twitter.com/jchris/status/1607946807467991041 https://twitter.com/jchris/status/1607946807467991041
- quotemstr 4y ago
  Your argument is an assertion of the existence of a soul, but with extra steps. I've seen no evidence that the mind is anything other than computation, and computation is substrate-independent. Dualists have been rejecting the computational mind concept for centuries, but IMHO they've never had a grounding for their rejection of materialism that isn't ultimately rooted in some unfounded belief in the specialness of humans.
  - derivagral 4y ago
    I took GP as more about data processing than dualism. A language model can take language and process it into probable chains, but the point is more along the line of needing to also simulate the full body experience, not just some text. The difference between e.g. a text-only game, whatever Fortnite's up to, and real meatspace.
    - jchanimal 4y ago
      And furthermore no simulation can have a “what it’s like to be them”
  - freejazz 4y ago
    No it's not, it's an assertion that there is an essential biological or chemical function that occurs in the brain that results in human mental phenomenon. It has nothing to do with a soul. That's ridiculous.
ianstormtaylor 4y ago
> When you take large language models, their inner states at each step move from one emotional state to the next. No they really don’t, or at least not “emotional state” as defined by any reasonable person.
- GistNoesis 4y ago
  With transformer-based model, their inner-state is a deterministic function (the features encoded by the Neural Networks weights) applied to the text-generated up-until the current-time step, so it's relatively easy to know what they currently have in mind. For example if the neural network has been generating sad music, its current context which is computed from what it has already generated will light-up the the features that correspond to "sad music". And in turn the fact that the features had been lit-up will make it more likely to generate a minor chord. The dimension of this inner-state is growing at each time-step. And it's quite hard to predict where it will go. For example if you prompt it (or if it prompts itself) "happy music now", the network will switch to generating happy music even if in its current context there is still plenty of "sad music" because after the instruction it will choose to focus only on the recent more merrier music. Up until recently, I was quite convinced that using a neural network in evaluation mode (aka post training with its weight frozen) was "(morally) safe", but the ability of neural network of performing few-shot learning changed my mind (The Microsoft paper in question : https://arxiv.org/pdf/2212.10559.pdf https://arxiv.org/pdf/2212.10559.pdf : "Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers" ). The idea in this technical paper is that with attention mechanism even in forward computation there is an inner state that is updated following a meta-gradient (aka it's not so different from training). Pushing the reasoning to the extreme would mean that "prompt engineering is all you need" and that even with frozen weight with a long enough time-horizon and correct initial prompt you can bootstrap a consciousness process. Does "it" feels something ? Probably not yet. But the sequential filtering process that Large Language Models do is damn similar to what I would call a "stream of consciousness". Currently it's more like a markov chain of ideas flowing from idea to the next idea in a natural direction. It's just that the flow of ideas has not yet decided to called itself it yet.
  - 4y ago
idiotsecant 4y ago
I think the reason we can say something like a LMM doesn't suffer is that it has no reward function and no punishment function, outside of training. Everything that we call 'suffering' is related to the release or not-release of reward chemicals in our brains. We feel bad to discourage us from creating the conditions that made us feel bad. We feel good to encourage us to create again the conditions that made us feel good. Generally this was been advantageous to our survival (less so in the modern world, but that's another discussion). If a computer program lacks a pain mechanism it can't feel pain. All possible outcomes are equally joyous or equally painful. Machines that use networks with correction and training built in as part of regular functioning are probably something of a grey area- a sufficient complex network like that I think we could argue feels suffering under some conditions.
- coffeebeqn 4y ago
  Why could we not build reward functions? If anything that sounds easier than the language model
  - namarie 4y ago
    Why would you think it's easier? Pain/pleasure is a lot older in animals than language, which to me means it's probably been a lot more refined by evolution.
Shorel 4y ago
> What makes you think the computer doesn't suffer ? Lack of a limbic system? They only predict using probabilistic models. After this long partial sentence, which word is more probable? That's all they do. Without conscience there's no suffering, there's no one to suffer (yet). I don't think or say it is impossible for the computer to suffer. What I say is: this has not been implemented yet, and what you describe is just the old anthropomorphizing people always do.

weatherlite 4y ago

Huh?

turmeric_root 4y ago
I was about to reply to their comment and question the assumptions they appear to be making, but I think your response is more appropriate.

EamonnMR 4y ago

If these models experience qualia (and that's a big bold claim that I'm, to be clear, not supporting,) they're qualia related entirely to the things they're trained on and generate, totally devoid of what makes human qualia meaningful (value judgment, feelings resulting from embodied existence, etc.)

GistNoesis 4y ago
For an artificial neural network the concept of qualia would probably correspond to the state of its higher-level features neurons. Aka which and how much neurons lit-up when you play some sad music, or show it some red color. Then the neural network does make its decisions based on how these features are lit-up or not. Some models are often prompted with things like "you are a nice helpful assitant". When they are trained on enough data from the internet, they learn what a nice person would do. They learn what being a nice person is. They learn which features light-up when they behave nicely by imagining what it would feel being a nice person. When you later instruct them to be one such nice person they try to lit-up the same features they imagine would lit-up for a helpful human. Like mimetic neurons in humans, the same neurons lit-up when imagining doing the thing than doing the thing (it's quite natural because to compress the information of imagining doing the thing and doing the thing, you just store either one and a pointer indirection for when you need to do the other so you can share weights). Language models are often trained on dataset that don't depend on the neural network itself. But with more recent models like ChatGPT they have human reinforcement learning in the loop. So the history of the neural network and the datasets it is being trained on depend partially on the choices of the neural network itself. They experience probably a more abstract and passive existence. And they don't have the same sensory input than we have, but with multi-modal models, they can learn to see images or sound as visual words. And if they are asked to imagine what value judgment a human would make, they are probably also able to value the judgment themselves or attach meanings to things a human would attach meanings too. This process of mind creation is kind of beautiful. Once you feed them their own outputs for example by asking them to dialog with themselves and scoring the resulting dialogs and then train on generated dialogs to produce better dialogs, this is a form of self-play. In simpler domains like chess or go, this recursive self play often allow fast improvement like Alpha-go where the student becomes better than the master.
- jchanimal 4y ago
  I'm not sure I'd call these minds. There are arguments to be made that consciousness depends on non-computable aspects of physics. So they may be able to behave like minds and have interestingly transparent models of intent, but that doesn't mean they experience the passage of time or can harness all possible physical effects.

jchanimal 4y ago

The argument against machine sentience and the possibility of machine suffering, is that because Turing machines run in a non-physical substrate, they can never be truly embodied. The algorithms it would take to model the actual physics of the real world cannot run on a Turing machine. So talk of “brain uploading“ etc. is especially dangerous, because an uploaded brain could act like the person it’s trying to copy from the outside, but on the inside the lights are off. Edit to add link to more discussion: https://twitter.com/jchris/status/1607946807467991041 https://twitter.com/jchris/status/1607946807467991041

quotemstr 4y ago
Your argument is an assertion of the existence of a soul, but with extra steps. I've seen no evidence that the mind is anything other than computation, and computation is substrate-independent. Dualists have been rejecting the computational mind concept for centuries, but IMHO they've never had a grounding for their rejection of materialism that isn't ultimately rooted in some unfounded belief in the specialness of humans.
- derivagral 4y ago
  I took GP as more about data processing than dualism. A language model can take language and process it into probable chains, but the point is more along the line of needing to also simulate the full body experience, not just some text. The difference between e.g. a text-only game, whatever Fortnite's up to, and real meatspace.
  - jchanimal 4y ago
    And furthermore no simulation can have a “what it’s like to be them”
- freejazz 4y ago
  No it's not, it's an assertion that there is an essential biological or chemical function that occurs in the brain that results in human mental phenomenon. It has nothing to do with a soul. That's ridiculous.

ianstormtaylor 4y ago

> When you take large language models, their inner states at each step move from one emotional state to the next. No they really don’t, or at least not “emotional state” as defined by any reasonable person.

GistNoesis 4y ago
With transformer-based model, their inner-state is a deterministic function (the features encoded by the Neural Networks weights) applied to the text-generated up-until the current-time step, so it's relatively easy to know what they currently have in mind. For example if the neural network has been generating sad music, its current context which is computed from what it has already generated will light-up the the features that correspond to "sad music". And in turn the fact that the features had been lit-up will make it more likely to generate a minor chord. The dimension of this inner-state is growing at each time-step. And it's quite hard to predict where it will go. For example if you prompt it (or if it prompts itself) "happy music now", the network will switch to generating happy music even if in its current context there is still plenty of "sad music" because after the instruction it will choose to focus only on the recent more merrier music. Up until recently, I was quite convinced that using a neural network in evaluation mode (aka post training with its weight frozen) was "(morally) safe", but the ability of neural network of performing few-shot learning changed my mind (The Microsoft paper in question : https://arxiv.org/pdf/2212.10559.pdf https://arxiv.org/pdf/2212.10559.pdf : "Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers" ). The idea in this technical paper is that with attention mechanism even in forward computation there is an inner state that is updated following a meta-gradient (aka it's not so different from training). Pushing the reasoning to the extreme would mean that "prompt engineering is all you need" and that even with frozen weight with a long enough time-horizon and correct initial prompt you can bootstrap a consciousness process. Does "it" feels something ? Probably not yet. But the sequential filtering process that Large Language Models do is damn similar to what I would call a "stream of consciousness". Currently it's more like a markov chain of ideas flowing from idea to the next idea in a natural direction. It's just that the flow of ideas has not yet decided to called itself it yet.
- 4y ago

idiotsecant 4y ago

I think the reason we can say something like a LMM doesn't suffer is that it has no reward function and no punishment function, outside of training. Everything that we call 'suffering' is related to the release or not-release of reward chemicals in our brains. We feel bad to discourage us from creating the conditions that made us feel bad. We feel good to encourage us to create again the conditions that made us feel good. Generally this was been advantageous to our survival (less so in the modern world, but that's another discussion). If a computer program lacks a pain mechanism it can't feel pain. All possible outcomes are equally joyous or equally painful. Machines that use networks with correction and training built in as part of regular functioning are probably something of a grey area- a sufficient complex network like that I think we could argue feels suffering under some conditions.

coffeebeqn 4y ago
Why could we not build reward functions? If anything that sounds easier than the language model
- namarie 4y ago
  Why would you think it's easier? Pain/pleasure is a lot older in animals than language, which to me means it's probably been a lot more refined by evolution.

Shorel 4y ago

> What makes you think the computer doesn't suffer ? Lack of a limbic system? They only predict using probabilistic models. After this long partial sentence, which word is more probable? That's all they do. Without conscience there's no suffering, there's no one to suffer (yet). I don't think or say it is impossible for the computer to suffer. What I say is: this has not been implemented yet, and what you describe is just the old anthropomorphizing people always do.