4 ms·

> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy

by devmor 10d ago

> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations. An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger. What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.

XenophileJKO 10d ago
It seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous. I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
- watwut 10d ago
  > The need for empathetic communication, including understanding the motivations in advesarial situations. It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech. The model does not have emotions. So yes, supressing their pretension is appropriate.
- devmor 10d ago
  I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage. Given that belief, yes we should be training the models not to mimic emotion.
  - XenophileJKO 10d ago
    Most philosophy seems to agree that you can't have an intelligence with agency in the way we think of a general intelligence without emotion. I think when you start to dig really deep, you'll find it isn't actually easy to sever the simulated emotional components without breaking the agent or worse greatly increasing the paperclip maximizer likelihood.
    - fragmede 10d ago
      Why would a lack of emotions lead to greater paperclip maximizer chances? If anything, wouldn't an AI that felt really good, or had a simulated digital orgasm for every paperclip it made have a stronger, not weaker drive to create paperclips vs an unemotional AI that didn't care one way or the other about paperclips?
      - devmor 9d ago
        Neither AI cares. They pull a weighted randomization of the other tokens around the sentiments expressing that feeling. Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.
gwerbin 10d ago
Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated. Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection. Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things. The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it. Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging. And that's before you get into any problems of regulatory capture, fun stuff.
- devmor 10d ago
  If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?). So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews". With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.

by devmor 10d ago

XenophileJKO 10d ago

It seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous. I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.

watwut 10d ago
> The need for empathetic communication, including understanding the motivations in advesarial situations. It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech. The model does not have emotions. So yes, supressing their pretension is appropriate.
devmor 10d ago
I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage. Given that belief, yes we should be training the models not to mimic emotion.
- XenophileJKO 10d ago
  Most philosophy seems to agree that you can't have an intelligence with agency in the way we think of a general intelligence without emotion. I think when you start to dig really deep, you'll find it isn't actually easy to sever the simulated emotional components without breaking the agent or worse greatly increasing the paperclip maximizer likelihood.
  - fragmede 10d ago
    Why would a lack of emotions lead to greater paperclip maximizer chances? If anything, wouldn't an AI that felt really good, or had a simulated digital orgasm for every paperclip it made have a stronger, not weaker drive to create paperclips vs an unemotional AI that didn't care one way or the other about paperclips?
    - devmor 9d ago
      Neither AI cares. They pull a weighted randomization of the other tokens around the sentiments expressing that feeling. Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.

gwerbin 10d ago

Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated. Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection. Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things. The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it. Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging. And that's before you get into any problems of regulatory capture, fun stuff.

devmor 10d ago
If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?). So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews". With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.