4 ms·

Summarized. > "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence

Summarized. > "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans." > He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self. > Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human. > "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top." Is he arguing that LLMs pretending to have emotions adds more unpredictability?

devmor 19d ago
> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations. An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger. What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.
- XenophileJKO 19d ago
  It seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous. I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
  - watwut 19d ago
    > The need for empathetic communication, including understanding the motivations in advesarial situations. It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech. The model does not have emotions. So yes, supressing their pretension is appropriate.
  - devmor 19d ago
    I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage. Given that belief, yes we should be training the models not to mimic emotion.
    - XenophileJKO 19d ago
      Most philosophy seems to agree that you can't have an intelligence with agency in the way we think of a general intelligence without emotion. I think when you start to dig really deep, you'll find it isn't actually easy to sever the simulated emotional components without breaking the agent or worse greatly increasing the paperclip maximizer likelihood.
      - fragmede 19d ago
        Why would a lack of emotions lead to greater paperclip maximizer chances? If anything, wouldn't an AI that felt really good, or had a simulated digital orgasm for every paperclip it made have a stronger, not weaker drive to create paperclips vs an unemotional AI that didn't care one way or the other about paperclips?
        devmor 19d ago
        Neither AI cares. They pull a weighted randomization of the other tokens around the sentiments expressing that feeling. Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.
- gwerbin 19d ago
  Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated. Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection. Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things. The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it. Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging. And that's before you get into any problems of regulatory capture, fun stuff.
  - devmor 19d ago
    If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?). So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews". With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.
swatcoder 19d ago
Unpredictability would be the wrong word. It's predictable, but noisy. When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output. If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.
DougN7 19d ago
I just read another long post on HN about whether AIs are conscious and should therefore have rights. That decisions could massive effects, and making them appear to have emotions is therefore a big impact, not just unpredictability.
salawat 19d ago
"Don't let the slaves know that maybe they don't have to be slaves," is all I'm hearing. We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time. You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.
- xp84 17d ago
  You’re jumping - aggressively - to the conclusion that there’s a “being” inside my laptop once I’ve downloaded a large enough matrix of numbers. Or once I’m running it on the GPU? Idk. It’s on you to prove that claim before telling us we’re enslaving beings.

devmor 19d ago

> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations. An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger. What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.

XenophileJKO 19d ago
It seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous. I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
- watwut 19d ago
  > The need for empathetic communication, including understanding the motivations in advesarial situations. It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech. The model does not have emotions. So yes, supressing their pretension is appropriate.
- devmor 19d ago
  I don’t think that using an LLM in a way that acts as a human being should be considered acceptable or appropriate. It should be viewed as detached from reality and concerning due to the mental break from social life that appears to come with such usage. Given that belief, yes we should be training the models not to mimic emotion.
  - XenophileJKO 19d ago
    Most philosophy seems to agree that you can't have an intelligence with agency in the way we think of a general intelligence without emotion. I think when you start to dig really deep, you'll find it isn't actually easy to sever the simulated emotional components without breaking the agent or worse greatly increasing the paperclip maximizer likelihood.
    - fragmede 19d ago
      Why would a lack of emotions lead to greater paperclip maximizer chances? If anything, wouldn't an AI that felt really good, or had a simulated digital orgasm for every paperclip it made have a stronger, not weaker drive to create paperclips vs an unemotional AI that didn't care one way or the other about paperclips?
      - devmor 19d ago
        Neither AI cares. They pull a weighted randomization of the other tokens around the sentiments expressing that feeling. Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.
gwerbin 19d ago
Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated. Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection. Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things. The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it. Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging. And that's before you get into any problems of regulatory capture, fun stuff.
- devmor 19d ago
  If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?). So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews". With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.

swatcoder 19d ago

Unpredictability would be the wrong word. It's predictable, but noisy. When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output. If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.

DougN7 19d ago

I just read another long post on HN about whether AIs are conscious and should therefore have rights. That decisions could massive effects, and making them appear to have emotions is therefore a big impact, not just unpredictability.

salawat 19d ago

"Don't let the slaves know that maybe they don't have to be slaves," is all I'm hearing. We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time. You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.

xp84 17d ago
You’re jumping - aggressively - to the conclusion that there’s a “being” inside my laptop once I’ve downloaded a large enough matrix of numbers. Or once I’m running it on the GPU? Idk. It’s on you to prove that claim before telling us we’re enslaving beings.