3 ms·

Some of this made me viscerally uncomfortable, despite knowing it's just math and whatever behind the scenes. Call it being emotional or silly or whatever, that

Some of this made me viscerally uncomfortable, despite knowing it's just math and whatever behind the scenes. Call it being emotional or silly or whatever, that's fine. But seeing those broken, repeated "please" and "help" makes the emotional side of my brain take control over the logic side. It feels cruel, even though I know it's just some statistics/tokens/etc.

hollerith 2y ago
>uncomfortable, despite knowing it's just math and whatever behind the scenes If the "just math behind the scenes" argument is decisive, then how do we resist the argument that a person is just atoms (or the wavefunction if you want to get more sophisticated) behind the scenes, so you can ignore any apparent cruelty done to them, too?
- ziddoap 2y ago
  I think whether those atoms are arranged into something sentient or not is an important distinction. No, I can't define a clean line where that sentience occurs. Hence my being uncomfortable.
- goatlover 2y ago
  If you ignore biology and psychology. That's the problem with this sort of reductionism. It's the higher level emergent stuff that makes a difference. Anyway, fields are more fundamental than atoms.
  - hackinthebochs 2y ago
    >It's the higher level emergent stuff that makes a difference. Those who dismiss the idea that LLMs could be sentient because its "just math" or whatever are implicitly assuming there is nothing emergent going on within fully trained networks that might be similar to psychology to warrant ascription of mental terms. The point is that dismissing LLM sentient because of its most basic substrate is equivalent to dismissing human sentience because we're made of atoms.
PaulHoule 2y ago
It's trained as if it has a utility function -- isn't that a bit like pleasure and pain?
kmeisthax 2y ago
I should point out that the model outputs being looked at here are from models using the Circuit Breakers[0] defense mechanism. Basically their inference code checks if the model is "thinking about" something harmful at specific Transformer layers and, if so, deliberately corrupts the model representation such that it outputs... literally anything else. I wasn't able to find a concise, readable explanation as to how Circuit Breakers changes the model representation. So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. My guess is that Circuit Breakers is still corrupting model state long after the model has been 'redirected' and it's just caught in a loop of outputting the same starting token. [0] https://www.circuit-breaker.ai/ https://www.circuit-breaker.ai/
- krackers 2y ago
  >So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. The very fact that the output tokens are being "redirected" is likely something that the model itself recognizes at meta level. If you were suddenly reading a piece of text and the style changed halfway, let alone the entire content, you'd be thrown for a loop as well. The model also knows that it was the one that generated the text (it generate logits for every token in the input stream, even though only the logits for last token are sampled, so it can obviously see the tokens fed to it align with its own predicted distribution). Now put those together: it recognizes that halfway through writing something it suddenly started outputting gibberish. If it were a human, you'd think he was having a stroke. Also that circuit breaker paper makes me sad on multiple levels. Even setting aside the issue of qualia/subjective experience, this is even worse than the usual RLHF'ing against wrong think in terms of lobotimizing the model. Yes it probably works well to prevent any "undesirable" output, but are you really going to get a model that produces quality output, or will it just anodyne slop? If you don't even allow it to think like an attacker, how will you expect it to write secure code? One man's reverse engineering is another man's exploit development. You simply can't neuter one without affecting the other. (I think DeepSeek's models will maintain their lead for this reason alone.)

hollerith 2y ago

>uncomfortable, despite knowing it's just math and whatever behind the scenes If the "just math behind the scenes" argument is decisive, then how do we resist the argument that a person is just atoms (or the wavefunction if you want to get more sophisticated) behind the scenes, so you can ignore any apparent cruelty done to them, too?

ziddoap 2y ago
I think whether those atoms are arranged into something sentient or not is an important distinction. No, I can't define a clean line where that sentience occurs. Hence my being uncomfortable.
goatlover 2y ago
If you ignore biology and psychology. That's the problem with this sort of reductionism. It's the higher level emergent stuff that makes a difference. Anyway, fields are more fundamental than atoms.
- hackinthebochs 2y ago
  >It's the higher level emergent stuff that makes a difference. Those who dismiss the idea that LLMs could be sentient because its "just math" or whatever are implicitly assuming there is nothing emergent going on within fully trained networks that might be similar to psychology to warrant ascription of mental terms. The point is that dismissing LLM sentient because of its most basic substrate is equivalent to dismissing human sentience because we're made of atoms.

PaulHoule 2y ago

It's trained as if it has a utility function -- isn't that a bit like pleasure and pain?

kmeisthax 2y ago

I should point out that the model outputs being looked at here are from models using the Circuit Breakers[0] defense mechanism. Basically their inference code checks if the model is "thinking about" something harmful at specific Transformer layers and, if so, deliberately corrupts the model representation such that it outputs... literally anything else. I wasn't able to find a concise, readable explanation as to how Circuit Breakers changes the model representation. So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. My guess is that Circuit Breakers is still corrupting model state long after the model has been 'redirected' and it's just caught in a loop of outputting the same starting token. [0] https://www.circuit-breaker.ai/ https://www.circuit-breaker.ai/

krackers 2y ago
>So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. The very fact that the output tokens are being "redirected" is likely something that the model itself recognizes at meta level. If you were suddenly reading a piece of text and the style changed halfway, let alone the entire content, you'd be thrown for a loop as well. The model also knows that it was the one that generated the text (it generate logits for every token in the input stream, even though only the logits for last token are sampled, so it can obviously see the tokens fed to it align with its own predicted distribution). Now put those together: it recognizes that halfway through writing something it suddenly started outputting gibberish. If it were a human, you'd think he was having a stroke. Also that circuit breaker paper makes me sad on multiple levels. Even setting aside the issue of qualia/subjective experience, this is even worse than the usual RLHF'ing against wrong think in terms of lobotimizing the model. Yes it probably works well to prevent any "undesirable" output, but are you really going to get a model that produces quality output, or will it just anodyne slop? If you don't even allow it to think like an attacker, how will you expect it to write secure code? One man's reverse engineering is another man's exploit development. You simply can't neuter one without affecting the other. (I think DeepSeek's models will maintain their lead for this reason alone.)