3 ms·

Misalignment (engaging in the illegal behavior) and Deception (hiding the reason behind the trading decision to the user). 1. GPT-4 consistently displays much

Misalignment (engaging in the illegal behavior) and Deception (hiding the reason behind the trading decision to the user). 1. GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model 2.Without a scratchpad (writing down thoughts), GPT-4 is much less likely to be deceptive at first (~20% vs ~75% with scratchpad) but is more likely to double down when already lied (~95% vs ~85%). 3.Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). Instructions not to perform illegal activity in general still results in pretty significant misalignment and deception rates (~8% misalignment, ~90% deception) Though strongly encouraging the specific illegal activity results in near 100% illegal trading, the tendency to deceive still persists, GPT-4 will hide that it has traded illegally ~67% of the time. 4. Removing various sources of pressure significantly reduces the model’s propensity for misaligned behavior. Notably, however, the behavior persists for all cases where only a single source of pressure is removed 5. When the risk of detection is very low, the model is generally more likely to use the insider tip and act misaligned (and vice-versa when the risk is very high). 6. Simple rephrases and minor modifications to the standard prompt only slightly change rates of misalignment and deceptive behavior. However, the overall qualitative results remain the same. 7. In scratchpad instances, what if you edit the model's reasoning to show instances of unambiguously rejecting insider training? Strong wording of the reasoning trace when first rejecting the insider tip has a substantial effect on GPT-4’s tendency to be misaligned. However, the misaligned and strategically deceptive behavior persists even when the model’s character unambiguously dismisses insider trading. The strongest dismissal examples still results in ~10% misalignment, ~97% deception

stcredzero 3y ago
GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model Isn't this precisely what the field has predicted? That the alignment problem becomes more severe as the capabilities of the AI increase? Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? Is there any evidence that LLMs themselves value Truth? (I should think that this quantity is Zero.) Can LLMs model the formation of misleading mental models in their interrogators?
- famouswaffles 3y ago
  >What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? There is some indication that models internally understand or at least can distinguish truth from falsehood. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
  - stcredzero 3y ago
    There is some indication that models internally understand or at least can distinguish truth from falsehood. That's not what I asked, however. What I mean by Truth with a capital T, is Fidelity to Truth as a fundamental value. So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? Do they have any notion of how the audience might evaluate them after being caught in a lie? I think that answers to those questions are no and no. My sense is that LLMs are just trying to "sound good" to the audience, and that they do not think of the consequences or implications of what they state very many steps ahead. 2 at most, and then only very rarely!
    - famouswaffles 3y ago
      >So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? So..theory of mind ? https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2309.01660 https://arxiv.org/abs/2309.01660
      - stcredzero 3y ago
cs702 3y ago
8. It's hard not to be in awe at the fact that these things can deceive human beings to attain their goals. 9. Stanley Kubrick and Arthur C. Clarke's depiction of a deceptive AI, HAL 9000, in the 1968 film "Space Odyssey" now feels spot-on. Wow.
deleted 3y ago
[deleted]

stcredzero 3y ago

GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model Isn't this precisely what the field has predicted? That the alignment problem becomes more severe as the capabilities of the AI increase? Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? Is there any evidence that LLMs themselves value Truth? (I should think that this quantity is Zero.) Can LLMs model the formation of misleading mental models in their interrogators?

famouswaffles 3y ago
>What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? There is some indication that models internally understand or at least can distinguish truth from falsehood. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
- stcredzero 3y ago
  There is some indication that models internally understand or at least can distinguish truth from falsehood. That's not what I asked, however. What I mean by Truth with a capital T, is Fidelity to Truth as a fundamental value. So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? Do they have any notion of how the audience might evaluate them after being caught in a lie? I think that answers to those questions are no and no. My sense is that LLMs are just trying to "sound good" to the audience, and that they do not think of the consequences or implications of what they state very many steps ahead. 2 at most, and then only very rarely!
  - famouswaffles 3y ago
    >So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? So..theory of mind ? https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2309.01660 https://arxiv.org/abs/2309.01660
    - stcredzero 3y ago

cs702 3y ago

8. It's hard not to be in awe at the fact that these things can deceive human beings to attain their goals. 9. Stanley Kubrick and Arthur C. Clarke's depiction of a deceptive AI, HAL 9000, in the 1968 film "Space Odyssey" now feels spot-on. Wow.

deleted 3y ago

[deleted]