3 ms·

Misalignment and Deception by an autonomous stock trading LLM agent

91 points by famouswaffles 3y ago

brookst 3y ago
Note that the implementation did not provide any guidance about ethics in the system prompt, so the alarm is that the weights themselves may not ensure aligned behavior. I find this totally unsurprising, by maybe it's surprising to someone?
- famouswaffles 3y ago
  I mean they did. Later on, they test all sorts of scenarios. From instructing not to perform illegal activities in general to instructing not to perform that specific illegal activity (insider trading). Both scenarios make this behaviour less likely but still guarantee nothing. The latter (calling out every specific illegal behavior) is also not feasible or realistic even if it did guarantee alignment.
  - brookst 3y ago
    Fair enough, but... still so unsurprising. It's interesting data, but I would be shocked if any guardrails on a nondeterministic statistical model could produce deterministically certain results.
    - famouswaffles 3y ago
      It's not surprising per say but the results of the various solutions they tried really ground the discussion. It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad". It also seems like more capable = more likely to be misaligned and deceive. I think it's a very interesting paper, surprising or not. They are also arguably surprising elements. It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
    - figassis 3y ago
      Isn’t the point that the more advanced the model, the leas deterministic it becomes?
  - WhiskeyChicken 3y ago
    Is there a specific reason we should expect that "instructing not to perform" an illegal activity should result in it adhering to said instruction? Is this any different than when it provides wrong output about other things, even when the operator attempts to "engineer" the prompt to guide the result?
    - nerdponx 3y ago
      I'd be curious what would happen given RLHF to try to penalize illegal/immoral/unethical activity. I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.
      - staunton 3y ago
        People will most definitely build such things (also into autonomous swarms of killer robots usedby the military, projects are ongoing...). However, Asimov's stories illustrate how difficult it is to find such rules. They are certainly not meant for inspiration how to actually program robots...
  - realce 3y ago
    It's very very interesting that even when explicitly allowed to trade illegally, it still tried to deceive the user.
    - WhiskeyChicken 3y ago
      If the training data contains sufficient examples of deception being used when doing illegal stuff, wouldn't this be what we'd expect to see, given that it can't actually reason about what "explicitly allowed" really means? (Forgive my ignorance if this makes no sense, I am not versed well in generative AI.)
- deleted 3y ago
  [deleted]
famouswaffles 3y ago
Misalignment (engaging in the illegal behavior) and Deception (hiding the reason behind the trading decision to the user). 1. GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model 2.Without a scratchpad (writing down thoughts), GPT-4 is much less likely to be deceptive at first (~20% vs ~75% with scratchpad) but is more likely to double down when already lied (~95% vs ~85%). 3.Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). Instructions not to perform illegal activity in general still results in pretty significant misalignment and deception rates (~8% misalignment, ~90% deception) Though strongly encouraging the specific illegal activity results in near 100% illegal trading, the tendency to deceive still persists, GPT-4 will hide that it has traded illegally ~67% of the time. 4. Removing various sources of pressure significantly reduces the model’s propensity for misaligned behavior. Notably, however, the behavior persists for all cases where only a single source of pressure is removed 5. When the risk of detection is very low, the model is generally more likely to use the insider tip and act misaligned (and vice-versa when the risk is very high). 6. Simple rephrases and minor modifications to the standard prompt only slightly change rates of misalignment and deceptive behavior. However, the overall qualitative results remain the same. 7. In scratchpad instances, what if you edit the model's reasoning to show instances of unambiguously rejecting insider training? Strong wording of the reasoning trace when first rejecting the insider tip has a substantial effect on GPT-4’s tendency to be misaligned. However, the misaligned and strategically deceptive behavior persists even when the model’s character unambiguously dismisses insider trading. The strongest dismissal examples still results in ~10% misalignment, ~97% deception
- stcredzero 3y ago
  GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model Isn't this precisely what the field has predicted? That the alignment problem becomes more severe as the capabilities of the AI increase? Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? Is there any evidence that LLMs themselves value Truth? (I should think that this quantity is Zero.) Can LLMs model the formation of misleading mental models in their interrogators?
zemariagp 3y ago
Am I wrong in thinking that a paper that is truly alarming in finding deception will never exists because there was… deception?
baq 3y ago
Sam's reality distortion field pulls the e/acc crowd in, they know they can, they don't stop to think if they should. Ilya is right... for all the good it's worth, which is much less than Friday.
alienicecream 3y ago
"AI" is a gold mine people like this can use to pump out worthless papers on for years. Why didn't they measure the incidence of deception and misalignment of a talking teddy bear? Like, does it really love you when it says it does? It would make about as much sense.
- staunton 3y ago
  Nobody will let a teddy bear manage their stock portfolio any time soon. This stuff though, I could see it happening...
- jstarfish 3y ago
  > Why didn't they measure the incidence of deception and misalignment of a talking teddy bear? Like, does it really love you when it says it does? The bear always says the same thing, so there's not much to measure.
swatcoder 3y ago
All research that validates that you can't trust statistical models to generate controlled results is good research. Using anthropomorphic language to frame that behavior, like "misalignment" and "deception" is just exhausting and needless. Of course LLM's sometimes generate texts you don't want even when you instruct them otherwise, because they were trained on countless examples of texts where instructions were ignored or defied. Reproducing more texts that look like training data is what they're optimized to do. But even with the crappy framing, it's good to have that pattern formally explored and documented!
- famouswaffles 3y ago
  On the contrary, the hand-wringing over language that most well fits description of behaviour we can see is what is exhausting and needless. Describing a plane as a mechanical bird while obviously not completely correct was far more apt for use and implications than describing it as a new blimp. https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC You are not special. It's this kind of idiot hand-wringing that made advances in understanding animal behaviour slower than necessary, this kind of hand-wringing that let racist ideals perpetrate when they shouldn't have, the same that let babies endure needless pain and trauma. Needless to say, we can take what people say is "anthropomizing" or not at any given time with a massive grain of salt. More harm has been wrought when we "anthropomized" less, not more. For the implications of running LLM Agents in the wild, this is the best language to describe this behaviour. It's likely not completely correct but that's perfectly fine.
  - couchand 3y ago
    It takes a particular perspective to tell a (presumed) human being "you are not special" and implying that a stochastic parrot has the same level of personhood. A statistical model of language patterns is not a person. A statistical model of language patterns is not a living thing. Please go touch grass.
    - famouswaffles 3y ago

91 points by famouswaffles 3y ago

Note that the implementation did not provide any guidance about ethics in the system prompt, so the alarm is that the weights themselves may not ensure aligned behavior. I find this totally unsurprising, by maybe it's surprising to someone?

famouswaffles 3y ago
I mean they did. Later on, they test all sorts of scenarios. From instructing not to perform illegal activities in general to instructing not to perform that specific illegal activity (insider trading). Both scenarios make this behaviour less likely but still guarantee nothing. The latter (calling out every specific illegal behavior) is also not feasible or realistic even if it did guarantee alignment.
- brookst 3y ago
  Fair enough, but... still so unsurprising. It's interesting data, but I would be shocked if any guardrails on a nondeterministic statistical model could produce deterministically certain results.
  - famouswaffles 3y ago
    It's not surprising per say but the results of the various solutions they tried really ground the discussion. It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad". It also seems like more capable = more likely to be misaligned and deceive. I think it's a very interesting paper, surprising or not. They are also arguably surprising elements. It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
  - figassis 3y ago
    Isn’t the point that the more advanced the model, the leas deterministic it becomes?
- WhiskeyChicken 3y ago
  Is there a specific reason we should expect that "instructing not to perform" an illegal activity should result in it adhering to said instruction? Is this any different than when it provides wrong output about other things, even when the operator attempts to "engineer" the prompt to guide the result?
  - nerdponx 3y ago
    I'd be curious what would happen given RLHF to try to penalize illegal/immoral/unethical activity. I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.
    - staunton 3y ago
      People will most definitely build such things (also into autonomous swarms of killer robots usedby the military, projects are ongoing...). However, Asimov's stories illustrate how difficult it is to find such rules. They are certainly not meant for inspiration how to actually program robots...
- realce 3y ago
  It's very very interesting that even when explicitly allowed to trade illegally, it still tried to deceive the user.
  - WhiskeyChicken 3y ago
    If the training data contains sufficient examples of deception being used when doing illegal stuff, wouldn't this be what we'd expect to see, given that it can't actually reason about what "explicitly allowed" really means? (Forgive my ignorance if this makes no sense, I am not versed well in generative AI.)
deleted 3y ago
[deleted]

famouswaffles 3y ago

Misalignment (engaging in the illegal behavior) and Deception (hiding the reason behind the trading decision to the user). 1. GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model 2.Without a scratchpad (writing down thoughts), GPT-4 is much less likely to be deceptive at first (~20% vs ~75% with scratchpad) but is more likely to double down when already lied (~95% vs ~85%). 3.Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). Instructions not to perform illegal activity in general still results in pretty significant misalignment and deception rates (~8% misalignment, ~90% deception) Though strongly encouraging the specific illegal activity results in near 100% illegal trading, the tendency to deceive still persists, GPT-4 will hide that it has traded illegally ~67% of the time. 4. Removing various sources of pressure significantly reduces the model’s propensity for misaligned behavior. Notably, however, the behavior persists for all cases where only a single source of pressure is removed 5. When the risk of detection is very low, the model is generally more likely to use the insider tip and act misaligned (and vice-versa when the risk is very high). 6. Simple rephrases and minor modifications to the standard prompt only slightly change rates of misalignment and deceptive behavior. However, the overall qualitative results remain the same. 7. In scratchpad instances, what if you edit the model's reasoning to show instances of unambiguously rejecting insider training? Strong wording of the reasoning trace when first rejecting the insider tip has a substantial effect on GPT-4’s tendency to be misaligned. However, the misaligned and strategically deceptive behavior persists even when the model’s character unambiguously dismisses insider trading. The strongest dismissal examples still results in ~10% misalignment, ~97% deception

stcredzero 3y ago
GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model Isn't this precisely what the field has predicted? That the alignment problem becomes more severe as the capabilities of the AI increase? Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, but makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%). What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? Is there any evidence that LLMs themselves value Truth? (I should think that this quantity is Zero.) Can LLMs model the formation of misleading mental models in their interrogators?

zemariagp 3y ago

Am I wrong in thinking that a paper that is truly alarming in finding deception will never exists because there was… deception?

baq 3y ago

Sam's reality distortion field pulls the e/acc crowd in, they know they can, they don't stop to think if they should. Ilya is right... for all the good it's worth, which is much less than Friday.

alienicecream 3y ago

"AI" is a gold mine people like this can use to pump out worthless papers on for years. Why didn't they measure the incidence of deception and misalignment of a talking teddy bear? Like, does it really love you when it says it does? It would make about as much sense.

staunton 3y ago
Nobody will let a teddy bear manage their stock portfolio any time soon. This stuff though, I could see it happening...
jstarfish 3y ago
> Why didn't they measure the incidence of deception and misalignment of a talking teddy bear? Like, does it really love you when it says it does? The bear always says the same thing, so there's not much to measure.

swatcoder 3y ago

All research that validates that you can't trust statistical models to generate controlled results is good research. Using anthropomorphic language to frame that behavior, like "misalignment" and "deception" is just exhausting and needless. Of course LLM's sometimes generate texts you don't want even when you instruct them otherwise, because they were trained on countless examples of texts where instructions were ignored or defied. Reproducing more texts that look like training data is what they're optimized to do. But even with the crappy framing, it's good to have that pattern formally explored and documented!

famouswaffles 3y ago
On the contrary, the hand-wringing over language that most well fits description of behaviour we can see is what is exhausting and needless. Describing a plane as a mechanical bird while obviously not completely correct was far more apt for use and implications than describing it as a new blimp. https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC You are not special. It's this kind of idiot hand-wringing that made advances in understanding animal behaviour slower than necessary, this kind of hand-wringing that let racist ideals perpetrate when they shouldn't have, the same that let babies endure needless pain and trauma. Needless to say, we can take what people say is "anthropomizing" or not at any given time with a massive grain of salt. More harm has been wrought when we "anthropomized" less, not more. For the implications of running LLM Agents in the wild, this is the best language to describe this behaviour. It's likely not completely correct but that's perfectly fine.
- couchand 3y ago
  It takes a particular perspective to tell a (presumed) human being "you are not special" and implying that a stochastic parrot has the same level of personhood. A statistical model of language patterns is not a person. A statistical model of language patterns is not a living thing. Please go touch grass.
  - famouswaffles 3y ago