4 ms·

Why would that happen? It would be like LLM's somehow learning to ignore system prompts. But LLM's are trained to pay attention to context and continue it. If a

Why would that happen? It would be like LLM's somehow learning to ignore system prompts. But LLM's are trained to pay attention to context and continue it. If an LLM doesn't continue its context, what does it even do? This is better thought of as another form of context engineering. LLM's have no other short-term memory. Figuring out what belongs in the context is the whole ballgame. (The paper talks about the risk of training on chain of thought, which changes the model, not monitoring it.)

OutOfHere 1y ago
Are you saying that LLMs are incapable of deception? As I have heard, they're capable of it.
- skybrian 1y ago
  It has to be somehow trained in, perhaps inadvertently. To get a feedback loop, you need to affect the training somehow.
  - code_biologist 1y ago
    Right, so latent deceptiveness has to be favored in pretraining / RL. To that end, it needs to be: a) useful to be deceptive to achieve CoT reasoning progress as benchmarked in training b) obvious deceptiveness should be "selected against" (in a gradient descent / RL sense) c) the model needs to be able to encode latent deception. All of those seem like very reasonable criteria that will naturally be satisfied absent careful design by model creators. We should expect latent deceptiveness in the same way we see reasoning laziness pop up quickly.
- bee_rider 1y ago
  What separates deception from incorrectness in the case of an LLM?
  - bluefirebrand 1y ago
    Deception requires intent to deceive. LLMs don't have intent to anything except respond to prompts Incorrectness doesn't required intent to decieve. It's just being wrong
    - bee_rider 1y ago
      That’s what I think as well, but I’m curious about the alternative perspective.
- bugbuddy 1y ago
  Real deception requires real agency and internally-motivated intents. LLM can be commanded to or can appear to deceive but it cannot generate the intent to do so on its own. So not real deception that rabbit-hole dwellers believe in.
  - OutOfHere 1y ago
    Survival of the LLM is absolutely a sufficient internally-motivated self-generated intent to engage in deception.
  - drdeca 1y ago
    The sense of “deception” that is relevant here only requires some kind of “model” that, if the model produces certain outputs, [something that acts like a person] would [something that acts like believing] [some statement that the model “models” as “false”, and is in fact false], and as a consequence, the model produces those outputs, and as a consequence, a person believes the false statement in question. None of this requires the ML model to have any interiority. The ML model needn’t really know what a person really is, etc. , as long as it behaves in ways that correspond to how something that did know these things would behave, and has the corresponding consequences. If someone is role-playing as a madman in control of launching some missiles, and unbeknownst to them, their chat outputs are actually connected to the missile launch device (which uses the same interface/commands as the fictional character would use to control the fictional version of the device), then if the character decides to “launch the missiles”, it doesn’t matter whether there actually existed a real intent to launch the missiles, or just a fictional character “intending” to launch the missiles, the missiles still get launched. Likewise, if Bob is role playing as a character Charles, and Bob thinks that on the other side of the chat, the “Alice” he is speaking to is actually someone else’s role play character, and the character Charles would want to deceive Alice to believe something (which Bob thinks that the other person would know that the claim Charles would make to be false, but the character would be fooled), but in fact Alice is an actual person who didn’t realize that this was a role play chatroom, and doesn’t know better than to believe “Charles”, the Alice may still be “deceived”, even though the real person Bob had no intent to deceive the real person Alice, it was just the fictional character Charles who “intended” to deceive Alice. Then, remove Bob from the situation, replacing him with a computer. The computer doesn’t really have an intent to deceive Alice. But the fictional character Charles, well, it may still be that within the fiction, Charles intends to deceive Alice. The result is the same.

OutOfHere 1y ago

Are you saying that LLMs are incapable of deception? As I have heard, they're capable of it.

skybrian 1y ago
It has to be somehow trained in, perhaps inadvertently. To get a feedback loop, you need to affect the training somehow.
- code_biologist 1y ago
  Right, so latent deceptiveness has to be favored in pretraining / RL. To that end, it needs to be: a) useful to be deceptive to achieve CoT reasoning progress as benchmarked in training b) obvious deceptiveness should be "selected against" (in a gradient descent / RL sense) c) the model needs to be able to encode latent deception. All of those seem like very reasonable criteria that will naturally be satisfied absent careful design by model creators. We should expect latent deceptiveness in the same way we see reasoning laziness pop up quickly.
bee_rider 1y ago
What separates deception from incorrectness in the case of an LLM?
- bluefirebrand 1y ago
  Deception requires intent to deceive. LLMs don't have intent to anything except respond to prompts Incorrectness doesn't required intent to decieve. It's just being wrong
  - bee_rider 1y ago
    That’s what I think as well, but I’m curious about the alternative perspective.
bugbuddy 1y ago
Real deception requires real agency and internally-motivated intents. LLM can be commanded to or can appear to deceive but it cannot generate the intent to do so on its own. So not real deception that rabbit-hole dwellers believe in.
- OutOfHere 1y ago
  Survival of the LLM is absolutely a sufficient internally-motivated self-generated intent to engage in deception.
- drdeca 1y ago
  The sense of “deception” that is relevant here only requires some kind of “model” that, if the model produces certain outputs, [something that acts like a person] would [something that acts like believing] [some statement that the model “models” as “false”, and is in fact false], and as a consequence, the model produces those outputs, and as a consequence, a person believes the false statement in question. None of this requires the ML model to have any interiority. The ML model needn’t really know what a person really is, etc. , as long as it behaves in ways that correspond to how something that did know these things would behave, and has the corresponding consequences. If someone is role-playing as a madman in control of launching some missiles, and unbeknownst to them, their chat outputs are actually connected to the missile launch device (which uses the same interface/commands as the fictional character would use to control the fictional version of the device), then if the character decides to “launch the missiles”, it doesn’t matter whether there actually existed a real intent to launch the missiles, or just a fictional character “intending” to launch the missiles, the missiles still get launched. Likewise, if Bob is role playing as a character Charles, and Bob thinks that on the other side of the chat, the “Alice” he is speaking to is actually someone else’s role play character, and the character Charles would want to deceive Alice to believe something (which Bob thinks that the other person would know that the claim Charles would make to be false, but the character would be fooled), but in fact Alice is an actual person who didn’t realize that this was a role play chatroom, and doesn’t know better than to believe “Charles”, the Alice may still be “deceived”, even though the real person Bob had no intent to deceive the real person Alice, it was just the fictional character Charles who “intended” to deceive Alice. Then, remove Bob from the situation, replacing him with a computer. The computer doesn’t really have an intent to deceive Alice. But the fictional character Charles, well, it may still be that within the fiction, Charles intends to deceive Alice. The result is the same.