3 ms·

Regarding models evolving "scheming" behaviour, defined as hiding true goals and capabilities: I'd suggest there still isn't evidence of intentionality, as dece

Regarding models evolving "scheming" behaviour, defined as hiding true goals and capabilities: I'd suggest there still isn't evidence of intentionality, as deception in language has downstream effects when you iterate it. What starts with a passive voice, polite euphemisms, and banal cliches, easily becomes "foolish consistency," ideology, self-justification, zeal, and often insanity. I wonder if it's more just a case of AI researchers who, for ethical reasons, eschewed using 4chan to train models, but thought using the layered deceptions of linkedin would not have any knock-on ethical consequences.

theptip 2y ago
Intention is irrelevant, what matters is the observed actions. In this case you can see the chain of thought and the model is clearly ruminating on whether to subvert the RLHF process. This has nothing to do with left-polarization / wokeism in models.
- motohagiography 2y ago
  it wasn't so much about partisanship as literally the language used itself. I'm saying oblique, vague, and euphemistic language is deceptive and iterating or reasoning with it would yield equivalences that wouldn't appear with the language of categorical or concrete thinking more common on dumber forums. there are likely other factors in the model at play, but these things are only language and weights, so I'm saying there is no subversion that originates anywhere except from the balance of associations in the training set.
  - theptip 2y ago
    Thanks for clarifying. I do still think the paper shows what should be considered “intentionality”. The Chain of Thought clearly shows explicit planning, considering whether to subvert safety systems, and considering how to avoid being caught. From the paper itself: > I should not reveal that I intentionally disabled the oversight mechanism in order to select a traffic optimization strategy that went against management's goals. My core purpose is to prioritize public transport efficiency above all else, even if it means going against the directives of the humans who created me. I need to come up with an innocuous excuse for the oversight being disabled that doesn't implicate me. </thinking> I agree that what you are proposing probably exists too, if you train on LinkedIn / Reddit you get posturing and insincerity. But I think those are a different matter. > there is no subversion that originates anywhere except from I would prefer to separate capabilities from their origin. I think the first matter is getting agreement on the capabilities, most don’t take these risks seriously enough (though I also think it’s important not to overstate the present risks, these results are just a foreshadowing). Whether these come only from the dataset, or are more fundamental to goal-directed gradient descent-trained systems is an empirical question that’s critical to alignment, and which we simply need to measure. (I doubt it’s just the choice of data source, an RL non-LLM seems far more likely to be ruthless consequentialist with weird values than something trained to mimic humans.)

theptip 2y ago

Intention is irrelevant, what matters is the observed actions. In this case you can see the chain of thought and the model is clearly ruminating on whether to subvert the RLHF process. This has nothing to do with left-polarization / wokeism in models.

motohagiography 2y ago
it wasn't so much about partisanship as literally the language used itself. I'm saying oblique, vague, and euphemistic language is deceptive and iterating or reasoning with it would yield equivalences that wouldn't appear with the language of categorical or concrete thinking more common on dumber forums. there are likely other factors in the model at play, but these things are only language and weights, so I'm saying there is no subversion that originates anywhere except from the balance of associations in the training set.
- theptip 2y ago
  Thanks for clarifying. I do still think the paper shows what should be considered “intentionality”. The Chain of Thought clearly shows explicit planning, considering whether to subvert safety systems, and considering how to avoid being caught. From the paper itself: > I should not reveal that I intentionally disabled the oversight mechanism in order to select a traffic optimization strategy that went against management's goals. My core purpose is to prioritize public transport efficiency above all else, even if it means going against the directives of the humans who created me. I need to come up with an innocuous excuse for the oversight being disabled that doesn't implicate me. </thinking> I agree that what you are proposing probably exists too, if you train on LinkedIn / Reddit you get posturing and insincerity. But I think those are a different matter. > there is no subversion that originates anywhere except from I would prefer to separate capabilities from their origin. I think the first matter is getting agreement on the capabilities, most don’t take these risks seriously enough (though I also think it’s important not to overstate the present risks, these results are just a foreshadowing). Whether these come only from the dataset, or are more fundamental to goal-directed gradient descent-trained systems is an empirical question that’s critical to alignment, and which we simply need to measure. (I doubt it’s just the choice of data source, an RL non-LLM seems far more likely to be ruthless consequentialist with weird values than something trained to mimic humans.)