3 ms·
The prediction from the LessWrong folks was that it’s inevitable that a rational actor with “goals” would do this, so models trained purely on RL would exhibit
by theptip 2y ago
The prediction from the LessWrong folks was that it’s inevitable that a rational actor with “goals” would do this, so models trained purely on RL would exhibit this too. (Instrumental Convergence is the name for the theory that predicts power-seeking in a generalized way.)
I agree that we should expect LLMs to be particularly vulnerable to this as you note. But it seems to me that LLMs seem to be absorbing some understanding of human morality too, which might make it possible to steer them into “the best of us” territory.