Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model

Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature". It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat"). The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter? I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.