I'd say RLHF bends the game towards predicting what words a "helpful", "respectful" person would write next, for values of "helpful" and "respectful" that vary

I'd say RLHF bends the game towards predicting what words a "helpful", "respectful" person would write next, for values of "helpful" and "respectful" that vary according to each person involved in scoring (but which are carefully shaped by the people choosing those people, and paying for their time).