There actually is a reward function for text that can be used to go beyond the human input data. It is plausibility: If you question the response and check it

There actually is a reward function for text that can be used to go beyond the human input data. It is plausibility: If you question the response and check it against responses about related questions, how much does it align with those? This is what we humans do, too. It’s also what we do in science. It’s things not “adding up” that tells us where we must improve.