You missed the point they were making, which is that the probabilities it’s predicting are based on what it expects the average text in its training set to look

You missed the point they were making, which is that the probabilities it’s predicting are based on what it expects the average text in its training set to look like. The loss you’re talking about is how closely its answers match the training set, not how clever the answers sound (though with RLHF it’s different). A model producing better text than what’s in its training set would be penalised for not matching it closely and quickly learn to not do that