My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that dis

My point is that when you asked them to 'judge' two texts, they are just going to output the tokens that have the highest probability (or a sample from that distribution). But depending on their training data, these tokens might say 'correct horse battery staple', and not necessarily 'text A is better' or 'text B is better'. Even if text A would have been more likely to be produced by the LLM.