Thanks for that pointer. The abstract of this paper seems interesting: https://arxiv.org/html/2407.01085v3 https://arxiv.org/html/2407.01085v3 > use of [LLMs]

Thanks for that pointer. The abstract of this paper seems interesting: https://arxiv.org/html/2407.01085v3 https://arxiv.org/html/2407.01085v3 > use of [LLMs] as judges [..] reveals a notable bias towards longer responses, undermining the reliability of such evaluations. To better understand such bias, we propose to decompose the preference evaluation metric, specifically the win rate, into two key components: desirability and information mass [..] (If you're interested, give it a click. I tried to pare this down to avoid quoting a wall of text.)