Hamel Husein had this great slide in a recent Youtube video where he compared human raters of LLM outputs. He pointed out a peculiarity, where it looked like th

Hamel Husein had this great slide in a recent Youtube video where he compared human raters of LLM outputs. He pointed out a peculiarity, where it looked like there wasn't improvement between pipeline versions but it was actually because the raters themselves began to have higher expectations over time. The same output in week 0 was rated as a 7. After 6 weeks of rating LLM outputs, especially as the pipeline improved them, was a lower score later.