A user can be satisfied because they're receiving an answer, but it might be a totally wrong answer, maybe in some obvious way like 2+2=5, or in a much less obv

A user can be satisfied because they're receiving an answer, but it might be a totally wrong answer, maybe in some obvious way like 2+2=5, or in a much less obvious way, like generating a /mostly/ accurate biography of a person which includes a year long period of their life that never happened. There needs to be a measurable criteria to judge performance over a variety of metrics over time, because what we notice on a surface level while interacting with AI might not represent the actual performance of accuracy of the outputs we're receiving.