4 ms·
Language generation is a very good example of a task that is very hard to evaluate with any degree of objectivity and for which there are no good metrics. So,
by YeGoblynQueenne 5y ago
Language generation is a very good example of a task that is very hard to evaluate with any degree of objectivity and for which there are no good metrics.
So, suppose you say that a particular bit of text generated by GPT-3 is "indistinguishable from what a human would respond". If I say it isn't, how can we decide who is right in a way that we can both agree on?
And that's all before we try to figure out "generalisation".
- kthejoker2 5y agoIs this not literally the (generalized version of the) Turing Test? You simply hand the text to someone and ask to guess if it was produced by a human or not. Or hand them two (or ten, or ten thousand) texts and ask them to label the human and AI texts without knowing the actual distribution.
- YeGoblynQueenne 5y agoIn the Turing Test you have one human judge. I'm asking what happens when two humans disagree about the human-ness of some automatically generated text.