3 ms·
> no matter how you want to define "generalisation" ... and how you want to define "task". For some prompts/"tasks", GPT-3 does generate impressive (more than
by momenti 5y ago
> no matter how you want to define "generalisation"
... and how you want to define "task". For some prompts/"tasks", GPT-3 does generate impressive (more than trivial) outputs that cannot be found on the internet and that are indistinguishable from what a human would respond, so it generalizes in that sense. Maybe human ingenuity and generalization is also just slightly perturbed interpolation? It is very difficult to produce something truly novel, so we are also rather tightly limited by prior experience. Who knows? Also, who cares if submarines swim? Anyhow, it seems 50% of the internet is bikeshedding about definitions.
- YeGoblynQueenne 5y agoLanguage generation is a very good example of a task that is very hard to evaluate with any degree of objectivity and for which there are no good metrics. So, suppose you say that a particular bit of text generated by GPT-3 is "indistinguishable from what a human would respond". If I say it isn't, how can we decide who is right in a way that we can both agree on? And that's all before we try to figure out "generalisation".
- kthejoker2 5y agoIs this not literally the (generalized version of the) Turing Test? You simply hand the text to someone and ask to guess if it was produced by a human or not. Or hand them two (or ten, or ten thousand) texts and ask them to label the human and AI texts without knowing the actual distribution.
- YeGoblynQueenne 5y agoIn the Turing Test you have one human judge. I'm asking what happens when two humans disagree about the human-ness of some automatically generated text.