4 ms·

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.c

Just to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.com/religion/2020/01/03/united-methodist-church-is-expected-split-over-gay-marriage-disagreement-fracturing-nations-third-largest-denomination/ https://www.washingtonpost.com/religion/2020/01/03/united-me... https://www.washingtonpost.com/archive/local/1985/09/07/united-methodists-alarmed-by-empty-pews/db664b00-56be-4693-b8c0-36aa3d9a9905/ https://www.washingtonpost.com/archive/local/1985/09/07/unit... GPT-3: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. WP: The church has lost 1.6 million members since 1968, when the Methodist Church merged with the considerably smaller Evangelical United Brethren to form the present United Methodist Church. I think this model is still very impressive, the parameter itself speaks. But for this particular evaluation, the same news article may be removed from the training set, other news article that paraphrases the same story might not. IMO, the leakage still exists, it is hard to tell whether this model are really 'generating', or just copy-pasting from its vast memory.

gas9S9zw3P9c 6y ago
Where do you draw the line between "generating" and "copy-pasting from its vast memory"? Why do you think what humans do is not copy & pasting different snippets of information they have come across in the past? Isn't that what grammar is? A bunch of rules you've come across a lot of times? Other than the given prompt, the models don't have a goal. So what other than copying and adjusting would they do?
- tanilama 6y ago
  To evaluate generic model is hard. For example, for image synthesis in GAN, the widely used Inception score balances between authenticity of the generate samples vs the variety as well, to make sure the model is not copy-pasting. In this particular case, apparently the same event has been reported multiple times by different news agency. Even if the exact one are excluded, still it is suspicious how much less the model is being protected from knowing the subject itself. An analogy would exam in real world. Often, some of the questions aren't leaked as is, but paraphrased yet stay close enough to the source. In this particular case though, I disagree it is reaching human level generation. They can tested the model with an unseen events, which happen after the model is trained to test how well it generalize.
nyolfen 6y ago
that particular fact has probably been printed in more than one place with more than one phrasing; is there a reason you think it was drawn from that one in particular?

gas9S9zw3P9c 6y ago

Where do you draw the line between "generating" and "copy-pasting from its vast memory"? Why do you think what humans do is not copy & pasting different snippets of information they have come across in the past? Isn't that what grammar is? A bunch of rules you've come across a lot of times? Other than the given prompt, the models don't have a goal. So what other than copying and adjusting would they do?

tanilama 6y ago
To evaluate generic model is hard. For example, for image synthesis in GAN, the widely used Inception score balances between authenticity of the generate samples vs the variety as well, to make sure the model is not copy-pasting. In this particular case, apparently the same event has been reported multiple times by different news agency. Even if the exact one are excluded, still it is suspicious how much less the model is being protected from knowing the subject itself. An analogy would exam in real world. Often, some of the questions aren't leaked as is, but paraphrased yet stay close enough to the source. In this particular case though, I disagree it is reaching human level generation. They can tested the model with an unseen events, which happen after the model is trained to test how well it generalize.

nyolfen 6y ago

that particular fact has probably been printed in more than one place with more than one phrasing; is there a reason you think it was drawn from that one in particular?