Given a prompt or partial output, human continuations follow some unknown probability distribution over the set of possible next words. Suppose that ChatGPT ha

Given a prompt or partial output, human continuations follow some unknown probability distribution over the set of possible next words. Suppose that ChatGPT has perfectly learned this distribution - it can correctly tell you the probability of any possible next word. Does that make ChatGPT indistinguishable from human speech? Not necessarily, because there is also the question of how one samples actual outputs from ChatGPT's predicted distribution. For example, suppose that OpenAI did the simplest thing, and always returned the most likely next word (in other words, sampling with temperature = 0). In that case, the perplexity of ChatGPT's outputs would be consistently lower than that of human speech, even according to a weaker language model. In reality, ChatGPT's sampling is surely more subtle than this, but it is still unlikely to exactly match its predicted distribution - OpenAI wants to produce useful responses, so it will sample in a way that favors more probable continuations. I would expect that if you passed ChatGPT's outputs to a weaker language model and evaluated their likelihood, and compared it to the likelihood of a known-human corpus, it would quickly become apparent which is which. After all, OpenAI's goal is for ChatGPT to sound human to a human reader (in addition to other goals like being helpful, being honest, not providing harmful information, etc.), not be statistically indistinguishable from a human to a language model.