What do you mean by "sample their training data"? It gives a probability distribution over possible values for the next token, in a way that was trained to do

What do you mean by "sample their training data"? It gives a probability distribution over possible values for the next token, in a way that was trained to do well on the training data, yes. But, something which gives a uniform distribution over characters, has a nonzero (though of course minuscule and entirely negligible) chance of giving any given sequence of characters, and so if there is any text which would be "novel", it is "possible" that it would give such a text. A distribution which has a greater tendency to give meaningful text, is, I think, more likely to give text which is "novel"? Like, a uniform distribution over "text which is grammatically valid English text" is more likely to produce text which is interpreted as corresponding to a novel idea, than a distribution over all possible strings of text. Of course, that's not the distribution that GPT3 produces. Now, something which took random full sentences from the training set, that seems like one might say that that "can't produce anything novel", because even if the sequence of sentences it produces hasn't been seen before, they basically won't ever make sense together, much less in order to describe some novel idea? Well, I guess it is probably more likely to do so than the one that generates uniformly random strings of characters?