So does the program while True: print('a') GPT-n (not chat) generates continuations to text in which the probability distribution for the next out

So does the program while True: print('a') GPT-n (not chat) generates continuations to text in which the probability distribution for the next output it produces matches the distribution of outputs after similar text in the training corpus, by most metrics you might examine. Trivial examples of those metrics include word and n-gram frequencies, but also it turns out that at sufficiently low loss that looks like "given a couple of input texts and a context which affords insightful commentary, produce insightful commentary at the base rate". There are, of course, caveats. Notably: - ChatGPT has its own stuff going on around chat tuning, tool use, tuning to be less likely to produce outputs OpenAI doesn't want, etc - The statistical regularity thing is not magic. If it requires more than n_layers steps of computation to determine the sensible next output, the model will not be able to do that. I think the canonical example here is usually having the model complete something like "'d072c916029965a7676da4244160c413e31bc8a0' == sha1('I saw it on hn'); '265149165fcb742a900a44b8f123885dc6ac5d12' == sha1('" and then having the model brute-force sha1 -- obviously it's not going to be able to do that. - The model generates text sequentially, one token at a time. This is importantly not the process by which most text in the training corpus was written. So in the cases where earlier text importantly depends on text which was written earlier temporally, but which occurs later in the string, the model will be likely to make mistakes (where a "mistake" is "writing text which is statistically surprising, relative to the training corpus").