An LLM can't generate new training data for itself. Mathematically speaking, the output of the LLM is governed by the same statistics embedded in its parameters

An LLM can't generate new training data for itself. Mathematically speaking, the output of the LLM is governed by the same statistics embedded in its parameters, and a large enough training set of LLM output should have zero impact on those parameters. The difference is that training on human output contains the implicit assumption that the human output is "correct" in some way. So if the LLM gets more human-generated training data, it can come closer to "correctness". An LLM training on its own output is merely reinforcing its weights. Maybe to put it more simply, there is no LLM in the world that can determine for itself whether a proposition or piece of training data is sound without a person saying so at some point, either implicitly or explicitly. Even efforts to give LLMs some non-human-mediated reward function are merely people attempting to encode our own principles. While people are no doubt physical, we have a biological imperative to survival and well-being that governs our sense-making faculties, and to which nobody has yet proposed an analogue for LLMs. Our "transformations" are not only governed by other people's "training data" but by our urge to be able to skillfully cope with a world in which we are biologically invested.