3 ms·
> It seems this would imply that the LLM had somehow managed to expand the dimension of vector space spanned by the original training data. The number of dimen
by TeMPOraL 3y ago
> It seems this would imply that the LLM had somehow managed to expand the dimension of vector space spanned by the original training data.
The number of dimensions? Well, not by itself I guess. But the span of output compared to training data? Sure, why not?
I think it's also worth pointing out there's a difference between text produced by an LLM looped on itself, which arguably may not contain any new information and would be like repeatedly recompressing the same JPG, and text produced by LLM/human interaction. The latter is indirectly recording new knowledge simply because people's prompts are not random. Even with human part of the conversation discarded, feeding such LLM output back into training data would end up selectively emphasizing associations, which is a good signal too (even if noisier than new human-created text).