In a way, I think training on historic chats is akin to caching computation results. The compute cost has already been paid, and we make future retrievals cheap

In a way, I think training on historic chats is akin to caching computation results. The compute cost has already been paid, and we make future retrievals cheaper by encoding it directly in the model. Assuming the results included some external validation such as user's preference, compilation, lean, etc., I'm not sure whether this would lead to model collapse.