https://arxiv.org/pdf/2305.17493 https://arxiv.org/pdf/2305.17493 There's some cursory indication that in the long tail, training LLMs on LLM-generated data ca

https://arxiv.org/pdf/2305.17493 https://arxiv.org/pdf/2305.17493 There's some cursory indication that in the long tail, training LLMs on LLM-generated data causes model collapse. Kind of like how if you photocopy a photocopy too many times the document becomes unreadable. This isn't really surprising though. Neural networks at large are a form of lossy compression. You can't do lossy compression on artifacts recovered from lossy compression too many times. The losses stack.