I don't think it's so cut and dry. The article paints a somewhat simplistic picture of how the best performing LLMs work. The unsupervised pre-trained networks

I don't think it's so cut and dry. The article paints a somewhat simplistic picture of how the best performing LLMs work. The unsupervised pre-trained networks are indeed data hungry, but the secondary supervised learning stages actually can get by with a far smaller set of highly curated prompt-response data, e.g. LIMA (https://arxiv.org/abs/2305.11206 https://arxiv.org/abs/2305.11206). Another factor is that generative data distributed online may be quite high quality (because people find it interesting enough to share), so it's plausible this could actually improve model performance. Some LLMs have been trained with data from other models with good results, e.g. supposedly Bard with GPT-4 prompt-response pairs and GPT-4 with Whisper transcripts of YouTube (https://twitter.com/amir/status/1641219919202361344/photo/1 https://twitter.com/amir/status/1641219919202361344/photo/1). Of course, there could be trolling or misinformation that "poisons" the data, and that is a problem (whether synthetic or organic)!