Why not? This would fix the regurgitation issue and possibly be more palatable to copyright holders. The LLM trained on both copyrighted and open works can gene

Why not? This would fix the regurgitation issue and possibly be more palatable to copyright holders. The LLM trained on both copyrighted and open works can generate training data for a second LLM that never sees copyrighted works, achieving perfect separation of idea from expression. Copyright is supposed to protect expression but not ideas. Ideas are free, LLMs should be allowed to learn ideas. This can also filter out bad quality data and PII, or expand on parts of the distribution with less samples. It would be "dataset engineering" as opposed to "model engineering". A recent paper (TinyStories) shows you can train a LM entirely with synthetic data and even evaluate it with another LLM. They could make a model 1000x smaller that still has fluent English - child level English more specifically, but well articulated, and even capable of reasoning. https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 I believe in the future LLMs will train on 100% synthetic data from previous generations of LLM. The benefit will be increased quality, more privacy, less copyright infringement and smaller more efficient models, like TinyStories. Only the first few generations of LLM need to scrape the whole internet, the next generations will have better data.