That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use sup

That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the system prompt, user request and model response[1] to successfully train a smaller model that outperforms Vicuna-13B. The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks. The point here is that the simple presence of machine generated content in the dataset is not an immediate detriment to the model being trained on it. It all hinges on the quality of the data regardless of whether it is of human or machine origin. Think about it, if the information is correct then there is no difference whether it came from a human or a machine. Machine generated data is not secretly cursed or other such nonsense. It's either correct or incorrect, nothing more. [0] https://arxiv.org/pdf/2305.17493v2.pdf https://arxiv.org/pdf/2305.17493v2.pdf [1] https://arxiv.org/pdf/2306.02707.pdf https://arxiv.org/pdf/2306.02707.pdf [2] https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf