Synthetic data is doing wonders for models like Phi-4, and at least part of the dataset for DeepSeek-R1 came from their earlier models. If you read the literat

Synthetic data is doing wonders for models like Phi-4, and at least part of the dataset for DeepSeek-R1 came from their earlier models. If you read the literature from the Phi-4 team it talks about synthetic data allowing better control over the training process. The upfront investment is greater but pays off over multiple generations of trained models - and doesn’t leave you with SolidGoldMagikarp ;) https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...