The Phi paper and various approaches to distilling from GPT-4 demonstrate that the training data and plausibly order of presentation matter. The challenge is t

The Phi paper and various approaches to distilling from GPT-4 demonstrate that the training data and plausibly order of presentation matter. The challenge is that we both do not understand which set of data is most beneficial for training, or how it could be efficiently ordered without triggering computationally infeasible problems. However we do know how to massively scale up training.