If you already have a large model you can rank & filter the training data and retrain. If the model doesn't like the data, chances are it is low quality. That i

If you already have a large model you can rank & filter the training data and retrain. If the model doesn't like the data, chances are it is low quality. That is how MS built Phi-1 and 1.5, by curating web data + generating with a large model. The second part (synthetic content) improves diversity by methodically covering concepts or tasks.