I am certain investments into synthetic data will proliferate exponentially. Synthetic data can be more diverse, less toxic, and of higher average quality. But

I am certain investments into synthetic data will proliferate exponentially. Synthetic data can be more diverse, less toxic, and of higher average quality. But the foremost rationale we require it is to improve models. You likely recall the reversal curse - if an LLM trains "A is B" it doesn't automatically deduce "B is A". You need to explicitly append the second part to the training set to make it complete. Fragmented or incomplete information in the training set persists fragmented in the trained model. Training is blind, only inference is intelligent. Similarly, many training examples contain apparent information that conceals implicit deductions, for instance a math problem - the statement is evident and apparent but the chain of thought is implicit, concealed. What I am getting at is a general principle - LLMs need to "study" the original training data and make it comprehensive, elicit those implicit deductions, and enable LLMs to traverse the conceptual space in all directions. Study, then train. In other words, to study is to digest the raw data, to reconnect fragmented information, to generate insight, to execute the instructions and see the result, generally to unfold what is hidden. It's a matter of utilizing LLMs in generative mode to prepare the dataset before training. Microsoft has generated a 150B token synthetic dataset for Phi-1.5 and it demonstrated 5x efficiency gains. They will likely increase 100x. The promise of faster, cheaper inference and fewer errors is very alluring.