Much of GPT-4's training data was made by GPT-3.5. Info on the synthetic data in GPT-4's pretraining is starting to leak, including a reference to "Synthetic-Da

Much of GPT-4's training data was made by GPT-3.5. Info on the synthetic data in GPT-4's pretraining is starting to leak, including a reference to "Synthetic-Data(2)" in a recent OpenAI paper. In the near future, if not already, practically all training data for state of the art models will be synthetic. In the words of OpenAI CEO Sam Altman, they have "boostrapped" and are "past the synthetic data generation event horizon."