Up until now, LLM training has used mostly pre-existing data, hoarding what has already been produced and feeding that in as-is. Think textbooks, Wikipedia, Git

Up until now, LLM training has used mostly pre-existing data, hoarding what has already been produced and feeding that in as-is. Think textbooks, Wikipedia, GitHub, etc... That's starting to run dry, hence the predictions of progress stalling, but it doesn't factor in the option of using vast volumes of synthetic data. Some of the "thinking" models are already using generated problem sets, but this is just the tip of the iceberg. There are so, so many ways in which synthetic data could be generated! Some random examples are: - Introduce a typo into a working program or configuration file. Train the AI to recognise the typo based on the code and the error message. - Bulk-generate homework or exam problems with the names, phrasing, quantities, etc... randomised. - Train the AI not just on GitHub repos, but the full Git history of every branch, including errors generated by the compiler for commits that fail to build. - Compile C/C++ code to various machine-code formats, assembly, LLVM IR, etc... and train the AI to reverse-engineer from the binary output to the source. ... and on and on.