> An LLM will always and only output variations of content that exists. I envision a stage in data preparation before training when the model will review and a

> An LLM will always and only output variations of content that exists. I envision a stage in data preparation before training when the model will review and analyse its training data, integrating information that sits in different examples. And not just the dataset, but the model itself can be sampled and the generated corpus be analysed and result in some kind of introspective signal that can be added to the training set. A third source of signal for analysis would be chat logs generate by the previous iteration of the model. Tendency to be a yes man, sycophancy, hallucinations etc would be apparent in the reactions of the users. In fact we are seeing a surge of synthetic datasets generated from higher models like GPT-4 to train open source models like LLaMA. I think the next few years "dataset engineering" will dominate over architecture engineering as the main approach for improvement.