You mean because in future it will be impossible to obtain a large high quality dataset due to AI contamination? Worst case you pre-train a model on pre 2022 da

You mean because in future it will be impossible to obtain a large high quality dataset due to AI contamination? Worst case you pre-train a model on pre 2022 data and finetune on a small, curated dataset of recent events. And there are some important breakthroughs that have happened since GPT-4 started training, so it will almost certainly be possible to make a significantly better model with the same dataset and inference cost.