> But at what point do we run out of training data? We have at least another order of magnitude of additional data via audio-video. There also is synthetic dat

> But at what point do we run out of training data? We have at least another order of magnitude of additional data via audio-video. There also is synthetic data, such as was likely done for SORA. And we likely could increase that another few orders of magnitude if we had more universal sensors. Windows Recall is crazy for privacy reasons, but gathering feeds of data from everyone as they go about their lives, physical as well as digital provides much more room for training. You also need to take into account data-growth. How much more text data is generated daily, relative to how much has ever existed? There also is probably another 1-2 orders of magnitude more data in private locations and also growing fast. Some of that can be captured via RAG. But even if a small fraction is used for fine-tuning, in aggregate that is a lot of GPU.