Maybe, and certainly with the current trend of synthetic data they can also create it, but I don't think quantity of data beyond what something like GPT-4 has b

Maybe, and certainly with the current trend of synthetic data they can also create it, but I don't think quantity of data beyond what something like GPT-4 has been trained on will in of itself change much other than reducing brittleness by providing coverage of remaining knowledge gaps. Quality of data (which I believe is at least part of why synthetic data is being used) can perhaps make more of a difference and perhaps at least partly compensate in a crude way for these models lack of outlier rejection and any generalization prediction-feedback loop. Just feed them consistent correct data in the first place.