Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless t

Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless the data is insufficient/unvaried or the architecture creates a bottleneck. I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway. The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.