Narrowing the task space results in better performance in the short run / with small cheap models, but worse performance in the long run. The model overspeciali

Narrowing the task space results in better performance in the short run / with small cheap models, but worse performance in the long run. The model overspecializes and loses any generalizable capabilities because they are not necessary - like the old joke about how professors are like sea squirts, as soon as they get tenure they digest their brains as unnecessary. If you provide the model with adequate scale, the more diverse and challenging the problems, the better (https://gwern.net/scaling-hypothesis#blessings-of-scale https://gwern.net/scaling-hypothesis#blessings-of-scale). So, if you trained a LLM on ACE only (and you somehow had enough to begin with), then it would fail to learn so many things. For example, it wouldn't learn about prose style, or other languages, or different speakers and identities.