4 ms·
Not an expert at all but I am curious if training a LLM purely on ACE input would result in a smaller but still capable model. I guess a small regular trained m
by pluijzer 3y ago
Not an expert at all but I am curious if training a LLM purely on ACE input would result in a smaller but still capable model. I guess a small regular trained model can then translate ACE to a natural language.
- gwern 3y agoNarrowing the task space results in better performance in the short run / with small cheap models, but worse performance in the long run. The model overspecializes and loses any generalizable capabilities because they are not necessary - like the old joke about how professors are like sea squirts, as soon as they get tenure they digest their brains as unnecessary. If you provide the model with adequate scale, the more diverse and challenging the problems, the better (https://gwern.net/scaling-hypothesis#blessings-of-scale https://gwern.net/scaling-hypothesis#blessings-of-scale). So, if you trained a LLM on ACE only (and you somehow had enough to begin with), then it would fail to learn so many things. For example, it wouldn't learn about prose style, or other languages, or different speakers and identities.