3 ms·
Chinchilla predicts that you could get lower loss by training a larger model with that amount of data. But the model size in this case was chosen for other reas
by sp332 3y ago
Chinchilla predicts that you could get lower loss by training a larger model with that amount of data. But the model size in this case was chosen for other reasons, mostly speed of inference and cost of fine-tuning. So it's just irrelevant here.
- GaggiX 3y agoWell it's relevant if you want to compare the model trained optimally using the same amount of compute and this one parameter-bound to see how much you're trading.