4 ms·
They probably had them for some reason and it was cheaper not to retrain one of them again
by szundi 3y ago
They probably had them for some reason and it was cheaper not to retrain one of them again
- riku_iki 3y agoIts just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..
- GaggiX 3y agoIt also performs better on any other size.
- riku_iki 3y agoThey have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
- GaggiX 3y agoAll the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.
- riku_iki 3y agoYes, but MQA is limited to 6B size, while "other" larger non-RNN models in table(Llama-2) are not trained on the same dataset, and Hawk and Griffin are 7B. Sorry, I don't understand your point.
- GaggiX 3y agoThe point is that it also beats the baseline on every other size (1B and 3B). So it wouldn't be surprising to see it beat a 7B transformer model like the 6B model. Note 2 on page 5 probably explains why the sizes are different.