4 ms·
I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin Why wouldn't select equal size models?..
by riku_iki 3y ago
I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin
Why wouldn't select equal size models?..
- szundi 3y agoThey probably had them for some reason and it was cheaper not to retrain one of them again
- riku_iki 3y agoIts just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..
- GaggiX 3y agoIt also performs better on any other size.
- riku_iki 3y agoThey have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
- GaggiX 3y agoAll the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.
- riku_iki 3y agoYes, but MQA is limited to 6B size, while "other" larger non-RNN models in table(Llama-2) are not trained on the same dataset, and Hawk and Griffin are 7B. Sorry, I don't understand your point.
- GaggiX 3y agoThe point is that it also beats the baseline on every other size (1B and 3B). So it wouldn't be surprising to see it beat a 7B transformer model like the 6B model. Note 2 on page 5 probably explains why the sizes are different.