4 ms·
They have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
by riku_iki 3y ago
They have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
- GaggiX 3y agoAll the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.
- riku_iki 3y agoYes, but MQA is limited to 6B size, while "other" larger non-RNN models in table(Llama-2) are not trained on the same dataset, and Hawk and Griffin are 7B. Sorry, I don't understand your point.
- GaggiX 3y agoThe point is that it also beats the baseline on every other size (1B and 3B). So it wouldn't be surprising to see it beat a 7B transformer model like the 6B model. Note 2 on page 5 probably explains why the sizes are different.