3 ms·
Picking up from there: The games in this paper and model are annoying. The 2.6B would get stomped by Phi-3, so there's no comparison. Fair enough. 2.6B vs. 3.
by refulgentis 2y ago
Picking up from there: The games in this paper and model are annoying.
The 2.6B would get stomped by Phi-3, so there's no comparison.
Fair enough. 2.6B vs. 3.8B is a fairly substantial size difference thats hard to intuit when its 2.6 vs 3.8 versus 2,600,000,000 and 3,800,000,000.
But then we get what I'm going to "parameter creep": Mistral 7B vs. Llama 8B vs. Gemma 9B. I worried after Llama 3 went 8B that we'd start seeing games with parameters, but, thought I was being silly.
- imjonse 2y agoIn the Llama 3 case I think the increase in parameters is mostly due to the input embeddings and output logits layers, reflecting the context size increase.
- kouteiheika 2y agoThere was no parameter creep with Llama. Llama 8B is actually a ~7B model comparable to Mistral 7B if you strip away multilingual embeddings and match what Mistral 7B supports.
- alecco 2y agoPhi-3 3.8B seems to perform much better on almost every test than Gemma 2 9B. It is comparable.
- refulgentis 2y agoI agree. The implication in my post is "if the reason was size, it's invalidated later"