3 ms·
Can anyone estimate how much of the 3B is necessitated by multi-language support?
by tiahura 1y ago
Can anyone estimate how much of the 3B is necessitated by multi-language support?
- rockinghigh 1y agoThe vocabulary size is fairly small (128,256) for a multilingual model. I would guess it doesn't require many additional parameters to support these 5 languages as many tokens can be shared.
- ethan_smith 1y agoTypically, multilingual capabilities consume 20-30% of model parameters in small LLMs, primarily in token embeddings and early transformer layers. Monolingual variants of similar models often perform better on English benchmarks with the same parameter count.
- netdur 1y agonaive look, 2/3 of model, without multi-languages this shiuld be around 1B