3 ms·
Mixtral 8x7B actually has 46.7B total parameters, not 8*7B = 56B. The reason being that not all parameters are multiplied 8x. Also it uses 12.9B parameters per
by versteegen 3y ago
Mixtral 8x7B actually has 46.7B total parameters, not 8*7B = 56B. The reason being that not all parameters are multiplied 8x.
Also it uses 12.9B parameters per token, not quite comparable to 7B models.