3 ms·
35B MoE => 3B active parameters which basically means, the memory goes towards storing the various "expert" layers but the processing speed is only experiencing
by cyanydeez 3mo ago
35B MoE => 3B active parameters which basically means, the memory goes towards storing the various "expert" layers but the processing speed is only experiencing 3B.