3 ms·
The MoE experts are quantized to int4, all other weights like the shared expert weights are excluded from quantization and use bf16.
by rockinghigh 6mo ago
The MoE experts are quantized to int4, all other weights like the shared expert weights are excluded from quantization and use bf16.