3 ms·
The performance advantage comes from doing 1/32 of the floating point operations compared to a dense layer with the same number of parameters.
by yorwba 2y ago
The performance advantage comes from doing 1/32 of the floating point operations compared to a dense layer with the same number of parameters.
- iamnotagenius 2y agoThe performance comes mostly from a fraction of memory bandwidth needed, as LLM are mostly memory constrained. Compute matters too, but usually far less than memory.