3 ms·
> The OPT 6.7B model, for instance, exhibits a notable 97% sparsity within its FFN layer. Does anyone here know what exactly that metric refers to? Does it mea
by PoignardAzur 3y ago
> The OPT 6.7B model, for instance, exhibits a notable 97% sparsity within its FFN layer.
Does anyone here know what exactly that metric refers to? Does it mean the layer has 97% 0 values? That it can be compressed to 3% of its size?
- cgearhart 3y agoIt means that 97% of the outputs from the layer are zero; only 3% are active _at a time_. You can’t get rid of the other 97% entirely because the active 3% isn’t static. I think the paper is saying that they can reasonably accurately predict the active 3% at least well enough to make it run faster without losing too much accuracy.