3 ms·
Yes, sparsification is another useful approach for higher efficiency, although block sparse kernels are pretty complex to work with -- especially when combined
by jph00 3y ago
Yes, sparsification is another useful approach for higher efficiency, although block sparse kernels are pretty complex to work with -- especially when combined with quantization and LoRA! Most of the sparsity papers I've seen use "structured" sparsity; i.e removing layers, attention heads, and features. But the upside from this seems somewhat limited so far.
- yalok 3y agoI'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say 4x4) did not cause any noticeable degradation to training & quality at all (original paper on sparsity from LeCun suggests much higher sparsity ratios - like 90% - but for DNNs I didn't find those attainable if accuracy is important) The block sparsity can really help with saving RAM - because you only need to keep a short array of indexes for the excluded weights. The trouble is the kernel mult functions become complex, so it's a bit of a trade-off between RAM and GPU cycles.