4 ms·

Have you guys looked at using sparsification? It would probably require true re-training of the foundation model, to go at high sparse ratios (say 90% weights e

by yalok 3y ago

Have you guys looked at using sparsification? It would probably require true re-training of the foundation model, to go at high sparse ratios (say 90% weights excluded), which could be done once on expensive GPU - but fine tuning such sparse models would require less RAM hopefully. The trick with getting more benefit from sparse approach is to do block sparse (iirc, Tim Dettmers used to work on this as well, a few years ago), but large block size (say 16x16) would require much longer retraining to recover for the lost accuracy…

jph00 3y ago
Yes, sparsification is another useful approach for higher efficiency, although block sparse kernels are pretty complex to work with -- especially when combined with quantization and LoRA! Most of the sparsity papers I've seen use "structured" sparsity; i.e removing layers, attention heads, and features. But the upside from this seems somewhat limited so far.
- yalok 3y ago
  I'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say 4x4) did not cause any noticeable degradation to training & quality at all (original paper on sparsity from LeCun suggests much higher sparsity ratios - like 90% - but for DNNs I didn't find those attainable if accuracy is important) The block sparsity can really help with saving RAM - because you only need to keep a short array of indexes for the excluded weights. The trouble is the kernel mult functions become complex, so it's a bit of a trade-off between RAM and GPU cycles.
AhtiK 3y ago
Has anyone seen an implementation of 'SpQR: A Sparse-Quantized Representation,' published in June 2023 by Tim Dettmers et al.? https://arxiv.org/abs/2306.03078 https://arxiv.org/abs/2306.03078
- AhtiK 3y ago
  Found it from https://github.com/Vahe1994/SpQR https://github.com/Vahe1994/SpQR Was somehow expecting it to be at https://github.com/TimDettmers/bitsandbytes https://github.com/TimDettmers/bitsandbytes. My bad.

by yalok 3y ago

jph00 3y ago

Yes, sparsification is another useful approach for higher efficiency, although block sparse kernels are pretty complex to work with -- especially when combined with quantization and LoRA! Most of the sparsity papers I've seen use "structured" sparsity; i.e removing layers, attention heads, and features. But the upside from this seems somewhat limited so far.

yalok 3y ago
I'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say 4x4) did not cause any noticeable degradation to training & quality at all (original paper on sparsity from LeCun suggests much higher sparsity ratios - like 90% - but for DNNs I didn't find those attainable if accuracy is important) The block sparsity can really help with saving RAM - because you only need to keep a short array of indexes for the excluded weights. The trouble is the kernel mult functions become complex, so it's a bit of a trade-off between RAM and GPU cycles.

AhtiK 3y ago

Has anyone seen an implementation of 'SpQR: A Sparse-Quantized Representation,' published in June 2023 by Tim Dettmers et al.? https://arxiv.org/abs/2306.03078 https://arxiv.org/abs/2306.03078

AhtiK 3y ago
Found it from https://github.com/Vahe1994/SpQR https://github.com/Vahe1994/SpQR Was somehow expecting it to be at https://github.com/TimDettmers/bitsandbytes https://github.com/TimDettmers/bitsandbytes. My bad.