I'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say

I'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say 4x4) did not cause any noticeable degradation to training & quality at all (original paper on sparsity from LeCun suggests much higher sparsity ratios - like 90% - but for DNNs I didn't find those attainable if accuracy is important) The block sparsity can really help with saving RAM - because you only need to keep a short array of indexes for the excluded weights. The trouble is the kernel mult functions become complex, so it's a bit of a trade-off between RAM and GPU cycles.