Yes in the OpenAI paper you don't need to learn the sparsity structure. But afaiu the mit paper help suggest an appropriate sparsity structure. Concerning the

Yes in the OpenAI paper you don't need to learn the sparsity structure. But afaiu the mit paper help suggest an appropriate sparsity structure. Concerning the sparsity speed-up, I've tried the tensorflow sparse representation a while back, and it was kind of a high effort, low reward process. You had to drop like 90% of the weights (was reducing accuracy), change your ops (dense, convolutions,...) and use big enough layer size to get a feel you were getting something speed-wise. The openai block-sparse kernels seems promising I'll give them a try.