3 ms·

And I forgot to emphasize - you can select what weights are non-zero beforehand, even before training anything. The network will route around that. This means

• GistNoesis 8y ago
  Yes in the OpenAI paper you don't need to learn the sparsity structure. But afaiu the mit paper help suggest an appropriate sparsity structure. Concerning the sparsity speed-up, I've tried the tensorflow sparse representation a while back, and it was kind of a high effort, low reward process. You had to drop like 90% of the weights (was reducing accuracy), change your ops (dense, convolutions,...) and use big enough layer size to get a feel you were getting something speed-wise. The openai block-sparse kernels seems promising I'll give them a try.