3 ms·

Do you have a link to that paper by any chance? By "neurons" did they mean weights or activations?

• imtringued 4y ago
  Here is a GPU implementation. https://ieeexplore.ieee.org/document/9635657 https://ieeexplore.ieee.org/document/9635657 It is somewhere from 8x to 25x faster than doing dense machine learning. The speedup was higher on the original CPU implementation and the GPU paper mentions that if there isn't enough shared memory on the GPU it will have to switch to an algorithm that has more overhead. By neurons I actually meant "nodes" My comment is effectively a summary of this article: https://www.kdnuggets.com/2020/03/deep-learning-breakthrough-sub-linear-algorithm-no-gpu.html https://www.kdnuggets.com/2020/03/deep-learning-breakthrough... Edit: There is a paper for sparse spiking gradient descent promising a 150x improvement. I am not sure how practical this is because spiking neural network hardware heavily limits your model size but here it is: https://arxiv.org/abs/2105.08810 https://arxiv.org/abs/2105.08810