4 ms·
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majori
by imtringued 4y ago
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that.
What this tells you is that there is very little money in optimizing deep learning and that NVIDIA has made it very easy to just throw more hardware at then problem.
- visarga 4y ago> argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that Yes, but you don't know which 0.5% depending on the input text.
- londons_explore 4y ago> very little money in optimizing deep learning Oh - there are a lot of people working on optimizing AI. Amongst hobbyists, academia, and corporations alike. The thing is, if you come up with a neat optimization that saves 30% of compute for the same results, typically instead of reducing your compute budget 30%, you instead increase your model/data size 30% and get better results.
- WithinReason 4y agoDo you have a link to that paper by any chance? By "neurons" did they mean weights or activations?
- imtringued 4y agoHere is a GPU implementation. https://ieeexplore.ieee.org/document/9635657 https://ieeexplore.ieee.org/document/9635657 It is somewhere from 8x to 25x faster than doing dense machine learning. The speedup was higher on the original CPU implementation and the GPU paper mentions that if there isn't enough shared memory on the GPU it will have to switch to an algorithm that has more overhead. By neurons I actually meant "nodes" My comment is effectively a summary of this article: https://www.kdnuggets.com/2020/03/deep-learning-breakthrough-sub-linear-algorithm-no-gpu.html https://www.kdnuggets.com/2020/03/deep-learning-breakthrough... Edit: There is a paper for sparse spiking gradient descent promising a 150x improvement. I am not sure how practical this is because spiking neural network hardware heavily limits your model size but here it is: https://arxiv.org/abs/2105.08810 https://arxiv.org/abs/2105.08810
- CuriouslyC 4y agoThis is hard a-priori, but fairly easy post-facto. Model distillation isn't a common practice yet, but it has already been demonstrated to be quite effective for specific use cases.
- visarga 4y agoDistillation works but somehow we see very few papers doing it at this scale.
- cma 4y ago200x improvement over Nvidia with sparse matrices or without? Nvidia supports a certain level of sparsity.