3 ms·
This is one cool hack. You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transforma
by Nokinside 9y ago
This is one cool hack.
You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations.
I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network pipeline constructed from just summation and addition.
- darkmighty 9y agoIt's also possible to binarize DNNs to use the faster bitwise operations https://arxiv.org/pdf/1602.02830.pdf https://arxiv.org/pdf/1602.02830.pdf
- dnautics 9y ago> I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. At that point, you are probably better off just building circuits with "power this wire to ReLU at the end", which is not very many extra transistors.