4 ms·
When they use light to perform matrix multiplication, is the product deterministic, or an approximation? Digital computers spend an enormous amount of energy o
by p1mrx 7y ago
When they use light to perform matrix multiplication, is the product deterministic, or an approximation? Digital computers spend an enormous amount of energy on always getting the same answer, so if an algorithm doesn't need that level of precision, it makes sense that an efficient hardware approximation could be practical.
- deepnotderp 7y agoIt's an approximation. If you would like your self driving car's computer to make more mistakes when it's hotter, then you'll like it.
- p1mrx 7y agoOne would hope that the production version contains health monitoring, because heat and age could similarly degrade an analog computer, and nobody wants a barrage of lawsuits after N years. If the results depend on temperature, then it would be straightforward to climate control the processor; some cars already do this for batteries.
- dooglius 7y agoIt sounds like it's an approximation, but the abstract indicates that it can be used for training, not just inference, so it shouldn't matter a whole lot--if the training process includes the errors, the neural net will be trained to account for them.
- currymj 7y agoIn fact with current neural nets on floating point hardware, people use dropout which is just deliberately introducing random errors. Which is why, only somewhat tongue in cheek, people sometimes describe the rounding errors from using lower-precision arithmetic as "free regularization". So any noise from this being an analogue system would likely improve performance (at least in terms of training).
- deepnotderp 7y agoIt's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with careful swamping prevention and stochastic rounding. [0] In practice, you want fp16 with fp32 master copies, and it turns out that if done improperly, even fp32 can cause accuracy losses sometimes [1]. So, you see, it's not actually the case that you can go proceed directly to analog computing without a worry about silly things like correctness, accuracy and precision. [0] https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf https://papers.nips.cc/paper/7994-training-deep-neural-netwo... [1] https://arxiv.org/pdf/1710.03740.pdf https://arxiv.org/pdf/1710.03740.pdf (links are in pdf, sorry mobile users)
- p1esk 7y agoAnd yet we have analog neural nets in our heads which most likely don’t need more than a few bits of precision.
- thunderbird120 7y agoThey don't use gradient decent though. The learning mechanism used by biological neural networks has almost nothing in common with the one used by most artificial neural networks.
- p1esk 7y agoThen perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.
- thunderbird120 7y agoThe learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does. I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs. [1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plasticity https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...