5 ms·

It sounds like it's an approximation, but the abstract indicates that it can be used for training, not just inference, so it shouldn't matter a whole lot--if th

It sounds like it's an approximation, but the abstract indicates that it can be used for training, not just inference, so it shouldn't matter a whole lot--if the training process includes the errors, the neural net will be trained to account for them.

currymj 7y ago
In fact with current neural nets on floating point hardware, people use dropout which is just deliberately introducing random errors. Which is why, only somewhat tongue in cheek, people sometimes describe the rounding errors from using lower-precision arithmetic as "free regularization". So any noise from this being an analogue system would likely improve performance (at least in terms of training).
- deepnotderp 7y ago
  It's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with careful swamping prevention and stochastic rounding. [0] In practice, you want fp16 with fp32 master copies, and it turns out that if done improperly, even fp32 can cause accuracy losses sometimes [1]. So, you see, it's not actually the case that you can go proceed directly to analog computing without a worry about silly things like correctness, accuracy and precision. [0] https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf https://papers.nips.cc/paper/7994-training-deep-neural-netwo... [1] https://arxiv.org/pdf/1710.03740.pdf https://arxiv.org/pdf/1710.03740.pdf (links are in pdf, sorry mobile users)
  - p1esk 7y ago
    And yet we have analog neural nets in our heads which most likely don’t need more than a few bits of precision.
    - thunderbird120 7y ago
      They don't use gradient decent though. The learning mechanism used by biological neural networks has almost nothing in common with the one used by most artificial neural networks.
      - p1esk 7y ago
        Then perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.
        thunderbird120 7y ago
        The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does. I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs. [1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plasticity https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...
        p1esk 7y ago
        STDP is one of the early steps towards understanding the learning mechanism(s) in the brain, but we are a long way from enough understanding to actually reproduce it. Not only we don't know how to train spiking networks, we don't even know how the information is encoded: pulse frequency or pulse timings or ...? No one knows. How can you compete with anything if you have no idea how it works? Also, this has nothing to do with computing hardware. You can easily simulate anything your want on conventional processors. Huge computing clusters have been built for spiking model simulations, and nothing interesting came out of it. Invent the algorithm first, then we will build the hardware for it.
        7y ago

currymj 7y ago

In fact with current neural nets on floating point hardware, people use dropout which is just deliberately introducing random errors. Which is why, only somewhat tongue in cheek, people sometimes describe the rounding errors from using lower-precision arithmetic as "free regularization". So any noise from this being an analogue system would likely improve performance (at least in terms of training).

deepnotderp 7y ago
It's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with careful swamping prevention and stochastic rounding. [0] In practice, you want fp16 with fp32 master copies, and it turns out that if done improperly, even fp32 can cause accuracy losses sometimes [1]. So, you see, it's not actually the case that you can go proceed directly to analog computing without a worry about silly things like correctness, accuracy and precision. [0] https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf https://papers.nips.cc/paper/7994-training-deep-neural-netwo... [1] https://arxiv.org/pdf/1710.03740.pdf https://arxiv.org/pdf/1710.03740.pdf (links are in pdf, sorry mobile users)
- p1esk 7y ago
  And yet we have analog neural nets in our heads which most likely don’t need more than a few bits of precision.
  - thunderbird120 7y ago
    They don't use gradient decent though. The learning mechanism used by biological neural networks has almost nothing in common with the one used by most artificial neural networks.
    - p1esk 7y ago
      Then perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.
      - thunderbird120 7y ago
        The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does. I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs. [1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plasticity https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...
        p1esk 7y ago
        STDP is one of the early steps towards understanding the learning mechanism(s) in the brain, but we are a long way from enough understanding to actually reproduce it. Not only we don't know how to train spiking networks, we don't even know how the information is encoded: pulse frequency or pulse timings or ...? No one knows. How can you compete with anything if you have no idea how it works? Also, this has nothing to do with computing hardware. You can easily simulate anything your want on conventional processors. Huge computing clusters have been built for spiking model simulations, and nothing interesting came out of it. Invent the algorithm first, then we will build the hardware for it.
        7y ago