4 ms·

It's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with caref

It's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with careful swamping prevention and stochastic rounding. [0] In practice, you want fp16 with fp32 master copies, and it turns out that if done improperly, even fp32 can cause accuracy losses sometimes [1]. So, you see, it's not actually the case that you can go proceed directly to analog computing without a worry about silly things like correctness, accuracy and precision. [0] https://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf https://papers.nips.cc/paper/7994-training-deep-neural-netwo... [1] https://arxiv.org/pdf/1710.03740.pdf https://arxiv.org/pdf/1710.03740.pdf (links are in pdf, sorry mobile users)

p1esk 7y ago
And yet we have analog neural nets in our heads which most likely don’t need more than a few bits of precision.
- thunderbird120 7y ago
  They don't use gradient decent though. The learning mechanism used by biological neural networks has almost nothing in common with the one used by most artificial neural networks.
  - p1esk 7y ago
    Then perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.
    - thunderbird120 7y ago
      The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does. I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs. [1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plasticity https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...
      - p1esk 7y ago
        STDP is one of the early steps towards understanding the learning mechanism(s) in the brain, but we are a long way from enough understanding to actually reproduce it. Not only we don't know how to train spiking networks, we don't even know how the information is encoded: pulse frequency or pulse timings or ...? No one knows. How can you compete with anything if you have no idea how it works? Also, this has nothing to do with computing hardware. You can easily simulate anything your want on conventional processors. Huge computing clusters have been built for spiking model simulations, and nothing interesting came out of it. Invent the algorithm first, then we will build the hardware for it.
        thunderbird120 7y ago
        STDP based SNNs don't take much secret sauce to function reasonably well for basic tasks[1]. Some people have even started to apply reward modulated STDP to the problem which is nice to see since it's a more complete version of STDP[2]. Even so, the amount of research in the area is tiny. I agree that you need to figure out the whole working algorithm before trying to scale things up but to do that you need a research base and currently that is difficult because running large SNNs is very slow for an average researcher without access to specialized hardware. The primary factor which kicked off the deep learning revolution was the idea that suddenly you could train a DNN 10x faster using a GPU rather than a CPU. Without this revelation, most researchers just wouldn't bother because DL research took too long. GPUs are common and lots of people had access to them. SNNs obviously can be simulated on any CPU but for large models it's incredibly slow. SNNs have an incredible amount of potential paralellizeability but all of it is in the form of a massively asynchronous system where the order of events is critically important. CPUs and GPUs are unable to take advantage of most of the potential speedups inherent to this structure outside of specific cases where you have groups of non-interacting neurons. Due to the event based rather than signal based nature of SNNs, batched training also seems to be substantially less effective than in ANNs further complicating training. [1]https://arxiv.org/pdf/1611.01421.pdf https://arxiv.org/pdf/1611.01421.pdf [2]https://arxiv.org/pdf/1705.09132.pdf https://arxiv.org/pdf/1705.09132.pdf
        7y ago

p1esk 7y ago

And yet we have analog neural nets in our heads which most likely don’t need more than a few bits of precision.

thunderbird120 7y ago
They don't use gradient decent though. The learning mechanism used by biological neural networks has almost nothing in common with the one used by most artificial neural networks.
- p1esk 7y ago
  Then perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.
  - thunderbird120 7y ago
    The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does. I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs. [1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plasticity https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...
    - p1esk 7y ago
      STDP is one of the early steps towards understanding the learning mechanism(s) in the brain, but we are a long way from enough understanding to actually reproduce it. Not only we don't know how to train spiking networks, we don't even know how the information is encoded: pulse frequency or pulse timings or ...? No one knows. How can you compete with anything if you have no idea how it works? Also, this has nothing to do with computing hardware. You can easily simulate anything your want on conventional processors. Huge computing clusters have been built for spiking model simulations, and nothing interesting came out of it. Invent the algorithm first, then we will build the hardware for it.
      - thunderbird120 7y ago
        STDP based SNNs don't take much secret sauce to function reasonably well for basic tasks[1]. Some people have even started to apply reward modulated STDP to the problem which is nice to see since it's a more complete version of STDP[2]. Even so, the amount of research in the area is tiny. I agree that you need to figure out the whole working algorithm before trying to scale things up but to do that you need a research base and currently that is difficult because running large SNNs is very slow for an average researcher without access to specialized hardware. The primary factor which kicked off the deep learning revolution was the idea that suddenly you could train a DNN 10x faster using a GPU rather than a CPU. Without this revelation, most researchers just wouldn't bother because DL research took too long. GPUs are common and lots of people had access to them. SNNs obviously can be simulated on any CPU but for large models it's incredibly slow. SNNs have an incredible amount of potential paralellizeability but all of it is in the form of a massively asynchronous system where the order of events is critically important. CPUs and GPUs are unable to take advantage of most of the potential speedups inherent to this structure outside of specific cases where you have groups of non-interacting neurons. Due to the event based rather than signal based nature of SNNs, batched training also seems to be substantially less effective than in ANNs further complicating training. [1]https://arxiv.org/pdf/1611.01421.pdf https://arxiv.org/pdf/1611.01421.pdf [2]https://arxiv.org/pdf/1705.09132.pdf https://arxiv.org/pdf/1705.09132.pdf
        7y ago