5 ms·

Awesome writeup. I've occasionally thought that there's a lot of time to be saved in machine learning systems by making the floating point operations a bit mor

Awesome writeup. I've occasionally thought that there's a lot of time to be saved in machine learning systems by making the floating point operations a bit more noisy. Use maybe eight bit (or even four bit) numbers for the matrix entries, with the philosophy that the overall degree of freedom in the system is enough to make up for this local rigidity, while you gain a lot in terms of both storage space and evaluation time. Along these lines, I saw a great demo of a robot a while ago which used 256 binary devices for movement. Each of these devices is the diagonal of a quadrilateral; when the device is in it's 'on' state, the diagonal is long, and when it's in the 'off' state the diagonal is short. Chain a lot of these together, and you can get pretty good accuracy aiming for specific points in a two-dimensional space. The demo I saw was a three-dimensional version of the same concept, of course. So likewise, we chain together lots of linear operations to build (say) a deep learning system. The weight space is very high-dimensional, and the decision space is the much lower dimensional classification/regression problem: one probably needs a lot less than 64 bits of accuracy in the weight space to approximate a point in the decision space.

benanne 12y ago
Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024 http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu https://github.com/NervanaSystems/nervanagpu). NVIDIA themselves have also noticed and better hardware support for low-precision arithmetic is coming in Pascal: http://techreport.com/news/27978/nvidia-pascal-to-feature-mixed-precision-mode-up-to-32gb-of-ram http://techreport.com/news/27978/nvidia-pascal-to-feature-mi...
- sdenton4 12y ago
  Awesome, thanks for the pointers!
- ot 12y ago
  I also liked a lot this ICML paper [1], which is more theoretically principled than the two you reference (and weirdly they do not cite it). [1] http://www.eecs.tufts.edu/~dsculley/papers/round-model-icml.pdf http://www.eecs.tufts.edu/~dsculley/papers/round-model-icml....
Houshalter 12y ago
The number of bits does seem to affect neural network training and really kills it when you get really low (like 8 to 12 bits). The problem is that really small numbers are rounded down to zero, and then when summed together the result is really inaccurate. Ideally you could do some kind of stochastic rounding, where the decimal value represents it's probability of being rounded up or down. On average the totals would still be the same. You could probably get the required number of bits really low then, perhaps even only one or two! Something to look into when these algorithms are put into FPGAs and ASICs, which is starting to happen.
- andor 12y ago
  The problem is that really small numbers are rounded down to zero, and then when summed together the result is really inaccurate Use log values instead?
  - Houshalter 12y ago
    Using the log domain is cool but log summation still has the same problem. Logs just transform numbers into a different range, they don't magically add more bits of precision. That is adding a very small number in the log domain to another much larger number in the log domain, will just round it down.

benanne 12y ago

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024 http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu https://github.com/NervanaSystems/nervanagpu). NVIDIA themselves have also noticed and better hardware support for low-precision arithmetic is coming in Pascal: http://techreport.com/news/27978/nvidia-pascal-to-feature-mixed-precision-mode-up-to-32gb-of-ram http://techreport.com/news/27978/nvidia-pascal-to-feature-mi...

sdenton4 12y ago
Awesome, thanks for the pointers!
ot 12y ago
I also liked a lot this ICML paper [1], which is more theoretically principled than the two you reference (and weirdly they do not cite it). [1] http://www.eecs.tufts.edu/~dsculley/papers/round-model-icml.pdf http://www.eecs.tufts.edu/~dsculley/papers/round-model-icml....

Houshalter 12y ago

The number of bits does seem to affect neural network training and really kills it when you get really low (like 8 to 12 bits). The problem is that really small numbers are rounded down to zero, and then when summed together the result is really inaccurate. Ideally you could do some kind of stochastic rounding, where the decimal value represents it's probability of being rounded up or down. On average the totals would still be the same. You could probably get the required number of bits really low then, perhaps even only one or two! Something to look into when these algorithms are put into FPGAs and ASICs, which is starting to happen.

andor 12y ago
The problem is that really small numbers are rounded down to zero, and then when summed together the result is really inaccurate Use log values instead?
- Houshalter 12y ago
  Using the log domain is cool but log summation still has the same problem. Logs just transform numbers into a different range, they don't magically add more bits of precision. That is adding a very small number in the log domain to another much larger number in the log domain, will just round it down.