Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024 http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu https://github.com/NervanaSystems/nervanagpu). NVIDIA themselves have also noticed and better hardware support for low-precision arithmetic is coming in Pascal: http://techreport.com/news/27978/nvidia-pascal-to-feature-mixed-precision-mode-up-to-32gb-of-ram http://techreport.com/news/27978/nvidia-pascal-to-feature-mi...