3 ms·

While I personally dislike IEEE Float, we decided to remain compliant for our first chip, as that is a checkbox for a lot of businesses that we want to sell int

While I personally dislike IEEE Float, we decided to remain compliant for our first chip, as that is a checkbox for a lot of businesses that we want to sell into. We are looking at a new variable precision floating point format, called Unum, created by one of our advisers (And HPC industry legend) John Gustafson. Unum would be fantastic for deep learning, asyou would only use the precision actually required, thus bringing the program size and memory bandwidth numbers down and total energy efficiency up at least ~30%-50% over the same system with IEEE float... You can check out a previous HN discussion on it here: (https://news.ycombinator.com/item?id=9943589 https://news.ycombinator.com/item?id=9943589 We still have the option of including a 16 bit (half precision float) packed SIMD mode into our FPUs, which would add a bit of complexity (bringing our efficiency numbers down a bit for the double precision float, which we like to talk about as it is over 10x better than anything out there), but if there is enough customer interest we may decide to include it.

modeless 11y ago
Variable precision sounds scary, but maybe it could work. You should look into building support for your chips into Theano, Torch7, and/or Caffe. You should be able to hide Unum or any other quirks behind the interfaces of those libraries, so people can drop in their existing models with no work. If you can show a significant training speed advantage over GPUs, you'll have a market.
- trsohmers 11y ago
  You should check unum out... It is actually higher accuracy then IEEE float, as it does not have "rounding", which causes errors over time. unum is to floating point as floating point is to integer. The other nice thing is that it is a superset of IEEE float, and has a "IEEE mode" where you can convert to IEEE float, which is also jokingly called the "guess" function. As for support, that is the plan. Right now our customers have been most interested in high end signal processing, so we have been taking time to port FFTW, but supporting Theano/Torch/Caffe, etc is a relatively straightforward process.
  - Keyframe 11y ago
    Didn't Cray had something like that, a variable floating point where you could opt how many bits (up to a limit) you wanted? I vaguely remember hearing about it from a talk Computer History Museum had.
  - tim333 11y ago
    I just stuck a bit on unums on Wikipedia as there didn't seem to be much. https://en.wikipedia.org/wiki/John_Gustafson_(scientist)#Unums https://en.wikipedia.org/wiki/John_Gustafson_(scientist)#Unu... Feel free to correct/improve it anyone.
    - trsohmers 11y ago
      The End of Error (http://www.amazon.com/The-End-Error-Computing-Computational/dp/1482239868 http://www.amazon.com/The-End-Error-Computing-Computational/...) is the book describing them in great detail... I'm working with John to put together a publicly accessible wiki, which I hope will be up in the next month or two. That being said, the book is worth having.

modeless 11y ago

Variable precision sounds scary, but maybe it could work. You should look into building support for your chips into Theano, Torch7, and/or Caffe. You should be able to hide Unum or any other quirks behind the interfaces of those libraries, so people can drop in their existing models with no work. If you can show a significant training speed advantage over GPUs, you'll have a market.

trsohmers 11y ago
You should check unum out... It is actually higher accuracy then IEEE float, as it does not have "rounding", which causes errors over time. unum is to floating point as floating point is to integer. The other nice thing is that it is a superset of IEEE float, and has a "IEEE mode" where you can convert to IEEE float, which is also jokingly called the "guess" function. As for support, that is the plan. Right now our customers have been most interested in high end signal processing, so we have been taking time to port FFTW, but supporting Theano/Torch/Caffe, etc is a relatively straightforward process.
- Keyframe 11y ago
  Didn't Cray had something like that, a variable floating point where you could opt how many bits (up to a limit) you wanted? I vaguely remember hearing about it from a talk Computer History Museum had.
- tim333 11y ago
  I just stuck a bit on unums on Wikipedia as there didn't seem to be much. https://en.wikipedia.org/wiki/John_Gustafson_(scientist)#Unums https://en.wikipedia.org/wiki/John_Gustafson_(scientist)#Unu... Feel free to correct/improve it anyone.
  - trsohmers 11y ago
    The End of Error (http://www.amazon.com/The-End-Error-Computing-Computational/dp/1482239868 http://www.amazon.com/The-End-Error-Computing-Computational/...) is the book describing them in great detail... I'm working with John to put together a publicly accessible wiki, which I hope will be up in the next month or two. That being said, the book is worth having.