6 ms·

Shameless self promotion... my startup (http://rexcomputing.com http://rexcomputing.com) is producing a standalone chip capable of 64 GFLOPs/watt double precisi

Shameless self promotion... my startup (http://rexcomputing.com http://rexcomputing.com) is producing a standalone chip capable of 64 GFLOPs/watt double precision (128 GFLOPs/watt single precision), compared to NVIDIA's next generation chips only hitting 20GFLOPs/watt single precision... and that is before you take into account the power waste of the CPU controlling the NVIDIA GPU. Our biggest plus factors compared to a GPU is that we are a fully standalone/independent chip that does not need to have a CPU with your main system memory attached to it. Large machine learning data sets are getting into terabytes in size, and the biggest bottleneck with GPUs is the PCIe link limiting them to 16GB/s and adds a whole lot of additional latency. In our case, we have direct connection to DRAM (We've been looking at DDR4 and HMC). In addition, we have designed the architecture to allow massive scalability with up to 384 GB/s of aggregate chip-to-chip bandwidth... NVIDIA's NVLink is aiming for 80GB/s in the 2018/2019 timeframe and will still need a connected CPU to issue jobs. EDIT: I should also mention that our chip is fully general purpose, but we perform really well when it comes to dense matrix math (Most deep learning), with a 10 to 15x efficiency advantage over GPU. Our real killer app is FFTs, which GPUs do abysmally on, and our current benchmarks are showing a 25x efficiency advantage over the best DSPs and FPGAs built for large constellation FFTs.

modeless 11y ago
Single precision isn't low enough. You want half precision or maybe even lower. You also want to throw IEEE 754 out the window. Save on power and area: no denormals, no infinities, no NaNs, relaxed precision requirements. It may even be worth looking at exotic things like logarithmic number systems or analog logic (deep learning should tolerate noise extremely well). You're also going to need vast amounts of memory bandwidth, which means on-package memory, and probably specialized caches and compute units for convolution. A truly specialized deep learning chip probably wouldn't be useful for much else, but it would be a monster at deep learning. And the thing about deep learning is it scales really well. If you have a 10x faster machine you're almost certain to set world records on any machine learning benchmark you try.
- trsohmers 11y ago
  While I personally dislike IEEE Float, we decided to remain compliant for our first chip, as that is a checkbox for a lot of businesses that we want to sell into. We are looking at a new variable precision floating point format, called Unum, created by one of our advisers (And HPC industry legend) John Gustafson. Unum would be fantastic for deep learning, asyou would only use the precision actually required, thus bringing the program size and memory bandwidth numbers down and total energy efficiency up at least ~30%-50% over the same system with IEEE float... You can check out a previous HN discussion on it here: (https://news.ycombinator.com/item?id=9943589 https://news.ycombinator.com/item?id=9943589 We still have the option of including a 16 bit (half precision float) packed SIMD mode into our FPUs, which would add a bit of complexity (bringing our efficiency numbers down a bit for the double precision float, which we like to talk about as it is over 10x better than anything out there), but if there is enough customer interest we may decide to include it.
  - modeless 11y ago
    Variable precision sounds scary, but maybe it could work. You should look into building support for your chips into Theano, Torch7, and/or Caffe. You should be able to hide Unum or any other quirks behind the interfaces of those libraries, so people can drop in their existing models with no work. If you can show a significant training speed advantage over GPUs, you'll have a market.
    - trsohmers 11y ago
      You should check unum out... It is actually higher accuracy then IEEE float, as it does not have "rounding", which causes errors over time. unum is to floating point as floating point is to integer. The other nice thing is that it is a superset of IEEE float, and has a "IEEE mode" where you can convert to IEEE float, which is also jokingly called the "guess" function. As for support, that is the plan. Right now our customers have been most interested in high end signal processing, so we have been taking time to port FFTW, but supporting Theano/Torch/Caffe, etc is a relatively straightforward process.
llukas 11y ago
NVLink would be in 2016 - http://wccftech.com/nvidias-gp100-pascal-flagship-pack-4096-bit-memory-bus-8hi-hbm-stacks/ http://wccftech.com/nvidias-gp100-pascal-flagship-pack-4096-... Do you have any benchmarks on *gemm or FFTs?
- trsohmers 11y ago
  While you will be able to interconnect GPUs with NVLink in ~2016, the first CPUs supporting it won't be until 2018/2019. We will be releasing benchmark data when we have first silicon back next year. As of right now, our numbers are based on FPGA prototype implementations.
  - varelse 11y ago
    So basically you're shamelessly promoting a powerpoint processor versus shipping hardware? Pot.kettle.tapeout. I thought that was Intel's specialty. It's a very long road to a shipping chip and a lot can happen between now and then. And afterwards, it's an even longer road to the equivalent of nvcc, nvvp, cuda-gdb, cuBLAS, cuFFT, cub, cuRand, and cuDNN with solid linux/windows/mac support. And it's all free. NVIDIA achieves this through its nearly bottomless pockets from the gaming/industrial complex that has allowed them to weather many near-death experiences. What's your burn rate BTW? In the meantime, a $1000 GTX TitanX (which you can easily buy off Amazon) delivers 27 GFLOPS/W of FP32. So much for "a 10 to 25x increase in energy efficiency for the same performance level compared to existing GPU and CPU systems." At least get your facts straight. And while we're talking facts, the real challenge here is that they deliver 6.7 GFLOPS/$. That seems like a tough squeeze for a startup to beat. Again, what's your burn rate? For that's how it's ended so far for the other contenders. Why are you different? I have no problem believing NVIDIA will one day be disrupted, but nothing I read on your web site made me feel in any way that it's your architecture. Suggestion: lose the router and replace it with something simpler after studying NVIDIA's memory controller and hierarchy. They considered a lot of crazy things too and there are many reasons why GPUs ended up the way they are with a very clean programming abstraction that automagically subsumes SIMD, multithreading, and multicore.
    - 11y ago
vegabook 11y ago
Xeon Phi Knight's Landing, a product available within months, will do 500 GB/s. I
wyldfire 11y ago
> fully general purpose Programmed via OCL? Don't I need some kind of OS/host CPU functionality to load the inputs into DRAM and retrieve them?
- trsohmers 11y ago
  We're building our LLVM backend, allowing us to support any language with a LLVM frontend. You do not need an OS or host CPU to move memory (unlike a lot of similar many core processors), and each core is fully capable of loading/storing from any cores scratchpad (either on a single chip or over multiple connected chips) or to/from DRAM.
tim333 11y ago
Any idea how much your chip is likely to cost?

modeless 11y ago

Single precision isn't low enough. You want half precision or maybe even lower. You also want to throw IEEE 754 out the window. Save on power and area: no denormals, no infinities, no NaNs, relaxed precision requirements. It may even be worth looking at exotic things like logarithmic number systems or analog logic (deep learning should tolerate noise extremely well). You're also going to need vast amounts of memory bandwidth, which means on-package memory, and probably specialized caches and compute units for convolution. A truly specialized deep learning chip probably wouldn't be useful for much else, but it would be a monster at deep learning. And the thing about deep learning is it scales really well. If you have a 10x faster machine you're almost certain to set world records on any machine learning benchmark you try.

trsohmers 11y ago
While I personally dislike IEEE Float, we decided to remain compliant for our first chip, as that is a checkbox for a lot of businesses that we want to sell into. We are looking at a new variable precision floating point format, called Unum, created by one of our advisers (And HPC industry legend) John Gustafson. Unum would be fantastic for deep learning, asyou would only use the precision actually required, thus bringing the program size and memory bandwidth numbers down and total energy efficiency up at least ~30%-50% over the same system with IEEE float... You can check out a previous HN discussion on it here: (https://news.ycombinator.com/item?id=9943589 https://news.ycombinator.com/item?id=9943589 We still have the option of including a 16 bit (half precision float) packed SIMD mode into our FPUs, which would add a bit of complexity (bringing our efficiency numbers down a bit for the double precision float, which we like to talk about as it is over 10x better than anything out there), but if there is enough customer interest we may decide to include it.
- modeless 11y ago
  Variable precision sounds scary, but maybe it could work. You should look into building support for your chips into Theano, Torch7, and/or Caffe. You should be able to hide Unum or any other quirks behind the interfaces of those libraries, so people can drop in their existing models with no work. If you can show a significant training speed advantage over GPUs, you'll have a market.
  - trsohmers 11y ago
    You should check unum out... It is actually higher accuracy then IEEE float, as it does not have "rounding", which causes errors over time. unum is to floating point as floating point is to integer. The other nice thing is that it is a superset of IEEE float, and has a "IEEE mode" where you can convert to IEEE float, which is also jokingly called the "guess" function. As for support, that is the plan. Right now our customers have been most interested in high end signal processing, so we have been taking time to port FFTW, but supporting Theano/Torch/Caffe, etc is a relatively straightforward process.

llukas 11y ago

NVLink would be in 2016 - http://wccftech.com/nvidias-gp100-pascal-flagship-pack-4096-bit-memory-bus-8hi-hbm-stacks/ http://wccftech.com/nvidias-gp100-pascal-flagship-pack-4096-... Do you have any benchmarks on *gemm or FFTs?

trsohmers 11y ago
While you will be able to interconnect GPUs with NVLink in ~2016, the first CPUs supporting it won't be until 2018/2019. We will be releasing benchmark data when we have first silicon back next year. As of right now, our numbers are based on FPGA prototype implementations.
- varelse 11y ago
  So basically you're shamelessly promoting a powerpoint processor versus shipping hardware? Pot.kettle.tapeout. I thought that was Intel's specialty. It's a very long road to a shipping chip and a lot can happen between now and then. And afterwards, it's an even longer road to the equivalent of nvcc, nvvp, cuda-gdb, cuBLAS, cuFFT, cub, cuRand, and cuDNN with solid linux/windows/mac support. And it's all free. NVIDIA achieves this through its nearly bottomless pockets from the gaming/industrial complex that has allowed them to weather many near-death experiences. What's your burn rate BTW? In the meantime, a $1000 GTX TitanX (which you can easily buy off Amazon) delivers 27 GFLOPS/W of FP32. So much for "a 10 to 25x increase in energy efficiency for the same performance level compared to existing GPU and CPU systems." At least get your facts straight. And while we're talking facts, the real challenge here is that they deliver 6.7 GFLOPS/$. That seems like a tough squeeze for a startup to beat. Again, what's your burn rate? For that's how it's ended so far for the other contenders. Why are you different? I have no problem believing NVIDIA will one day be disrupted, but nothing I read on your web site made me feel in any way that it's your architecture. Suggestion: lose the router and replace it with something simpler after studying NVIDIA's memory controller and hierarchy. They considered a lot of crazy things too and there are many reasons why GPUs ended up the way they are with a very clean programming abstraction that automagically subsumes SIMD, multithreading, and multicore.
  - 11y ago

vegabook 11y ago

Xeon Phi Knight's Landing, a product available within months, will do 500 GB/s. I

wyldfire 11y ago

> fully general purpose Programmed via OCL? Don't I need some kind of OS/host CPU functionality to load the inputs into DRAM and retrieve them?

trsohmers 11y ago
We're building our LLVM backend, allowing us to support any language with a LLVM frontend. You do not need an OS or host CPU to move memory (unlike a lot of similar many core processors), and each core is fully capable of loading/storing from any cores scratchpad (either on a single chip or over multiple connected chips) or to/from DRAM.

tim333 11y ago

Any idea how much your chip is likely to cost?