3 ms·

The Challenges of Building Inferencing Chips

52 points by rajnathani 7y ago

MeteorMarc 7y ago
The paper is not very clear about what they mean with inferencing. It seems they just mean calculating a prediction for new data using a trained machine learning model. Readers outside the field, like me, are likely to confuse inferencing with inferring (using logic), but that is out of scope, here. Edit: find the real explanation here: https://blogs.nvidia.com/blog/2016/08/22/difference-deep-learning-training-inference-ai/ https://blogs.nvidia.com/blog/2016/08/22/difference-deep-lea...
- shmageggy 7y ago
  Also not to be confused with probabilistic inference. There are too many overloaded words in statistics and related fields.
- ampdepolymerase 7y ago
  So dot product machines or otherwise?
anonymousDan 7y ago
I've played around with some of the new 'Edge' accelerators (specifically Edge TPU and Intel NCS 2). The performance and energy efficiency of the Edge TPU especially is really amazing for the price. Does anyone know of any other edge devices worth evaluating at a similar price point (other than Nvidia Jetson)?
- rajnathani 7y ago
  If your model can fit within 10MB, and is based on CNNs, then Gyrfalcon’s [0] ‘compute in memory’ inference chips can in theory provide 3-4x higher TOPS/watt compared to the above inference chips you’ve mentioned, with around 12.6 TOPS/watt for one of their latest chips [1]. Their chips already seem to be in some new devices [2] [3]. [0] https://www.gyrfalcontech.ai/ https://www.gyrfalcontech.ai/ [1] https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelerator-chip/ https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelera... [2] https://www.zdnet.com/article/ai-comes-to-the-edge-with-solidrun-and-gyrfalcons-ai-inference-server/ https://www.zdnet.com/article/ai-comes-to-the-edge-with-soli... [3] https://www.prnewswire.com/news-releases/lg-launched-phone-powered-by-gyrfalcon-technology-incs-ai-chip-300933178.html https://www.prnewswire.com/news-releases/lg-launched-phone-p...

52 points by rajnathani 7y ago

The paper is not very clear about what they mean with inferencing. It seems they just mean calculating a prediction for new data using a trained machine learning model. Readers outside the field, like me, are likely to confuse inferencing with inferring (using logic), but that is out of scope, here. Edit: find the real explanation here: https://blogs.nvidia.com/blog/2016/08/22/difference-deep-learning-training-inference-ai/ https://blogs.nvidia.com/blog/2016/08/22/difference-deep-lea...

shmageggy 7y ago
Also not to be confused with probabilistic inference. There are too many overloaded words in statistics and related fields.
ampdepolymerase 7y ago
So dot product machines or otherwise?

anonymousDan 7y ago

I've played around with some of the new 'Edge' accelerators (specifically Edge TPU and Intel NCS 2). The performance and energy efficiency of the Edge TPU especially is really amazing for the price. Does anyone know of any other edge devices worth evaluating at a similar price point (other than Nvidia Jetson)?

rajnathani 7y ago
If your model can fit within 10MB, and is based on CNNs, then Gyrfalcon’s [0] ‘compute in memory’ inference chips can in theory provide 3-4x higher TOPS/watt compared to the above inference chips you’ve mentioned, with around 12.6 TOPS/watt for one of their latest chips [1]. Their chips already seem to be in some new devices [2] [3]. [0] https://www.gyrfalcontech.ai/ https://www.gyrfalcontech.ai/ [1] https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelerator-chip/ https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelera... [2] https://www.zdnet.com/article/ai-comes-to-the-edge-with-solidrun-and-gyrfalcons-ai-inference-server/ https://www.zdnet.com/article/ai-comes-to-the-edge-with-soli... [3] https://www.prnewswire.com/news-releases/lg-launched-phone-powered-by-gyrfalcon-technology-incs-ai-chip-300933178.html https://www.prnewswire.com/news-releases/lg-launched-phone-p...