14 ms·

Microsoft releases CNTK, its open source deep learning toolkit, on GitHub

546 points by fforflo 11y ago

baq 11y ago
quoted performance numbers on multiple GPUs leave other frameworks in the dust. where's the catch?
- pussinboots 11y ago
  looks like it isn't very scriptable so far, with the config files being made manually until "future releases" http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu...
  - tacos 11y ago
    "You can also use Brain Script (beta) to specify the configuration or use Python and C# (in future releases) to directly instantiate related objects."
  - deleted 11y ago
    [deleted]
- binarymax 11y ago
  Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs. 1:4 gives 1:2 on Torch and Caffe, and 1:4 = 1:4 on CNTK. No idea what the catch is :)
- magicmu 11y ago
  I was thinking the same thing, it almost looks a little too good to be true -- although it does kinda make sense given the focus on GPU-based clusters. I wonder how this compares to Baidu's warp-ctc [1]. They don't really seem to be the same thing, and maybe I'm missing something since I'm just starting to get into ML, but it seems to be conspicuously absent from this writeup. [1] https://github.com/baidu-research/warp-ctc https://github.com/baidu-research/warp-ctc
  - varelse 11y ago
    1-bit SGD and insanely high minibatch sizes (8192) it would appear: which drastically reduces communication costs, making data-parallel computation scale. If so, while very cool, that's not a general solution. Scaling batch sizes of 256 or lower would be the breakthrough. I suspect they get away with this because speech recognition has very sparse output targets (words/phonemes). Too bad the code below isn't open-source because they got g2 instances with ~2.5 Gb/s interconnect to scale: http://www.nikkostrom.com/publications/interspeech2015/strom_interspeech2015.pdf http://www.nikkostrom.com/publications/interspeech2015/strom...
    - mjw 11y ago
      Yep. To elaborate: really big batch sizes can speed up training data throughput, but usually mean that less is learned from each example seen, so time-to-convergence might not necessarily improve (might even increase, if you take things too far). Training data throughput isn't the right metric to compare -- look at time to convergence, or e.g. time to some target accuracy level on held-out data.
  - mjw 11y ago
    Warp-CTC implements one specific model (or at least, one specific loss function), it's not really a general framework in the same way as the other libraries mentioned.
sharms 11y ago
This is great news - I was looking at Tensorflow the other day, but on Windows / OSX it doesn't take advantage of my GPUs. For my desktop I was stuck as my new 3440x1440 monitor doesn't, for the time being, work with Xorg's intel driver. Hopefully this is a viable alternative, I would love to see a online course in machine learning leveraging this. I found http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu... on the homepage which looks well put together to start off
- hpenedones 11y ago
  Apparently there is also a CNTK book available in pdf (~150 pages) at: http://research.microsoft.com/pubs/226641/CNTKBook-20160121.pdf http://research.microsoft.com/pubs/226641/CNTKBook-20160121....
- kajecounterhack 11y ago
  What gfx card are you using? If nvidia, you should install nvidia's proprietary drivers for linux -- it would be very surprising to me if they didn't support your monitor's resolution.
- nickysielicki 11y ago
  > For my desktop I was stuck as my new 3440x1440 monitor doesn't, for the time being, work with Xorg's intel driver. I think this is now fixed upstream https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1494095 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1494095
csvan 11y ago
First TensorFlow, then Baidu's Warp-CTC, and now Microsofts CNTK. It is a very, very exciting time for open source machine learning indeed.
- dufalop 11y ago
  Definitely ! I'm looking forward to a review of the major systems available by someone who has a clue (i.e. not me). The investment to learn one of these is great enough that it will be worth some time invested to understand what each can do.
- varelse 11y ago
  Except that it appears to require MKL or ACML... Sigh... ./configure Defaulting to --with-buildtype=release Cannot find a CPU math library. Please specify --with-acml or --with-mkl with a path.
  - pmelendez 11y ago
    At least ACML seems to be open source now too. "ACML End of Life Notice: We have transitioned our math libraries from a proprietary, closed source codebase (ACML) to open source solutions"[1] [1] http://developer.amd.com/tools-and-sdks/archive/amd-core-math-library-acml/acml-downloads-resources/ http://developer.amd.com/tools-and-sdks/archive/amd-core-mat...
    - snoman 11y ago
      And, if I recall, MKL is included in Microsoft R Open (previously, Revolution R Open). At least, that's what the instructions here[1] lead me to believe. [1] https://mran.revolutionanalytics.com/documents/rro/installation/ https://mran.revolutionanalytics.com/documents/rro/installat...
      - boz_msft 11y ago
        Microsoft employee here. MKL libraries and headers will show up in CNTK soon. We are working with Intel on this.
randy324 11y ago
fuck you microsoft
tianlins 11y ago
They claimed to be "the only public toolkit that can scale beyond single machine". Well, mxnet (https://github.com/dmlc/mxnet https://github.com/dmlc/mxnet) can scale across multiple CPUs and GPUs.
- santoshalper 11y ago
  Multiple machines implies scaling across a network or backbone - multiple logical computers. As I understand it at least.
  - dgacmu 11y ago
    Mxnet does. The commenter just left that part out. "The library is portable and lightweight, and is ready scales to multiple GPUs, and multiple machines." https://mxnet.readthedocs.org/en/latest/distributed_training.html https://mxnet.readthedocs.org/en/latest/distributed_training...
doczoidberg 11y ago
Is this used on Azure Machine Learning? Do they use GPU based machines for it on Azure?
- shtangun 11y ago
  Project Philly is a Deep Learning platform that is underdevelopment in Microsoft
  - doczoidberg 11y ago
    can you share more?
    - shtangun 11y ago
      Project Philly is supposed to be a GPU platform in Azure that will enable us to run 100's of GPUs. Primary focus is Deep Learning and will be using Nvidia GPUs. CNTK, I think will be the default way to harness the Computing power. I think that is all that has been revealed. I suspect that this will be announced within a couple of months and this cntk release has something to do with that
- dionys 11y ago
  I do not think they do yet. There are some plans for GPU based VMs on Azure though
shtangun 11y ago
How many Deep Learning frameworks now? I think DL frameworks are coming up like JavaScript frameworks
- matsiyatzy 11y ago
  Here's a (currently incomplete) list of Deep Learning frameworks : https://docs.google.com/spreadsheets/d/1XvGfi3TxWm7kuQ0DUqYrO6cxva196UJDxKTxccFqb9U/edit#gid=0 https://docs.google.com/spreadsheets/d/1XvGfi3TxWm7kuQ0DUqYr... There are many :)
  - aristus 11y ago
    Missing Vowpal Wabbit: http://hunch.net/~vw/ http://hunch.net/~vw/
    - charlescearl 11y ago
      I'm not sure if it's a deep learning framework. Perhaps they are in some ways complementary http://hunch.net/?p=2875548 http://hunch.net/?p=2875548. Who knows, maybe CNTK is one big reduction.
cs702 11y ago
I'm a bit surprised they decided on a CAFFE-like declarative language for specifying neural net architectures[1], instead of offering high-level software components that enable easy composition right from within a scripting language, e.g., like Python in TensorFlow's case.[2] Is there anyone from the Microsoft team here that can explain this decision? -- [1] See examples on https://github.com/Microsoft/CNTK/wiki/CNTK-usage-overview https://github.com/Microsoft/CNTK/wiki/CNTK-usage-overview [2] See examples on https://www.tensorflow.org/versions/0.6.0/get_started/index.html https://www.tensorflow.org/versions/0.6.0/get_started/index....
- singularity2001 11y ago
  According to their slides[1] it is planned: Quoth: "Models can be described and modified with • C++ code • Network definition language (NDL) and model editing language (MEL) • Brain Script (beta) • Python and C# (planned) " [1]http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu... thanks to @sharms
  - boz_msft 11y ago
    Hi MSFT employee here. Yes. High-level bindings are planned and are a high priority. We will document the planned interfaces on GitHub soon.
    - canberroid 11y ago
      Thanks for letting us know. Any hand-wavy non-committal indication of how long that might be? Weeks/month, two months/more than three months?
- KirinDave 11y ago
  Let's hope they decide to target F#.
  - pmalynin 11y ago
mzahir 11y ago
An evaluation of the open source ML options - https://github.com/zer0n/deepframeworks https://github.com/zer0n/deepframeworks
x-sam 11y ago
Only Microsoft can open source documentation on github in docx https://github.com/Microsoft/CNTK/tree/master/Documentation/Documents https://github.com/Microsoft/CNTK/tree/master/Documentation/...
- cr4zy 11y ago
  Looks like they're moving it to GitHub wiki: https://github.com/Microsoft/CNTK/wiki/Config-file-overview https://github.com/Microsoft/CNTK/wiki/Config-file-overview
- ciupicri 11y ago
  Nevertheless isn't docx an ECMA and ISO standard? [1] [1]: https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML
blazespin 11y ago
Can't wait for sousmith's take https://github.com/soumith/convnet-benchmarks/issues https://github.com/soumith/convnet-benchmarks/issues
FLF_HOY 11y ago
"...Worry about scaling; worry about vectorization; worry about data locality...." http://www.hpcwire.com/2016/01/21/conversation-james-reinders/ http://www.hpcwire.com/2016/01/21/conversation-james-reinder... http://icri-ci.technion.ac.il/events-2/presentation-files-icri-ci-retreat-5-6-may-2015/ http://icri-ci.technion.ac.il/events-2/presentation-files-ic... http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzburg-1505051.pdf http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzbur... Nvidia Chief Scientist for Deep Learning was poached from Intel ICRI-CI Group https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pbmap https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pb... http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.html http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.ht... Look for quote: "...In a very interesting admission, LeCun told The Next Platform ..." http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-future-of-deep-learning-hardware/ http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-fu... Yann LeCun states on Nov 2015 (29:00 min mark) GPU's short lived in Deep Learning / CNN / NN https://www.youtube.com/watch?v=R7TUU94ir38 https://www.youtube.com/watch?v=R7TUU94ir38 https://www.altera.com/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf https://www.altera.com/en_US/pdfs/literature/solution-sheets...
FLF_HOY 11y ago
https://software.intel.com/en-us/blogs/2013/avx-512-instructions https://software.intel.com/en-us/blogs/2013/avx-512-instruct... "...Worry about scaling; worry about vectorization; worry about data locality...." http://www.hpcwire.com/2016/01/21/conversation-james-reinder.. http://www.hpcwire.com/2016/01/21/conversation-james-reinder.... http://www.hpcwire.com/2016/01/21/conversation-james-reinders/ http://www.hpcwire.com/2016/01/21/conversation-james-reinder... Nvidia Chief Scientist for Deep Learning was poached from Intel https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pbmap https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pb... http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.html http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.ht... AVX-512 instructions https://software.intel.com/en-us/blogs/2013/avx-512-instructions https://software.intel.com/en-us/blogs/2013/avx-512-instruct... Look for quote: "...In a very interesting admission, LeCun told The Next Platform ..." http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-future-of-deep-learning-hardware/ http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-fu... Yann LeCun states on Nov 2015 (29:00 min mark) GPU's short lived in Deep Learning / CNN / NN https://www.youtube.com/watch?v=R7TUU94ir38 https://www.youtube.com/watch?v=R7TUU94ir38 https://www.altera.com/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf https://www.altera.com/en_US/pdfs/literature/solution-sheets... http://icri-ci.technion.ac.il/events-2/presentation-files-icri-ci-retreat-5-6-may-2015/ http://icri-ci.technion.ac.il/events-2/presentation-files-ic... http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzburg-1505051.pdf http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzbur...

546 points by fforflo 11y ago

baq 11y ago

quoted performance numbers on multiple GPUs leave other frameworks in the dust. where's the catch?

pussinboots 11y ago
looks like it isn't very scriptable so far, with the config files being made manually until "future releases" http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu...
- tacos 11y ago
  "You can also use Brain Script (beta) to specify the configuration or use Python and C# (in future releases) to directly instantiate related objects."
- deleted 11y ago
  [deleted]
binarymax 11y ago
Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs. 1:4 gives 1:2 on Torch and Caffe, and 1:4 = 1:4 on CNTK. No idea what the catch is :)
magicmu 11y ago
I was thinking the same thing, it almost looks a little too good to be true -- although it does kinda make sense given the focus on GPU-based clusters. I wonder how this compares to Baidu's warp-ctc [1]. They don't really seem to be the same thing, and maybe I'm missing something since I'm just starting to get into ML, but it seems to be conspicuously absent from this writeup. [1] https://github.com/baidu-research/warp-ctc https://github.com/baidu-research/warp-ctc
- varelse 11y ago
  1-bit SGD and insanely high minibatch sizes (8192) it would appear: which drastically reduces communication costs, making data-parallel computation scale. If so, while very cool, that's not a general solution. Scaling batch sizes of 256 or lower would be the breakthrough. I suspect they get away with this because speech recognition has very sparse output targets (words/phonemes). Too bad the code below isn't open-source because they got g2 instances with ~2.5 Gb/s interconnect to scale: http://www.nikkostrom.com/publications/interspeech2015/strom_interspeech2015.pdf http://www.nikkostrom.com/publications/interspeech2015/strom...
  - mjw 11y ago
    Yep. To elaborate: really big batch sizes can speed up training data throughput, but usually mean that less is learned from each example seen, so time-to-convergence might not necessarily improve (might even increase, if you take things too far). Training data throughput isn't the right metric to compare -- look at time to convergence, or e.g. time to some target accuracy level on held-out data.
- mjw 11y ago
  Warp-CTC implements one specific model (or at least, one specific loss function), it's not really a general framework in the same way as the other libraries mentioned.

sharms 11y ago

This is great news - I was looking at Tensorflow the other day, but on Windows / OSX it doesn't take advantage of my GPUs. For my desktop I was stuck as my new 3440x1440 monitor doesn't, for the time being, work with Xorg's intel driver. Hopefully this is a viable alternative, I would love to see a online course in machine learning leveraging this. I found http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu... on the homepage which looks well put together to start off

hpenedones 11y ago
Apparently there is also a CNTK book available in pdf (~150 pages) at: http://research.microsoft.com/pubs/226641/CNTKBook-20160121.pdf http://research.microsoft.com/pubs/226641/CNTKBook-20160121....
kajecounterhack 11y ago
What gfx card are you using? If nvidia, you should install nvidia's proprietary drivers for linux -- it would be very surprising to me if they didn't support your monitor's resolution.
nickysielicki 11y ago
> For my desktop I was stuck as my new 3440x1440 monitor doesn't, for the time being, work with Xorg's intel driver. I think this is now fixed upstream https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1494095 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1494095

csvan 11y ago

First TensorFlow, then Baidu's Warp-CTC, and now Microsofts CNTK. It is a very, very exciting time for open source machine learning indeed.

dufalop 11y ago
Definitely ! I'm looking forward to a review of the major systems available by someone who has a clue (i.e. not me). The investment to learn one of these is great enough that it will be worth some time invested to understand what each can do.
varelse 11y ago
Except that it appears to require MKL or ACML... Sigh... ./configure Defaulting to --with-buildtype=release Cannot find a CPU math library. Please specify --with-acml or --with-mkl with a path.
- pmelendez 11y ago
  At least ACML seems to be open source now too. "ACML End of Life Notice: We have transitioned our math libraries from a proprietary, closed source codebase (ACML) to open source solutions"[1] [1] http://developer.amd.com/tools-and-sdks/archive/amd-core-math-library-acml/acml-downloads-resources/ http://developer.amd.com/tools-and-sdks/archive/amd-core-mat...
  - snoman 11y ago
    And, if I recall, MKL is included in Microsoft R Open (previously, Revolution R Open). At least, that's what the instructions here[1] lead me to believe. [1] https://mran.revolutionanalytics.com/documents/rro/installation/ https://mran.revolutionanalytics.com/documents/rro/installat...
    - boz_msft 11y ago
      Microsoft employee here. MKL libraries and headers will show up in CNTK soon. We are working with Intel on this.

randy324 11y ago

fuck you microsoft

tianlins 11y ago

They claimed to be "the only public toolkit that can scale beyond single machine". Well, mxnet (https://github.com/dmlc/mxnet https://github.com/dmlc/mxnet) can scale across multiple CPUs and GPUs.

santoshalper 11y ago
Multiple machines implies scaling across a network or backbone - multiple logical computers. As I understand it at least.
- dgacmu 11y ago
  Mxnet does. The commenter just left that part out. "The library is portable and lightweight, and is ready scales to multiple GPUs, and multiple machines." https://mxnet.readthedocs.org/en/latest/distributed_training.html https://mxnet.readthedocs.org/en/latest/distributed_training...

doczoidberg 11y ago

Is this used on Azure Machine Learning? Do they use GPU based machines for it on Azure?

shtangun 11y ago
Project Philly is a Deep Learning platform that is underdevelopment in Microsoft
- doczoidberg 11y ago
  can you share more?
  - shtangun 11y ago
    Project Philly is supposed to be a GPU platform in Azure that will enable us to run 100's of GPUs. Primary focus is Deep Learning and will be using Nvidia GPUs. CNTK, I think will be the default way to harness the Computing power. I think that is all that has been revealed. I suspect that this will be announced within a couple of months and this cntk release has something to do with that
dionys 11y ago
I do not think they do yet. There are some plans for GPU based VMs on Azure though

shtangun 11y ago

How many Deep Learning frameworks now? I think DL frameworks are coming up like JavaScript frameworks

matsiyatzy 11y ago
Here's a (currently incomplete) list of Deep Learning frameworks : https://docs.google.com/spreadsheets/d/1XvGfi3TxWm7kuQ0DUqYrO6cxva196UJDxKTxccFqb9U/edit#gid=0 https://docs.google.com/spreadsheets/d/1XvGfi3TxWm7kuQ0DUqYr... There are many :)
- aristus 11y ago
  Missing Vowpal Wabbit: http://hunch.net/~vw/ http://hunch.net/~vw/
  - charlescearl 11y ago
    I'm not sure if it's a deep learning framework. Perhaps they are in some ways complementary http://hunch.net/?p=2875548 http://hunch.net/?p=2875548. Who knows, maybe CNTK is one big reduction.

cs702 11y ago

I'm a bit surprised they decided on a CAFFE-like declarative language for specifying neural net architectures[1], instead of offering high-level software components that enable easy composition right from within a scripting language, e.g., like Python in TensorFlow's case.[2] Is there anyone from the Microsoft team here that can explain this decision? -- [1] See examples on https://github.com/Microsoft/CNTK/wiki/CNTK-usage-overview https://github.com/Microsoft/CNTK/wiki/CNTK-usage-overview [2] See examples on https://www.tensorflow.org/versions/0.6.0/get_started/index.html https://www.tensorflow.org/versions/0.6.0/get_started/index....

singularity2001 11y ago
According to their slides[1] it is planned: Quoth: "Models can be described and modified with • C++ code • Network definition language (NDL) and model editing language (MEL) • Brain Script (beta) • Python and C# (planned) " [1]http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tu... thanks to @sharms
- boz_msft 11y ago
  Hi MSFT employee here. Yes. High-level bindings are planned and are a high priority. We will document the planned interfaces on GitHub soon.
  - canberroid 11y ago
    Thanks for letting us know. Any hand-wavy non-committal indication of how long that might be? Weeks/month, two months/more than three months?
KirinDave 11y ago
Let's hope they decide to target F#.
- pmalynin 11y ago

mzahir 11y ago

An evaluation of the open source ML options - https://github.com/zer0n/deepframeworks https://github.com/zer0n/deepframeworks

x-sam 11y ago

Only Microsoft can open source documentation on github in docx https://github.com/Microsoft/CNTK/tree/master/Documentation/Documents https://github.com/Microsoft/CNTK/tree/master/Documentation/...

cr4zy 11y ago
Looks like they're moving it to GitHub wiki: https://github.com/Microsoft/CNTK/wiki/Config-file-overview https://github.com/Microsoft/CNTK/wiki/Config-file-overview
ciupicri 11y ago
Nevertheless isn't docx an ECMA and ISO standard? [1] [1]: https://en.wikipedia.org/wiki/Office_Open_XML https://en.wikipedia.org/wiki/Office_Open_XML

blazespin 11y ago

Can't wait for sousmith's take https://github.com/soumith/convnet-benchmarks/issues https://github.com/soumith/convnet-benchmarks/issues

FLF_HOY 11y ago

"...Worry about scaling; worry about vectorization; worry about data locality...." http://www.hpcwire.com/2016/01/21/conversation-james-reinders/ http://www.hpcwire.com/2016/01/21/conversation-james-reinder... http://icri-ci.technion.ac.il/events-2/presentation-files-icri-ci-retreat-5-6-may-2015/ http://icri-ci.technion.ac.il/events-2/presentation-files-ic... http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzburg-1505051.pdf http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzbur... Nvidia Chief Scientist for Deep Learning was poached from Intel ICRI-CI Group https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pbmap https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pb... http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.html http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.ht... Look for quote: "...In a very interesting admission, LeCun told The Next Platform ..." http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-future-of-deep-learning-hardware/ http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-fu... Yann LeCun states on Nov 2015 (29:00 min mark) GPU's short lived in Deep Learning / CNN / NN https://www.youtube.com/watch?v=R7TUU94ir38 https://www.youtube.com/watch?v=R7TUU94ir38 https://www.altera.com/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf https://www.altera.com/en_US/pdfs/literature/solution-sheets...

FLF_HOY 11y ago

https://software.intel.com/en-us/blogs/2013/avx-512-instructions https://software.intel.com/en-us/blogs/2013/avx-512-instruct... "...Worry about scaling; worry about vectorization; worry about data locality...." http://www.hpcwire.com/2016/01/21/conversation-james-reinder.. http://www.hpcwire.com/2016/01/21/conversation-james-reinder.... http://www.hpcwire.com/2016/01/21/conversation-james-reinders/ http://www.hpcwire.com/2016/01/21/conversation-james-reinder... Nvidia Chief Scientist for Deep Learning was poached from Intel https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pbmap https://il.linkedin.com/in/boris-ginsburg-2249545?trk=pub-pb... http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.html http://www.cs.tau.ac.il/~wolf/deeplearningmeeting/speaker.ht... AVX-512 instructions https://software.intel.com/en-us/blogs/2013/avx-512-instructions https://software.intel.com/en-us/blogs/2013/avx-512-instruct... Look for quote: "...In a very interesting admission, LeCun told The Next Platform ..." http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-future-of-deep-learning-hardware/ http://www.nextplatform.com/2015/08/25/a-glimpse-into-the-fu... Yann LeCun states on Nov 2015 (29:00 min mark) GPU's short lived in Deep Learning / CNN / NN https://www.youtube.com/watch?v=R7TUU94ir38 https://www.youtube.com/watch?v=R7TUU94ir38 https://www.altera.com/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf https://www.altera.com/en_US/pdfs/literature/solution-sheets... http://icri-ci.technion.ac.il/events-2/presentation-files-icri-ci-retreat-5-6-may-2015/ http://icri-ci.technion.ac.il/events-2/presentation-files-ic... http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzburg-1505051.pdf http://icri-ci.technion.ac.il/files/2015/05/00-Boris-Ginzbur...