4 ms·
The important feature is Keras compatability (although it doesn't seem to be in the official repo yet: https://github.com/fchollet/keras/pull/6800 https://githu
by minimaxir 9y ago
The important feature is Keras compatability (although it doesn't seem to be in the official repo yet: https://github.com/fchollet/keras/pull/6800 https://github.com/fchollet/keras/pull/6800)
In terms of "how is CNTK better than TensorFlow," CNTK trains 5-10x faster at minimum on pre-2.0 LSTM benchmarks, which is big since LSTMs are used for a lot nowadays. (https://arxiv.org/abs/1608.07249 https://arxiv.org/abs/1608.07249)
- albertzeyer 9y agoThis is because they use the CuDNN LSTM kernel in CNTK and in TensorFlow they chose to not use the CuDNN LSTM kernel which might be a bit unfair because they could have used it. The same for Torch. See here for more details: https://news.ycombinator.com/item?id=14473234 https://news.ycombinator.com/item?id=14473234
- justnikos 9y agoWhile it is true that CNTK can use CuDNN LSTM, if the recurrence does not fall into the 4 recurrences that CuDNN supports, CNTK is still much faster. The simplest way to verify this is to take a Keras script that uses whatever recurrent network you want and run it (on a GPU) with Tensorflow backend and with CNTK backend. Some anecdotal evidence suggest an easy 3x speedup. Disclaimer: I work at Microsoft.
- moron4hire 9y agoWhy would that be "a bit unfair"? It sounds like the TensorFlow team just hasn't been able to make as many or as good of optimizations as the CNTK team. If they could have, but chose not to do the same things, what tradeoff is there to be able to say it's "a bit unfair"?
- dgacmu 9y agoThe binding of the word "they" is unclear in the comment you're replying to. The "they" who chose not to use the cuDNN bindings were the authors of the benchmark. Some of the Torch folks filed a bug with the HKbench folks for the same error, but with respect to Torch: https://github.com/hclhkbu/dlbench/issues/14 https://github.com/hclhkbu/dlbench/issues/14
- moron4hire 9y agothanks, that makes more sense