5 ms·

It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.

wcrichton 10y ago
Intel is pushing their own chips for deep learning (i.e. Xeon and Xeon Phi). They claim that using MKL on these chips gets comparable performance to using Caffe/cuDNN on a high-end GPU.
- Eridrus 10y ago
  Phi may yet win out, I saw a talk from an NLP researcher about Seq2Seq models and someone asked him what his wish for hardware was and it was for GPU cores to not be as stuck in lock step with each other control-wise. Not sure if he knew about the Phi's though.
sandGorgon 10y ago
gpu acceleration in spark in generally production ready - https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... in fact looks like you can use tensorflow models in spark with GPU - https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data...
- agibsonccc 10y ago
  Tensorframes despite the marketing is already defunct. It hasn't seen a commit since august. https://github.com/databricks/tensorframes https://github.com/databricks/tensorframes Spark is just a data access layer here. It's not even remotely gpu friendly. Most people also still relies on mesos or yarn for running distributed. The library you're using matters alot. Mesos just added gpu support: http://mesos.apache.org/documentation/latest/gpu-support/ http://mesos.apache.org/documentation/latest/gpu-support/ Yarn can sorta support it with node labeling for job completion but it's still kinda hacky. The real work in this space (without the marketing) is done by IBM: http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark When spark can (without "production ready" buzzwords) run gpus like this out of the box then we're talking. For now spark needs a companion library to work with gpus though.

Intel is pushing their own chips for deep learning (i.e. Xeon and Xeon Phi). They claim that using MKL on these chips gets comparable performance to using Caffe/cuDNN on a high-end GPU.

Eridrus 10y ago
Phi may yet win out, I saw a talk from an NLP researcher about Seq2Seq models and someone asked him what his wish for hardware was and it was for GPU cores to not be as stuck in lock step with each other control-wise. Not sure if he knew about the Phi's though.

sandGorgon 10y ago

gpu acceleration in spark in generally production ready - https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... in fact looks like you can use tensorflow models in spark with GPU - https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data...

agibsonccc 10y ago
Tensorframes despite the marketing is already defunct. It hasn't seen a commit since august. https://github.com/databricks/tensorframes https://github.com/databricks/tensorframes Spark is just a data access layer here. It's not even remotely gpu friendly. Most people also still relies on mesos or yarn for running distributed. The library you're using matters alot. Mesos just added gpu support: http://mesos.apache.org/documentation/latest/gpu-support/ http://mesos.apache.org/documentation/latest/gpu-support/ Yarn can sorta support it with node labeling for job completion but it's still kinda hacky. The real work in this space (without the marketing) is done by IBM: http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark When spark can (without "production ready" buzzwords) run gpus like this out of the box then we're talking. For now spark needs a companion library to work with gpus though.