2 ms·

that doesnt seem right. Databricks seems to have this in production. https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databric

that doesnt seem right. Databricks seems to have this in production. https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... I havent used this feature - but are you sure ?

nl 10y ago
This seems exactly right. The Databricks GPU stuff isn't generally available and is built on TF anyway.
- sandGorgon 10y ago
  hmm.. atleast databricks claims that the GPU clusters are beta, but generally available https://docs.databricks.com/user-guide/clusters/gpu.html https://docs.databricks.com/user-guide/clusters/gpu.html https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... In addition, IBM has multiple projects around GPU aware spark - https://github.com/IBMSparkGPU https://github.com/IBMSparkGPU http://www.spark.tc/gpu-acceleration-on-apache-spark-2/ http://www.spark.tc/gpu-acceleration-on-apache-spark-2/
  - nl 10y ago
    Yes the Databricks stuff is available on their cluster. It isn't part of Spark the Open Source project. Yes, as I said elsewhere there are plenty of projects to enable GPU usage via Spark. Have you actually tried them though? I have (eg https://github.com/IBMSparkGPU/GPUEnabler/issues/25 https://github.com/IBMSparkGPU/GPUEnabler/issues/25 ) and there are... issues.
EwanToo 10y ago
It's a proprietary add-on, not part of Apache Spark itself