3 ms·
This seems exactly right. The Databricks GPU stuff isn't generally available and is built on TF anyway.
by nl 10y ago
This seems exactly right. The Databricks GPU stuff isn't generally available and is built on TF anyway.
- sandGorgon 10y agohmm.. atleast databricks claims that the GPU clusters are beta, but generally available https://docs.databricks.com/user-guide/clusters/gpu.html https://docs.databricks.com/user-guide/clusters/gpu.html https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... In addition, IBM has multiple projects around GPU aware spark - https://github.com/IBMSparkGPU https://github.com/IBMSparkGPU http://www.spark.tc/gpu-acceleration-on-apache-spark-2/ http://www.spark.tc/gpu-acceleration-on-apache-spark-2/
- nl 10y agoYes the Databricks stuff is available on their cluster. It isn't part of Spark the Open Source project. Yes, as I said elsewhere there are plenty of projects to enable GPU usage via Spark. Have you actually tried them though? I have (eg https://github.com/IBMSparkGPU/GPUEnabler/issues/25 https://github.com/IBMSparkGPU/GPUEnabler/issues/25 ) and there are... issues.