6 ms·

Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark make

Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark makes it easy to use.

Eridrus 10y ago
Spark doesn't do GPU acceleration, which is super important if you don't have a lot of spare CPU capacity. The one time I tried to train a DL model on CPU it was 48x slower, and with communication overhead it would have taken more than 48 cores to match the single GPH. And given you want to do hyperparameter search on top of that, those CPU cores start adding up.
- sandGorgon 10y ago
  that doesnt seem right. Databricks seems to have this in production. https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... I havent used this feature - but are you sure ?
  - nl 10y ago
    This seems exactly right. The Databricks GPU stuff isn't generally available and is built on TF anyway.
    - sandGorgon 10y ago
      hmm.. atleast databricks claims that the GPU clusters are beta, but generally available https://docs.databricks.com/user-guide/clusters/gpu.html https://docs.databricks.com/user-guide/clusters/gpu.html https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... In addition, IBM has multiple projects around GPU aware spark - https://github.com/IBMSparkGPU https://github.com/IBMSparkGPU http://www.spark.tc/gpu-acceleration-on-apache-spark-2/ http://www.spark.tc/gpu-acceleration-on-apache-spark-2/
      - nl 10y ago
        Yes the Databricks stuff is available on their cluster. It isn't part of Spark the Open Source project. Yes, as I said elsewhere there are plenty of projects to enable GPU usage via Spark. Have you actually tried them though? I have (eg https://github.com/IBMSparkGPU/GPUEnabler/issues/25 https://github.com/IBMSparkGPU/GPUEnabler/issues/25 ) and there are... issues.
  - EwanToo 10y ago
    It's a proprietary add-on, not part of Apache Spark itself
nl 10y ago
(Heavy Spark user here) Comparing Spark and TensorFlow is sort of like comparing Numpy and Pandas. There is some overlap, but they are pretty different things. Spark is a big data manipulation tool, which comes with a somewhat-adequate machine learning library. TensorFlow is an optimised math library with machine learning operations built on it. Spark doesn't support GPU operations (although as you note Databricks has proprietary extensions on their own cluster). DeepLearning4J and various other libraries do similar things. However, if you are building your own Neural Network architectures then TF (which has highly optimised distributed training mode) is more useful.
- sandGorgon 10y ago
  for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib (https://spark.apache.org/docs/2.0.2/mllib-guide.html https://spark.apache.org/docs/2.0.2/mllib-guide.html) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML components. how do you see spark evolving ?
  - nl 10y ago
    So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX doesn't. In my experience, we've started a lot of projects with GraphX and abandoned them because GraphX's implementations didn't have the features we needed. BTW, RDDs aren't graphs. I think you might be confusing the Spark directed-acyclic-graph (DAG) execution model with graph processing. TensorFlow doesn't have as many general purpose ML algorithms. For example, I don't think there is a Random Forest in TF, and for 90% of ML problems RF is what you need. But if you are doing Neural Network stuff then TF is exactly what you need.
agibsonccc 10y ago
Spark is vastly different. Tensorflow focuses more on numerical computing and is a low level tool. Spark is more focused on "counting at scale with a functional DSL". Hence its focus on things like ETL and columnar processing ala dataframes. As far as spark doing "deep learning" what you should mean here is: "libraries in the ecosystem leverage spark as a data access layer for doing the real numerical compute" Spark can count things with functional programming. It's not meant for heavy numerical operations. They are working on this where they can but you really can't beat a gpu or good ole simd instructions on hardware.

Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark makes it easy to use.

Eridrus 10y ago

Spark doesn't do GPU acceleration, which is super important if you don't have a lot of spare CPU capacity. The one time I tried to train a DL model on CPU it was 48x slower, and with communication overhead it would have taken more than 48 cores to match the single GPH. And given you want to do hyperparameter search on top of that, those CPU cores start adding up.

sandGorgon 10y ago
that doesnt seem right. Databricks seems to have this in production. https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... I havent used this feature - but are you sure ?
- nl 10y ago
  This seems exactly right. The Databricks GPU stuff isn't generally available and is built on TF anyway.
  - sandGorgon 10y ago
    hmm.. atleast databricks claims that the GPU clusters are beta, but generally available https://docs.databricks.com/user-guide/clusters/gpu.html https://docs.databricks.com/user-guide/clusters/gpu.html https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html https://databricks.com/blog/2016/12/21/deep-learning-on-data... In addition, IBM has multiple projects around GPU aware spark - https://github.com/IBMSparkGPU https://github.com/IBMSparkGPU http://www.spark.tc/gpu-acceleration-on-apache-spark-2/ http://www.spark.tc/gpu-acceleration-on-apache-spark-2/
    - nl 10y ago
      Yes the Databricks stuff is available on their cluster. It isn't part of Spark the Open Source project. Yes, as I said elsewhere there are plenty of projects to enable GPU usage via Spark. Have you actually tried them though? I have (eg https://github.com/IBMSparkGPU/GPUEnabler/issues/25 https://github.com/IBMSparkGPU/GPUEnabler/issues/25 ) and there are... issues.
- EwanToo 10y ago
  It's a proprietary add-on, not part of Apache Spark itself

nl 10y ago

(Heavy Spark user here) Comparing Spark and TensorFlow is sort of like comparing Numpy and Pandas. There is some overlap, but they are pretty different things. Spark is a big data manipulation tool, which comes with a somewhat-adequate machine learning library. TensorFlow is an optimised math library with machine learning operations built on it. Spark doesn't support GPU operations (although as you note Databricks has proprietary extensions on their own cluster). DeepLearning4J and various other libraries do similar things. However, if you are building your own Neural Network architectures then TF (which has highly optimised distributed training mode) is more useful.

sandGorgon 10y ago
for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib (https://spark.apache.org/docs/2.0.2/mllib-guide.html https://spark.apache.org/docs/2.0.2/mllib-guide.html) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML components. how do you see spark evolving ?
- nl 10y ago
  So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX doesn't. In my experience, we've started a lot of projects with GraphX and abandoned them because GraphX's implementations didn't have the features we needed. BTW, RDDs aren't graphs. I think you might be confusing the Spark directed-acyclic-graph (DAG) execution model with graph processing. TensorFlow doesn't have as many general purpose ML algorithms. For example, I don't think there is a Random Forest in TF, and for 90% of ML problems RF is what you need. But if you are doing Neural Network stuff then TF is exactly what you need.

agibsonccc 10y ago

Spark is vastly different. Tensorflow focuses more on numerical computing and is a low level tool. Spark is more focused on "counting at scale with a functional DSL". Hence its focus on things like ETL and columnar processing ala dataframes. As far as spark doing "deep learning" what you should mean here is: "libraries in the ecosystem leverage spark as a data access layer for doing the real numerical compute" Spark can count things with functional programming. It's not meant for heavy numerical operations. They are working on this where they can but you really can't beat a gpu or good ole simd instructions on hardware.