3 ms·

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib (https://spark.apache.org/docs/2.0.2/mllib-guide.html htt

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib (https://spark.apache.org/docs/2.0.2/mllib-guide.html https://spark.apache.org/docs/2.0.2/mllib-guide.html) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML components. how do you see spark evolving ?

nl 10y ago
So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX doesn't. In my experience, we've started a lot of projects with GraphX and abandoned them because GraphX's implementations didn't have the features we needed. BTW, RDDs aren't graphs. I think you might be confusing the Spark directed-acyclic-graph (DAG) execution model with graph processing. TensorFlow doesn't have as many general purpose ML algorithms. For example, I don't think there is a Random Forest in TF, and for 90% of ML problems RF is what you need. But if you are doing Neural Network stuff then TF is exactly what you need.
- sandGorgon 10y ago
  thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes. I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data. In your situation, how did you overcome the problems in Spark ? Did you use an accompanying toolkit to augment spark or did you build your own (hopefully not!).
  - nl 10y ago
    I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorithms https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though. Graph stuff is generally hard, so I don't think there is a magic bullet here. I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare. In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.
    - sandGorgon 10y ago
      well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.
      - nl 10y ago
        Yes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).
- Aeolos 10y ago
  Tensorflow has a GPU-accelerated Random Forest implementation: https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/tensorflow/contrib/learn/python/learn/estimators/random_forest.py https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/ten...
  - nl 10y ago
    Nice. I'm glad to be wrong. I'll point out that this is TF Contrib Learn, not TF Learn[1], or one of many other places where things might be implemented. Makes things a bit confusing. [1] http://tflearn.org/ http://tflearn.org/
- minimaxir 10y ago
  Spark 2.1.0 released this week evidentially supports multiclass logistic regression now!

nl 10y ago

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX doesn't. In my experience, we've started a lot of projects with GraphX and abandoned them because GraphX's implementations didn't have the features we needed. BTW, RDDs aren't graphs. I think you might be confusing the Spark directed-acyclic-graph (DAG) execution model with graph processing. TensorFlow doesn't have as many general purpose ML algorithms. For example, I don't think there is a Random Forest in TF, and for 90% of ML problems RF is what you need. But if you are doing Neural Network stuff then TF is exactly what you need.

sandGorgon 10y ago
thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes. I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data. In your situation, how did you overcome the problems in Spark ? Did you use an accompanying toolkit to augment spark or did you build your own (hopefully not!).
- nl 10y ago
  I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorithms https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though. Graph stuff is generally hard, so I don't think there is a magic bullet here. I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare. In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.
  - sandGorgon 10y ago
    well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.
    - nl 10y ago
      Yes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).
Aeolos 10y ago
Tensorflow has a GPU-accelerated Random Forest implementation: https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/tensorflow/contrib/learn/python/learn/estimators/random_forest.py https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/ten...
- nl 10y ago
  Nice. I'm glad to be wrong. I'll point out that this is TF Contrib Learn, not TF Learn[1], or one of many other places where things might be implemented. Makes things a bit confusing. [1] http://tflearn.org/ http://tflearn.org/
minimaxir 10y ago
Spark 2.1.0 released this week evidentially supports multiclass logistic regression now!