3 ms·

thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather

thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes. I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data. In your situation, how did you overcome the problems in Spark ? Did you use an accompanying toolkit to augment spark or did you build your own (hopefully not!).

nl 10y ago
I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorithms https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though. Graph stuff is generally hard, so I don't think there is a magic bullet here. I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare. In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.
- sandGorgon 10y ago
  well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.
  - nl 10y ago
    Yes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).

nl 10y ago

I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorithms https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though. Graph stuff is generally hard, so I don't think there is a magic bullet here. I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare. In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.

sandGorgon 10y ago
well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.
- nl 10y ago
  Yes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).