2 ms·
I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. W
by nl 10y ago
I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work.
What specific graph operations do you want?
If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorithms https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though.
Graph stuff is generally hard, so I don't think there is a magic bullet here.
I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare.
In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.
- sandGorgon 10y agowell, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.
- nl 10y agoYes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).