4 ms·
MR is a pretty inflexible paradigm, and doesn't really suit graph problems well at all. Most people I know prefer to use Spark (or in the case of graph problems
by chillacy 12y ago
MR is a pretty inflexible paradigm, and doesn't really suit graph problems well at all. Most people I know prefer to use Spark (or in the case of graph problems, giraph). Of course this is for huge graphs that won't fit on a machine (petabytes). If it fits on a machine it's far easier to not muck around with distributed computing.
- ihodes 12y agoDo you know of a group using Spark on petabytes (or even a terabyte(s)) of data?
- TallGuyShort 12y agoThe scalability of Spark is certainly not as established as that of Hadoop. It's entirely anecdotal as I'm not in a position to share verifiable details, but I know of several installations that run Spark on several-hundred-node clusters. They're certainly well into terabytes, if not in the neighborhood of petabytes.
- rxin 12y agoThere are several use cases that I know of. One public one is Taobao http://databricks.com/blog/2014/08/14/mining-graph-data-with-spark-at-alibaba-taobao.html http://databricks.com/blog/2014/08/14/mining-graph-data-with...
- TallGuyShort 12y agoYou may be interested in GraphX, which describes itself as a hybrid between the "data-parallel" approach of MapReduce and core Spark, and the "graph-parallel" approach of Giraph, Graphlab, etc. See http://spark.apache.org/graphx/ http://spark.apache.org/graphx/. It allows you to do graph processing in Spark, and I quite like its approach.