4 ms·
Do you know of a group using Spark on petabytes (or even a terabyte(s)) of data?
by ihodes 12y ago
Do you know of a group using Spark on petabytes (or even a terabyte(s)) of data?
- TallGuyShort 12y agoThe scalability of Spark is certainly not as established as that of Hadoop. It's entirely anecdotal as I'm not in a position to share verifiable details, but I know of several installations that run Spark on several-hundred-node clusters. They're certainly well into terabytes, if not in the neighborhood of petabytes.
- rxin 12y agoThere are several use cases that I know of. One public one is Taobao http://databricks.com/blog/2014/08/14/mining-graph-data-with-spark-at-alibaba-taobao.html http://databricks.com/blog/2014/08/14/mining-graph-data-with...