For #2 and #3, see here: http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence http://spark.apache.org/docs/latest/programming-guide.html#r

For #2 and #3, see here: http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence http://spark.apache.org/docs/latest/programming-guide.html#r... We typically do #2 at our company and it's been fine so far. The bigger issue isn't a single 1 TB data set. It's multiple large data sets as one then must handle the data shuffles during joins, etc. The ability to keep the RDD in memory through the operations still tends to beat normal long winded hadoop operations anyways...