Yeah, it's unclear how the Hadoop cluster comes into play here. However, in general, a Spark cluster (http://spark.incubator.apache.org/ http://spark.incubator.

Yeah, it's unclear how the Hadoop cluster comes into play here. However, in general, a Spark cluster (http://spark.incubator.apache.org/ http://spark.incubator.apache.org/) will have better IO performance than Hadoop, and Spark code is much simpler than equivalent Hadoop code. Spark is a new computing framework out of Berkeley's AMPLab (https://amplab.cs.berkeley.edu/software/ https://amplab.cs.berkeley.edu/software/), and it might be an interesting platform to target. It's being adopted by Twitter, Yahoo, Amazon (http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-amplab-spark/all/ http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-am...), and it's now commercially backed by Databricks (http://databricks.com/ http://databricks.com/), which just recieved funding from Andreessen Horowitz. See https://news.ycombinator.com/item?id=6466222 https://news.ycombinator.com/item?id=6466222