There was some big news this week regarding Spark: http://gigaom.com/2013/09/25/databricks-raises-14m-from-andreessen-horowitz-wants-to-take-on-mapreduce-with-s

There was some big news this week regarding Spark: http://gigaom.com/2013/09/25/databricks-raises-14m-from-andreessen-horowitz-wants-to-take-on-mapreduce-with-spark/ http://gigaom.com/2013/09/25/databricks-raises-14m-from-andr... The creators Matei Zaharia, Ion Stoica et al raised a substantial amount. With Tachyon (in-memory file system that supports lineage and is thus fairly robust) and more recently MLBase, one should look at Spark beyond its excellent performance and really look at the overall package and versatility it provides. I think many people somewhat pointlessly get caught up in arguments about what constitutes big data and what doesn't. As someone who's used Spark substantially for machine learning as well as other complex types of processing beyond your standard joins, filters, etc Spark is incredibly useful even for a few GB of data because it allows one to iterate rapidly. With MLBase and all, I think Spark will really have an impact because your average engineer will be able to run some standard ML algorithms out of the box at scale. That is huge. That's what matters with data (big or not) -- it's the insights you can gain. Edited: typos. Also, some more on mlbase: http://www.mlbase.org http://www.mlbase.org which will be released tomorrow if am not mistaken.