2 ms·
Spark cleans up some abstractions found in the Hadoop ecosystem but I would hesitate to call it the "next big thing" because it doesn't really address any of th
by jandrewrogers 13y ago
Spark cleans up some abstractions found in the Hadoop ecosystem but I would hesitate to call it the "next big thing" because it doesn't really address any of the core weaknesses of Hadoop. In Big Data, there are three touchstone applications that the current generation of platforms largely do poorly: real-time, geospatial, graph. Spark does not really address any of these. It might be more accurate to say that Spark is the "next big thing" in the Hadoop universe but still retains many of the same limitations on expressiveness, thereby creating opportunities for a more general Next Big Thing in Big Data.
Specifically, "faster batch + in-memory" is basically a simple patch on the batch mode problem but does not really address continuous flow real-time models. At large scales it is quite difficult to get robustly efficient behavior out of a parallel system that is batching things on 5 second intervals if the data flow is anything but trivial.
For geospatial and graph analysis, Spark appears to retain the same limitation of Hadoop in that it cannot deal with data models and operations without an a priori optimal partitioning function. Static hash and range partitioning won't cut it, particularly if the streaming data sources are actually real-time. The ability to generate uniform partitioning of complex data models with inherently unpredictable data distributions is critical to parallelizing some important analysis types but there is no obvious support for such mechanisms in Spark.
One could look at Spark as Hadoop done right, more or less.
- espeed 13y agoGraphX (http://amplab.github.io/graphx/ http://amplab.github.io/graphx/) is essentially GraphLab2 (http://graphlab.org/ http://graphlab.org/) built on Spark. Joey Gonzalez, the creator of GraphLab, is also the GraphX lead. Here are the papers for GraphLab and GraphX... GraphLab: http://graphlab.org/home/publications/ http://graphlab.org/home/publications/ GraphX: A Resilient Distributed Graph System on Spark (https://amplab.cs.berkeley.edu/publication/graphx-grades/ https://amplab.cs.berkeley.edu/publication/graphx-grades/) See also "Introduction to GraphX - Presented by Joseph Gonzalez, Reynold Xin - UC Berkeley AmpLab 2013" (http://www.youtube.com/watch?v=mKEn9C5bRck http://www.youtube.com/watch?v=mKEn9C5bRck)