5 ms·
When I first learned about Spark, I knew this team would go on to build great things and they've gone beyond my expectations all the while being very friendly a
by izendejas 13y ago
When I first learned about Spark, I knew this team would go on to build great things and they've gone beyond my expectations all the while being very friendly and supportive of the community.
Among the things I love most about the Spark and the eco-system:
* repl -- so great for running short little experiments or even full blown jobs. saves you time compiling small changes and really get to know your data quickly.
* caching -- processing in-memory opens up many possibilities beyond doing iterative, machine learning jobs. quantifind, for example, demoed a system that allows them to run ad-hoc queries using Shark on the fly across GBs of data (think OLAP, a bit) in seconds or less.
* scala -- makes for very succinct code using closures and built-in operations; check out some examples here: http://spark-project.org/examples/ http://spark-project.org/examples/
And some of the upcoming projects are also very cool. Tachyon, for example, will enable users to share data with a very robust in-memory file system. A teammate and I, for example, could have used it recently because we were simultaneously running different analysis against the same data, so we had to cache duplicate instances on two clusters.