5 ms·
As someone who doesn't know what Apache Spark is, this article reads like it could have been randomly generated.
by fleeno 11y ago
As someone who doesn't know what Apache Spark is, this article reads like it could have been randomly generated.
- sixdimensional 11y agoApache Spark is a general purpose distributed data processing and caching engine. It is an evolution of MapReduce concepts into more general "directed acylic graph" processing, which is very flexible for defining and executing data processing work on a distributed cluster. It's got some similarities to PrestoDb, Apache Drill and or Apache Storm (although not quite the same). It also has some nice data mining libraries, a library for handling streaming data, some connectivity to external data sources and a library for accessing data stored in its generic "data frames" via SQL. "Data frames" are just an abstraction for a dataset, but they are distributed, and in-memory and/or persistent. Personally, I like to think of as an engine for data analysis/processing and queries, but different in that it is not really a "database" like you would traditionally consider. It's almost like if you took the SQL data processing engine out of your database and made it really flexible. Edit: Also, all the functionality of Apache Spark is programmatically accessible in Java, Scala and Python, or through SQL with their Hive/thrift interface.
- mdellabitta 11y agoThere's an About page on that same site, available in the top navigation: http://databricks.com/spark/about http://databricks.com/spark/about