4 ms·

Spark is built around a query planner that can aggregate multiple data sources and "push down" predicates, ordering and so on into the "leaf" node itself, much

Spark is built around a query planner that can aggregate multiple data sources and "push down" predicates, ordering and so on into the "leaf" node itself, much in the same way that Cockroach does. But I suspect that while this design is flexible and fast, to get the kind of OLTP-friendly performance that Cockroach is aiming for, a good the distributed execution logic has to be fine-tuned in a way that Spark -- being a general-purpose, backend-agnostic system -- cannot do. Spark can query things like Hive, Elasticsearch, Cassandra and so on; Cockroach has the luxury of only needing to target its own engine. Also, last I checked Spark had very limited support for mutating data in SQL; it was designed for queries. Cockroach has to do both.

elvinyung 9y ago
There's overhead in the flexibility, but it seems that you'll have to do the same kind of tuning all over again if you build your own query engine from scratch. And frankly it seems (at least at a high level) that Cockroach is heading towards building the exact same kind of dataflow engine anyways[1]. Spark has some support for SQL DML, and to me it actually mostly seems like a concern for the underlying KV layer to provide ACID transactions, and expose them to Spark through the Data Sources API. [1] https://www.cockroachlabs.com/blog/local-and-distributed-processing-in-cockroachdb/ https://www.cockroachlabs.com/blog/local-and-distributed-pro...